 MSDN Magazine September 2003

MSDN Magazine September 2003

Office 2003: Bring the Power of Visual Studio .NET to Business Solutions Built with Microsoft Office

Microsoft Visual Studio Tools for the Microsoft Office System is a new technology that brings the advanced features of Visual Studio .NET and the .NET Framework to apps built on Microsoft Word 2003 and Excel 2003. Now you can use Visual Basic .NET and C# to write document-centric, managed code solutions that run in-process with Word 2003 or Excel 2003, taking advantage of the rich object models they expose. Along the way you get the benefits of the managed environment in which a fully compiled .NET-based application executes, including code access security.

Ken Getz and Brian A. Randell InfoPath: Turn User Input into XML with Custom Forms Using Office InfoPath 2003

Office InfoPath 2003 is a new Microsoft Office product that lets you design your own data collection forms that, when submitted, turn the user-entered data into XML for any XML-supporting process to use. With an InfoPath solution in place, you can convert all those commonly used paper forms into Microsoft Office-based forms and end the cycle of handwriting and reentering data into your systems. Today organizations are beginning to realize the value of the mountains of data they collect every day, how hard it is to access it, and are striving to mine it effectively. InfoPath will aid in the design of effective data collection systems. Here the author shows you how to get started.

Aaron Skonnard .NET Remoting: Create a Custom Marshaling Implementation Using .NET Remoting and COM Interop

The .NET Framework offers several methods for customizing the presentation of native .NET and COM object types. One such technique, custom marshaling, refers to the notion of specializing object type presentations. There are times, like when a legacy COM component needs to implement a new interface or when you need to make calls across process or machine boundaries, when custom marshaling saves the day. Elements of COM Interop permit the customizing of COM types while .NET Remoting offers the developer the ability to tailor native .NET types. This article examines these techniques.

Jim Sievert .NET Internals: Rewrite MSIL Code on the Fly with the .NET Framework Profiling API

In this article, the author shows how to dynamically rewrite Microsoft Intermediate Language code on the fly using the Profiling API of the CLR. Unlike approaches based on Reflection.Emit, this scheme works with the existing assemblies and doesn't require the creation of proxy or dynamic assemblies. The need for IL code rewriting emerges when you want to make your changes transparent to the client and preserve the identity of classes. This technique can be used for creation of interceptors, pre- and post-processing method calls, and code instrumentation and verification.

Aleksandr Mikunov Coroutines: Implementing Coroutines for .NET by Wrapping the Unmanaged Fiber API

Coroutines are a powerful feature of many programming languages including CLU, Scheme, Python, Ruby, and ICON. Coroutines can save processor overhead and reduce redundancy because they allow you to stop execution of a procedure midstream, return a value, and resume exactly where the procedure left off.This article shows how coroutines can be implemented for the .NET Framework by using the Fiber API and Managed Extensions for C++, and how they can be easily used with other .NET-compliant languages. This article also shows a sophisticated use of the runtime host for running multiple managed threads on a single OS thread.

Ajai Shankar Visual C++ 6.0: Don't Let Memory Allocation Failures Crash Your Legacy STL Application

Most C++ developers make extensive use of the Standard Template Library (STL) in their code. If you are one of them and are using STL and Visual C++ 6.0 directly out of the box, your application is at high risk of crashing under low memory conditions. The problem arises because checking for failure of operator new is such an uncommon practice. To make things worse, when new does fail, the response is not standard. Some language compilers return NULL while others throw an exception.In addition, if you are using STL in an MFC project, be aware that MFC has its own set of rules. This article discusses these problems, explains how the default behavior has changed in Visual C++ .NET 2003, and outlines the changes you must make if you're using Visual C++ 6.0 so that you can safely use the STL when operator new fails.

James Hebben Editor's Note: Tales of a GPS Demo

One of the most annoying things about the Internet is that no matter how hard you try to make something permanent, it can be gone tomorrow with no forwarding address. We found this out the hard way earlier this year, and we're hoping to remedy the situation with this Editor's Note.

New Stuff: Resources for Your Developer Toolbox

ISYS/Odyssey Development Inc. now supports the Microsoft® . NET Framework with the availability of the ISYS:web. asp 6. 0 search engine product. Developers using ASP. NET can interface directly with the core ISYS engine, powering their Web applications with the search and retrieval features of ISYS:web.

Theresa W. Carey Web Q&A: InfoPath Back End, WSH Script Signing, and More

Edited by Nancy Michell Data Points: Developing Apps with the .NET Compact Framework, SQL Server CE, and Replication

John Papa Cutting Edge: Managing Your Remote Windows Clipboard

Dino Esposito The XML Files: Introducing the Web Services Enhancements 2.0 Messaging API

Aaron Skonnard Advanced Basics: Creating Text Images On the Fly with GDI+

Ken Spencer The ASP Column: The Internet Explorer Toolbar Control

George Shepherd .NET Column: Introducing Generics in the CLR

Jason Clark C++ Q&A: Retrieving Hidden Path Names, Mouse Events in C#

Paul DiLascia Resource File: Microsoft Virtual Machine Remediation

Developers who use the Microsoft virtual machine (VM) will need to transition away from its use by January 2004. To comply with a court settlement with Sun Microsystems, Microsoft will not be able to update the Microsoft VM, even to repair critical security vulnerabilities.


Office 2003: Bring the Power of Visual Studio .NET to Business Solutions Built with Microsoft Office

Office 2003

Bring the Power of Visual Studio .NET to Business Solutions Built with Microsoft Office

Ken Getz and Brian A. Randell
This article assumes you're familiar with Office, C# or Visual Basic .NET
Level of Difficulty 1 2 3
Code download available at: MicrosoftOffice2003.exe (152KB)
SUMMARY

Microsoft Visual Studio Tools for the Microsoft Office System is a new technology that brings the advanced features of Visual Studio .NET and the .NET Framework to apps built on Microsoft Word 2003 and Excel 2003. Now you can use Visual Basic .NET and C# to write document-centric, managed code solutions that run in-process with Word 2003 or Excel 2003, taking advantage of the rich object models they expose. Along the way you get the benefits of the managed environment in which a fully compiled .NET-based application executes, including code access security.

Contents
Benefits
Getting Started
Investigating the Plumbing
Completing the Demonstration

M
icrosoft® Office provides a rich environment for creating business applications. With the COM-based object models exposed by Office applications, you can use Visual Basic® for Applications (VBA) code, out-of-process COM Automation, and COM add-ins to create unmanaged apps that target Office.

You can run VBA code directly in the Office application itself. This code runs in process, and the code and compiled bits are generally stored with the document (all Office applications except Outlook® store code within the document). Keep in mind that hosted VBA code is compiled to p-code, not native code, and this incurs a slight performance penalty, although running in process enables optimized method calls to Office objects.

You can also write applications in Visual Basic or other COM-aware languages that interact with the publicly exposed interfaces provided by the Office applications. In general, these applications run out of process, requiring calls through the COM proxy/stub layer. Automation applications are often compiled to native code and therefore execute faster than VBA code. Mitigating this benefit, however, is their decreased performance because of cross-application method calls.

Starting with Office 2000, Office applications search for registered add-in components that implement the IDTExtensibility2 interface, and load the add-ins either at startup or on demand, depending on how the add-in was configured. Using Visual Basic and other COM-aware languages and VBA, you can create applications that implement this interface—even using the Office Developer product and VBA, you can create COM add-ins. This level of flexibility provides a good compromise, allowing fully compiled applications to run in process. This technique does require registration of the components so that Office applications are aware of the add-in's existence. You can create a single COM add-in application that will load within multiple Office applications, and the COM add-in itself provides a cross-document toolset—its behavior doesn't inherently change from one document to the next.

There may be times when you find yourself in the world of managed code, and you'd like to interact with Office in a secure, garbage-collected world. It turns out that due to the wonders of COM interop and the common language runtime's (CLR) runtime-callable wrapper (RCW), you can still call out-of-process to methods within an Office application, and with a little effort, you can create COM add-ins using managed code. VBA, however, remains unmanaged and is likely to remain so in the foreseeable future.

Writing Framework-based code that uses Automation to control Office applications is fine if your intent is to create a front-end application yourself; users will need to start your application, which might then load Word or another application either visibly or invisibly. If you need to create a solution that can run in multiple Office hosts and is document agnostic, you can create a COM add-in using managed code.

It's possible that neither of these models fits your needs. Perhaps your users are used to loading a Word document and having your document management application run, or opening an Excel spreadsheet and performing statistical analysis on the contents of the workbook. Perhaps you want to use managed code to respond to Office application events. Maybe you want to be able to use Visual Basic .NET or C# code to react to a button or menu click that occurred within a Word application. Perhaps you want to display a Windows® Form created in Visual Studio® .NET in response to a request for input on an Excel spreadsheet. To satisfy these requirements, Microsoft has released Visual Studio Tools for Microsoft Office, an add-in for Visual Studio .NET 2003 that lets you create document-centric, managed applications that run in process with either Word 2003 or Excel 2003.

Benefits

There are many benefits to Visual Studio Tools for Office. You can use all the features of Visual Studio .NET to create Office-based applications. You can use either Visual Basic .NET or C# to create your applications. Both languages are more powerful than VBA. You can leverage all the features of the .NET Framework, from XML readers to ADO.NET to Web Services, from within Office applications.

You can also take advantage of the code access security that's built into the CLR. Unlike VBA macros, managed solutions using Visual Studio Tools for Office simply won't run unless they're explicitly trusted. The default settings used by the Visual Studio Tools for Office loader do not permit any assembly to run, which protects users from viruses and other malicious code. Before users can load a document with managed code extensions, the administrator must explicitly grant full trust to an assembly or location. Visual Studio Tools for Office allows you to take advantage of no-touch deployment. Users can automatically receive updates to the application with no intervention on their part.

Getting Started

In order to get your feet wet, we'll look at installation, then walk you through the process of creating a simple Excel application using Visual Basic .NET or C# code that will retrieve data and chart it.

Installing Visual Studio Tools for Office adds a set of project templates to Visual Studio .NET 2003, enabling you to create Word documents, template projects, or Excel worksheets using Visual Basic .NET or C# code to react to events and automate the products. Once you've installed Visual Studio .NET 2003, followed by Microsoft Office 2003, and finally, Visual Studio Tools for the Microsoft Office System (in that order), you'll be ready to create applications that take advantage of the Microsoft Office 2003 Primary Interop Assemblies (PIAs) stored in the Global Assembly Cache (GAC). You'll be able to create Word or Excel projects from within Visual Studio .NET that use Visual Basic .NET or C# code to automate Office in process. (And no, you haven't missed anything here—Visual Studio Tools for Office only allows you to create Word and Excel projects from within Visual Studio .NET at this time. You may see expanded support for other Office products as time goes on.)

To get started with the sample app, you need to create an Excel project in Visual Studio .NET. First, verify that you've installed Visual Studio .NET 2003, Office 2003, and Visual Studio Tools for Office. When installing Office 2003, it's imperative that you either perform a full installation or that you individually select .NET support for each of the Office products so that the appropriate PIAs are installed in the GAC. It's easiest to install the full Office product, but you can also individually install the PIAs for Word, Excel, Microsoft Forms, and Microsoft Graph.

Next, in Visual Studio .NET, use the File | New | Project menu item to select the project and template type. First select Microsoft Office 2003 Projects. Select either Visual Basic Projects or C# Projects, and then select Excel Workbook as the template type. Enter a project name and click OK when you're finished.

Finally, the second page of the Microsoft Office Project Wizard allows you to choose whether to use an existing document or to create a new one. You can also use the Security Settings link on this page to bypass the wizard's default offer to set up the necessary code access security for your project. For now, select the Create New Document option, choosing a convenient location for the project. Click Finish to complete the wizard's steps and create the new project. (Don't change the default security settings now. Wait until you're an expert with code access security and with the settings used by Visual Studio Tools for Office. Changing security settings at this point might make it impossible to run or debug your applications.)

Once you've created the project, you'll find that the Solution Explorer window includes references to the appropriate Office application, along with a file containing a class named OfficeCodeBehind that enables you to hook up event handlers for the document's events.

At the top of the code file (ThisDocument or ThisWorkbook, with the appropriate file extension for your selected language), you'll find Imports or using statements that simplify typing references to the objects within your host's object model:

' Visual Basic .NET
Imports Excel = Microsoft.Office.Interop.Excel

// C#
using Excel = Microsoft.Office.Interop.Excel;
Next, you'll find an attribute describing the class:
' Visual Basic .NET
' Office integration attribute. Identifies the startup class
' for the document. Do not modify.
<Assembly: System.ComponentModel.DescriptionAttribute( _
   "OfficeStartupClass, Version=1.0, _
   Class=ChartDemo.OfficeCodeBehind")>

// C#
// Office integration attribute. Identifies the startup class
// for the document. Do not modify.
[assembly:System.ComponentModel.DescriptionAttribute(
  "OfficeStartupClass, Version=1.0,
  Class=ChartDemo.OfficeCodeBehind")]

Although the included comment indicates that you shouldn't modify this attribute, there are occasions in which you must. For example, if you modify the class name or namespace, you'll need to manually update this attribute to match. The Visual Studio Tools for Office loader uses this attribute to determine the name of the class it should load as users open the host document—more on that later in this article.

Next, you'll find declarations for variables that correspond to the application and main document objects (ThisWorkbook and ThisApplication for Excel, and ThisDocument and ThisApplication for Word), as shown in Figure 1. In addition, the classes contain event handlers for basic events (Workbook.Open and Workbook.BeforeClose in Excel, Document.Open and Document.Close in Word), as shown in Figure 2.

Figure 2 Event Handlers (Close Figure 2)

Visual Basic .NET
Private Sub ThisWorkbook_Open() Handles ThisWorkbook.Open

End Sub

Private Sub ThisWorkbook_BeforeClose(ByRef Cancel As Boolean) _
    Handles ThisWorkbook.BeforeClose
    Cancel = False
End Sub
C#
private Excel.WorkbookEvents_OpenEventHandler openEvent;
private Excel.WorkbookEvents_BeforeCloseEventHandler
    beforeCloseEvent;

protected void ThisWorkbook_Open()
{

}

protected void ThisWorkbook_BeforeClose(ref bool Cancel)
{
    Cancel = false;
}
Figure 1 Declarations (Close Figure 1)

Visual Basic .NET
Friend WithEvents ThisWorkbook As Excel.Workbook
Friend WithEvents ThisApplication As Excel.Application
C#
internal Excel.Application ThisApplication
{
  get { return thisApplication;}
}

internal Excel.Workbook ThisWorkbook
{
  get { return thisWorkbook;}
}

private Excel.Application thisApplication = null;
private Excel.Workbook thisWorkbook = null;

There is a bit more going on and if you expand the region named Generated initialization code within your class you'll find a few more procedures, which we'll discuss later in this article. For now, scroll down within the class and locate the ThisWorkbook_Open procedure. Add code so that the procedures look like the examples in Figure 3. Note the differences between the Visual Basic and C# code here. For example, in Visual Basic .NET, you can simply call the Application.Range property, passing a single parameter. The Application.Range property has been defined to accept two optional parameters, but C# supports neither parameterized properties nor optional parameters. Therefore, when creating the PIA for Excel, the team added the get_Range method, one of several accessor methods that make it possible for C# developers to work with the Excel object model. The same issue applies to the Range.Value property, which accepts a single parameter. In C#, you must use the Value2 property, which doesn't accept any parameters. In Visual Basic you can also use Value2, but there's no advantage to doing so.

Figure 3 Set Properties (Close Figure 3)

Visual Basic .NET
Private Sub ThisWorkbook_Open() Handles ThisWorkbook.Open
  Dim rng As Excel.Range = ThisApplication.Range("A1")
  rng.Value = DateTime.Now.ToString()
  rng.Font.Name = "Verdana"
  rng.Font.Size = 16
End Sub
C#
protected void ThisWorkbook_Open()
{
  Excel.Range rng = ThisApplication.get_Range("A1", Type.Missing);
  rng.Value2 = DateTime.Now.ToString();
  rng.Font.Name = "Verdana";
  rng.Font.Size = 16;
}

Select Debug | Start to run the application. Visual Studio .NET compiles and runs the application, which loads Excel. If you've followed the steps carefully, you should see a spreadsheet appear with the correctly formatted text in cell A1. When you're finished admiring your work, close Excel and return to design mode.

To see what you've done, save your project, close Visual Studio .NET, and use Windows Explorer to browse to the folder containing your project and the Excel workbook file. Double-click the XLS file to load Excel, and verify that the appropriate text appears in the worksheet. Reload Visual Studio .NET and your sample project to continue the walkthrough.

As trivial as it may seem, you've accomplished something that would be quite tricky to tackle on your own—you've written managed code that reacts to an Excel event by interacting with the Excel object model from within Visual Basic .NET or C# code.

Investigating the Plumbing

When you created your project, you specified the location of the Word or Excel document that you wanted associated with the project. When Visual Studio .NET created the project, it set various properties associated with the project, including AssemblyLinkLocation and OfficeDocument properties (see Figure 4). The AssemblyLinkLocation property indicates where the Office document expects to find the assembly file needed to run your code, and the OfficeDocument property indicates where the Visual Studio .NET solution can find the associated Office document.

Figure 4 Project Properties
Figure 4 Project Properties

In addition, creating the project sets up some debugging capabilities. Starting the project from within Visual Studio .NET loads Excel or Word along with your Office document, and runs your code as necessary. In order for Visual Studio to be able to debug your application, it has to know what application to load. When you create your project, Visual Studio .NET retrieves the location of the appropriate Office application and stores that location along with necessary command-line parameters. If you take a look at Figure 5 you'll see the project property pages, including both the application location and command-line arguments.

Figure 5 Set Location Properties
Figure 5 Set Location Properties

Once you've started the debugging session (or simply loaded the Office document from Windows Explorer), how does the document know which assembly to load and what code to run? Unbeknownst to users, Visual Studio .NET has inserted two custom properties into the document you specified when you created the project. With your document loaded in Excel, you can select File | Properties to display the dialog box shown in Figure 6. On the Custom tab you'll find two properties named _AssemblyName0 and _AssemblyLocation0. These properties contain the path and file name of the associated assembly. By default, the assembly will be located in a folder with the same name as the project, with "_bin" appended. For example, if your project name is ChartDemo, the default location for the assembly will be the ChartDemo_bin folder.

Figure 6 ChartDemo Dialog
Figure 6 ChartDemo Dialog

Clearly, there's some piece of code that causes Excel or Word to load and run the assembly specified in the custom document properties. At document load time, if Excel (or Word) finds these special document properties, it fires up the unmanaged loader DLL for Visual Studio Tools for Office, OTKLOADR.DLL, which loads and links your managed assembly.

Once the host Office application has loaded the assembly you've created, how does it know what code to run? That's where the assembly attribute discussed previously comes into play—this attribute describes the fully qualified name of the class containing your Visual Basic .NET or C# code.

Once it has loaded the appropriate assembly, OTKLOADR. DLL passes the procedure in your class named _Startup to the CLR for execution. You don't create this procedure—the project template creates it and places it in the Generated initialization code region. Although you could modify this procedure, you probably should not. The loader also passes to _Startup a reference to the application (either the Word or Excel Application object) and a reference to the document itself (an instance of either Excel.Workbook or Word.Document). The _Startup method copies this information into local variables and hooks up any necessary event handling, as shown in Figure 7.

Figure 7 _Startup Method (Close Figure 7)

Visual Basic .NET
' Required procedure. Do not modify.
Public Sub _Startup(ByVal application As Object, _
 ByVal workbook As Object)
  ThisApplication = CType(application, Excel.Application)
  ThisWorkbook = CType(workbook, Excel.Workbook)
End Sub
C#
// Required procedure. Do not modify.
public void _Startup(object application, object workbook)
{
  this.thisApplication = application as Excel.Application;
  this.thisWorkbook = workbook as Excel.Workbook;

  openEvent =
    new Excel.WorkbookEvents_OpenEventHandler(ThisWorkbook_Open);
  thisWorkbook.Open += openEvent;

  beforeCloseEvent =
    new Excel.WorkbookEvents_BeforeCloseEventHandler(
    ThisWorkbook_BeforeClose);
  thisWorkbook.BeforeClose += beforeCloseEvent;
}

The region also contains a public procedure named _Shutdown, which gets called automatically by the loader as the application closes the document:

' Visual Basic .NET
' Required procedure. Do not modify.
Public Sub _Shutdown()
  ThisApplication = Nothing
  ThisWorkbook = Nothing
End Sub

// C#
// Required procedure. Do not modify.
public void _Shutdown()
{
  thisApplication = null;
  thisWorkbook = null;
}

From then on, your assembly uses the standard COM event marshaling mechanisms to react to events raised by the Word and Excel objects. Note that there's no defined interaction between VBA event handlers and managed event handlers. If your existing Word or Excel document includes VBA code that handles an event, adding Visual Basic .NET or C# code that handles the same event will work just fine, but you can't be sure of the order in which the event handlers will run. Make sure your applications don't rely on a particular order.

Of course, one of the main benefits of using managed code is the support for code access security, and code you write in your assembly takes advantage of the user's security policies. Although Visual Studio .NET created an appropriate security policy for your assembly when you created your project (granting full rights), this won't be the case when you deploy your solution. You have many options when distributing solutions created using Visual Studio Tools for Office. You can deploy both the Office document and the assembly to the client computer. You can deploy the Office document to the local computer and host the assembly on a shared location. You can even deploy both the Office document and the assembly to a shared location.

Each of these options has its pros and cons, including the ability to update the document and/or assembly when they're hosted in a shared location. In any case, when deploying solutions created using Visual Studio Tools for Office, you'll need to take into account the requirements of code access security, and allot time in your deployment schedule for setting up and testing various security scenarios.

Completing the Demonstration

Follow these seven steps to retrieve data from SQL Server™, add it to a worksheet, and then chart the data:

  1. Run the project. Within Excel, right-click on the main menu bar, and select the Control Toolbox toolbar.
  2. From this toolbar, select and draw a CommandButton control on the worksheet, covering cells A3 to A4.
  3. On the Control Toolbox, select the Properties button. In the Properties window, set the CommandButton control's Name property to cmdChart and its Caption property to Create Chart.
  4. On the Control Toolbox, deselect the Design Mode tool, placing the control in run mode. Close the Control Toolbox window. When you're finished, the worksheet should look like Figure 8.
  5. Close Excel, saving the changes to your workbook.
  6. Back in Visual Studio .NET, scroll to the top of the current file, and add the appropriate statement:
    ' Visual Basic .NET
Imports System.Data
Imports System.Data.SqlClient

// C#
using System.Data
using System.Data.SqlClient;
  7. At the bottom of the current class, add the procedure that is shown in Figure 9. Then modify the value of the connectionInfo variable to reflect the location and security settings of your SQL Server installation.

Figure 9 RetrieveData (Close Figure 9)

Visual Basic .NET
Private Sub RetrieveData()
  ' Modify this string for your own situation.
  Dim connectionInfo As String = _
   "Server=.;Database=Northwind;Integrated Security=True"
  Dim commandText As String = "[Ten Most Expensive Products]"

  Dim connection As SqlConnection
  Dim command As SqlCommand
  Dim dataReader As SqlDataReader
  Try

    connection = New SqlConnection(connectionInfo)
    command = New SqlCommand(commandText, connection)
    command.CommandType = CommandType.StoredProcedure

    connection.Open()
    dataReader = _
     command.ExecuteReader(CommandBehavior.CloseConnection)

    Dim rng As Excel.Range = ThisApplication.Range("A5")
    Dim i As Integer

    rng.CurrentRegion.Clear();
    While dataReader.Read
      rng.Offset(i, 0).Value = dataReader(0)
      rng.Offset(i, 1).Value = dataReader(1)
      i += 1
    End While

  Catch ex As Exception
    MessageBox.Show(ex.Message)

  Finally
    If Not dataReader Is Nothing Then
      dataReader.Close()
    End If

    If Not command Is Nothing Then
      command.Dispose()
    End If

    If Not connection Is Nothing Then
      connection.Dispose()
    End If
  End Try
End Sub
C#
private void RetrieveData()
{
  // Modify this string for your own situation.
  string connectionInfo =
    "Server=.;Database=Northwind;Integrated Security=true";
  string commandText = "[Ten Most Expensive Products]";

  SqlDataReader dataReader = null;

  using (SqlConnection connection =
    new SqlConnection(connectionInfo))
  {
    using (SqlCommand command =
      new SqlCommand(commandText, connection))
    {
      try
      {
        command.CommandType = CommandType.StoredProcedure;

        connection.Open();
        dataReader =
          command.ExecuteReader(CommandBehavior.CloseConnection);

        Excel.Range rng = ThisApplication.
          get_Range("A5", Type.Missing);
        int i = 0;

        rng.CurrentRegion.Clear();
        while (dataReader.Read())
        {
          rng.get_Offset(i, 0).Value2 = dataReader[0];
          rng.get_Offset(i, 1).Value2 = dataReader[1];
          i += 1;
        }
      }
      catch (Exception ex)
      {
        MessageBox.Show(ex.Message);
      }
      finally
      {
        if (dataReader != null)
        {
          dataReader.Close();
        }
      }
    }
  }
}
Figure 8 CommandButton
Figure 8 CommandButton

Besides the standard ADO.NET code, the RetrieveData procedure clears all cells around cell A5, then loops through the rows returned by the SqlDataReader object, and inserts each of the two columns into cells in the worksheet. Note again the difference between the Visual Basic .NET and C# code here—the Offset property expects two parameters, and because C# cannot handle parameterized properties, the Excel PIA provides the get_Offset method that C# developers must use.

Your next task is to hook up an event handler for the CommandButton control embedded on the worksheet. That control is part of the Microsoft Forms package—a COM-based set of controls used by most of the Office products. In order to make it easier for you to hook up event handlers to the controls in this package, the Visual Studio .NET project template includes two overloaded versions of a FindControl method (one version accepts just the control name; the other accepts the control name and a document reference, in case the control isn't on the current sheet). You'll call this method to retrieve a reference to a control on a document.

Although the final release of Visual Studio Tools for Office will take care of the following details for you, if you're working with the public beta of the product, you may need to take a few extra steps in order to work with controls provided by the Microsoft Forms package. If you see an Imports/using statement referring to Microsoft Forms at the top of your module (indicating that you're working with a late beta or final release of the product), skip ahead a bit to the steps that describe adding a variable named button. Otherwise, add a reference to the Microsoft Forms 2.0 COM library in the Visual Studio .NET Solution Explorer window. (You may find two references to the library in the list. If so, select the first one.) Click Select and then click OK to dismiss the dialog box.

Scroll to the top of the code file, and add the following statement, depending on your chosen language (Visual Basic .NET or C#):

' Visual Basic .NET
Imports MSForms = Microsoft.Vbe.Interop.Forms

// C#
using MSForms = Microsoft.Vbe.Interop.Forms;
Add the following declaration immediately following the declarations for the ThisApplication and ThisWorkbook variables:
' Visual Basic .NET
Private WithEvents button As MSForms.CommandButton

// C#
private MSForms.CommandButton button;
Next, add the following procedure to the class:
' Visual Basic .NET
Private Sub button_Click() Handles button.Click
  RetrieveData()
End Sub

// C#
private void button_Click()
{
  RetrieveData();
}
Add this code to the class's ThisWorkbook_Open procedure:
' Visual Basic .NET
button = DirectCast(FindControl("cmdChart"), _
    MSForms.CommandButton)

// C#
button = (MSForms.CommandButton) FindControl("cmdChart");
button.Click +=new
  MSForms.CommandButtonEvents_ClickEventHandler(button_Click);

Save and then run the project. Verify that clicking the button on the worksheet loads data from SQL Server and displays it on the worksheet, as shown in Figure 10. When you're finished, close Excel. You can save the workbook if you like.

Figure 10 Display Data
Figure 10 Display Data

Back in Visual Studio .NET, add the procedure in Figure 11 to the class, which will delete the chart if it already exists. As you can see by investigating this code, many methods in Excel (and Word) return Object types, meaning that you'll need to carefully cast the return values to the correct types. (You can avoid this hassle in Visual Basic .NET by not using Option Strict On, but that's not recommended. It makes it just too easy to create code that compiles but doesn't run correctly.)

Figure 11 Delete Chart (Close Figure 11)

Visual Basic .NET
Private Sub DeleteExistingChart()
  Try
    Dim ws As Excel.Worksheet = _
     DirectCast(ThisApplication.Worksheets(1), Excel.Worksheet)
    Dim charts As Excel.ChartObjects = _
     DirectCast(ws.ChartObjects, Excel.ChartObjects)

    If charts.Count > 0 Then
      DirectCast(charts.Item(1), Excel.ChartObject).Delete()
    End If
  Catch ex As Exception
    MessageBox.Show(ex.Message, ex.Source)
  End Try
End Sub
C#
private void DeleteExistingChart()
{
  try
  {
    Excel.Worksheet ws =
      (Excel.Worksheet)ThisApplication.Worksheets[1];
    Excel.ChartObjects charts =
      (Excel.ChartObjects)ws.ChartObjects(Type.Missing);

    if (charts.Count > 0)
    {
      ((Excel.ChartObject)charts.Item(1)).Delete();
    }
  }
  catch (Exception ex)
  {
    MessageBox.Show(ex.Message, ex.Source);
  }
}

Next, add the procedure in Figure 12 to the class, which will create the chart. Note how different the C# code is from the corresponding Visual Basic .NET code. Several of the Excel methods accept optional parameters which, as we discussed earlier, C# can't support. To work around this limitation, the C# code must pass the Type.Missing field for each optional parameter that's a reference type. (For value-type optional parameters, C# code must pass the actual default value.) In the DeleteExistingChart method, the WorkSheet.ChartObjects method lets you specify either an index of the chart you want to retrieve or no parameter to retrieve a ChartObjects collection. In C#, to retrieve the entire collection, you must pass Type.Missing for the optional parameter. The same rule applies when calling the Charts.Add method in the CreateChart method. In general, for any method that requires optional reference-type parameters, C# developers can pass Type.Missing.

Figure 12 Create Chart (Close Figure 12)

Visual Basic .NET
Private Sub CreateChart()
  Try
    DeleteExistingChart()

    Dim cht As Excel.Chart = _
     DirectCast(ThisApplication.Charts.Add(), Excel.Chart)
    cht.ChartType = Excel.XlChartType.xl3DBarClustered
    cht.Location(Excel.XlChartLocation.xlLocationAsObject, _
     "Sheet1")

    cht = ThisApplication.ActiveChart
    cht.HasLegend = False
    cht.SetSourceData( _
     ThisApplication.Range("A5").CurrentRegion, _
     Excel.XlRowCol.xlColumns)
    DirectCast(cht.ChartGroups(1), Excel.ChartGroup). _
     VaryByCategories = True

  Catch ex As Exception
    MessageBox.Show(ex.Message, ex.Source)
  End Try
End Sub
C#
private void CreateChart()
{
  try
  {
    DeleteExistingChart();

    Excel.Chart cht =
      (Excel.Chart)ThisApplication.Charts.Add(
      Type.Missing, Type.Missing, Type.Missing, Type.Missing);
    cht.ChartType = Excel.XlChartType.xl3DBarClustered;
    cht.Location(Excel.XlChartLocation.xlLocationAsObject,
      "Sheet1");

    cht = ThisApplication.ActiveChart;
    cht.HasLegend = false;
    cht.SetSourceData(ThisApplication.
      get_Range("A5", Type.Missing).CurrentRegion,
      Excel.XlRowCol.xlColumns);
    ((Excel.ChartGroup)cht.ChartGroups(1)).VaryByCategories = true;
  }
  catch (Exception ex)
  {
    MessageBox.Show(ex.Message, ex.Source);
  }
}

Modify the Click event handler for the CommandButton control, adding a call to the CreateChart procedure:

' Visual Basic .NET
Private Sub button_Click() Handles button.Click
  RetrieveData()
  CreateChart()
End Sub

// C#
private void button_Click()
{
  RetrieveData();
  CreateChart();
}

Finally, save and run the project, click the button, and verify that you've managed to load the data and create the chart. Quit Excel when you're finished, saving the workbook if you like.

If you followed this procedure carefully, you should have created a chart that looks like Figure 13. Along the way, you saw how to take advantage of several of the Excel objects and how to hook up events to controls embedded on documents. You also saw some of the differences involved in coding Office objects from Visual Basic .NET and from C#. Although this example didn't take advantage of menus and toolbars available in Office, you can work with these items programmatically from within your managed code.

Figure 13 Bar Chart
Figure 13 Bar Chart

Of course, you haven't seen how code access security affects your project's ability to run. When you created your project, Visual Studio .NET added a code access policy that allowed your assembly to run from its current location. Copying it anywhere else on your hard drive should cause the code to fail. To verify the behavior of code access security, go to Visual Studio .NET and change the Assembly Link Location property to any other location. Now rebuild your application and run the project. You'll find that the code compiles and Visual Studio .NET loads Excel along with your workbook, but the code won't run. Figure 14 shows the alert you'll receive. Reset the Assembly Link Location property and repeat the previous steps to verify that your code runs again once the folder name has been reset.

Figure 14 Alert Message
Figure 14 Alert Message

Code access security ensures that you cannot run your assembly from any location other than those that have been granted permissions to run. By default, Visual Studio .NET sets up a code access policy that allows your code to run from its development location. Change the location and your code won't run.

A full discussion of the deployment and security options, which are inexorably intertwined, would take a complete article on its own. For now, be aware that in order to deploy your applications to other users' computers, you'll need to define a security policy that allows your code to run. The behavior is the same for assemblies created using Visual Studio Tools for Office as it is for assemblies downloaded from the Internet or an intranet—the assembly is downloaded to the user's download cache and run from there. The same security issues apply for both types of assemblies.

Using Visual Studio Tools for Office will require some knowledge of the Office object models, a good knowledge of .NET security and deployment issues, and a business need. Finally, developers will be able to take advantage of all the power of the CLR and the .NET Framework from within Office applications. These new applications will look to users as if they were simply running Word or Excel. Updates can be performed automatically using no-touch deployment, and untrusted code can't run without an administrator's knowledge. These are just some of the features that provide the power, security, and flexibility of Visual Studio Tools for the Microsoft Office System.

For related articles see:
Installation and New Project QuickStarts for Visual Studio Tools for Office Beta
Visual Studio Tools for Office Frequently Asked Questions
Visual Studio Tools for Office
Understanding the Excel Object Model from a .NET Developer's Perspective
Understanding the Word Object Model from a .NET Developer's Perspective

For background information see:
Programming Microsoft Word 2002 and Excel 2002 with Microsoft Visual C#
Working with the Office XP Primary Interop Assemblies
Managed COM Add-In Shim

Ken Getz is a senior consultant with MCW Technologies. He is coauthor of ASP.NET Developer's Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2001), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.

Brian A. Randell is a senior consultant with MCW Technologies and an instructor for DevelopMentor. Brian speaks regularly worldwide at VSLive! and Tech•Ed. He is the coauthor of Effective Visual Basic (Addison-Wesley, 2001).

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


InfoPath: Turn User Input into XML with Custom Forms Using Office InfoPath 2003

InfoPath

Turn User Input into XML with Custom Forms Using Office InfoPath 2003

Aaron Skonnard
This article assumes you're familiar with Microsoft Office and XML
Level of Difficulty 1 2 3
SUMMARY

Office InfoPath 2003 is a new Microsoft Office product that lets you design your own data collection forms that, when submitted, turn the user-entered data into XML for any XML-supporting process to use. With an InfoPath solution in place, you can convert all those commonly used paper forms into Microsoft Office-based forms and end the cycle of handwriting and reentering data into your systems. Today organizations are beginning to realize the value of the mountains of data they collect every day, how hard it is to access it, and are striving to mine it effectively. InfoPath will aid in the design of effective data collection systems. Here the author shows you how to get started.

Contents
InfoPath Overview
Designing Form Templates
Starting from XML Schema or WSDL
Saving and Publishing Forms
Filling Out Forms
Submitting Forms
Advanced Features
Summing it Up

O
rganizations depend on information to drive business processes and decision-making. The quality and accuracy of this information is key, as is the ability to gather and analyze it quickly.

Information is recorded in a variety of formats, the most primitive (and probably the most common) is a paper form. Just think of all those expense reports, customer evaluations, sales trip reports, and time cards. Forms may be easy to fill out, but eventually the data collected in them must be rekeyed into the computer system (into yet another data format) so it can be used by internal business processes, risking redundancy and human error, both of which reduce productivity.

Word processing documents like those created in Microsoft® Word or Excel are also common. Working with these formats is challenging as well because it's difficult to extract the information and metadata from them programmatically.

The flexibility of XML allows organizations to define their own XML schemas for handling their specific data representation needs. With well-known XML schema definitions in place, any business process can be programmed to consume an XML document and understand its meaning thanks to XML support in virtually all platforms and programming languages.

The Web Services platform builds on XML by using it for information exchange over protocols like TCP, HTTP, SMTP, and potentially many others. Combining XML with these open protocols makes it possible to build an infrastructure for sharing information between business processes in a standard way.

All that is needed to reap the benefits across the enterprise is an easy way to get previously hand-written data into XML. InfoPath, previously known as XDocs, is a new member of the Microsoft Office System of products that let's you do just that.

InfoPath provides an environment for designing forms built around XML Schema or Web Services Description Language (WSDL) definitions. In a matter of seconds, you can use InfoPath to build a new form that's capable of outputting XML documents conforming to an XML Schema Definition (XSD) or communicating with a Web Service conforming to a WSDL definition. XML Web Services and InfoPath can be used together to replace their legacy information-gathering techniques.

InfoPath is chock-full of functionality, including rich client functionality and off-line capabilities that surpass those of traditional Web Forms. Best of all it's much easier to use than traditional Web Services development environments. This article will focus on the main features of InfoPath.

InfoPath Overview

InfoPath supports two main activities: designing form templates and filling out forms. You must have InfoPath installed on your machine to perform either task. When you first run InfoPath, you'll see a window like the one shown in Figure 1. Notice that both the File menu and task bar (on the right) provide an option for each of these main tasks.

Figure 1 InfoPath Environment
Figure 1 InfoPath Environment

InfoPath provides an easy-to-use WYSIWYG interface for designing new forms that can be based on XSD or WSDL. After you design a new form template, InfoPath allows you to publish it to a centralized location so others can access it. After a user browses to a form template, InfoPath opens the form and allows the user to fill it out. When the user is finished, the form can be saved or submitted for processing. The InfoPath save functionality lets you work with forms without being connected to the network, effectively providing an off-line mode. Submitting the form can include posting the XML file to a Web site directory or to a Web Service endpoint using SOAP.

Designing Form Templates

When you design a new form template, you can either start from scratch by selecting New Blank Form or base it on an existing data source by selecting New from Data Source (see Figure 1). The task list in the righthand pane of a blank form provides links to the various tasks you might want to perform while designing a new form. Specifically, you'll want to control the form layout, place controls on the form for capturing different types of information, define how those controls map to your underlying data source, maybe define different data views of the form, and ultimately publish it. After selecting New Blank Form, InfoPath provides a blank design surface that you can drag controls onto and arrange in a variety of ways to suit your needs.

Alternatively, you can choose from a collection of built-in layout options including various tables and sections. The designer also makes it easy to change color schemes, fonts, and other aesthetics as you're building your form and provides a nice palette of built-in controls that fit a wide range of data representation needs. As you're designing, you can select Preview Form to see what it will look like when the user fills it out.

As you design a form from scratch, InfoPath automatically builds an XML Schema definition behind the scenes to represent the information that will be captured by your form. You can explicitly define how each control maps to a corresponding XML Schema datatype by manipulating the layout and properties of each control placed on the form. For example, consider the form shown in Figure 2, which captures employee information.

Figure 2 New Employee Form
Figure 2 New Employee Form

When I designed this form, I defined the Name textbox to be of type xsd:string, the Salary textbox to be of type xsd:double, and the Date of birth textbox to be of type xsd:date. The schema produced by InfoPath to represent this information is shown in Figure 3. Unfortunately, you don't have much control over the schema when using this approach. The order in which you place the controls determines the hierarchy of the schema. Your only mechanisms for influencing the schema are the various layout and control properties available through the WYSIWYG designer.

Figure 3 Employee Form XML Schema Definition (Close Figure 3)
<xsd:schema
  targetNamespace="http://schemas.microsoft.com/office/
  infopath/2003/myXSD/2003-06-04T16:51:19"
  xmlns:my="http://schemas.microsoft.com/office/infopath/
  2003/myXSD/2003-06-04T16:51:19"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <xsd:element name="myFields">
      <xsd:complexType>
        <xsd:sequence>
          <xsd:element ref="my:name" minOccurs="0"/>
          <xsd:element ref="my:salary" minOccurs="0"/>
          <xsd:element ref="my:birthdate" minOccurs="0"/>
    </xsd:sequence>
        <xsd:anyAttribute processContents="lax"
          namespace="http://www.w3.org/XML/1998/namespace"/>
      </xsd:complexType>
    </xsd:element>
    <xsd:element name="name" type="xsd:string"/>
    <xsd:element name="salary" nillable="true"
      type="xsd:double"/>
    <xsd:element name="birthdate" nillable="true"
      type="xsd:date"/>
</xsd:schema>

If you fill out this employee form and save it to disk, InfoPath will produce an XML document that conforms to the schema, like the one shown in Figure 4. Although this start-from-scratch approach limits its use as an XML Schema designer, this was never intended to be its primary role. The fundamental purpose of InfoPath is to be able to dynamically build forms from already existing XML Schemas or WSDL definitions.

Figure 4 Saved Employee Form (Close Figure 4)
<?mso-infoPathSolution
   solutionVersion="1.0.0.5"
   productVersion="11.0.4920"
   PIVersion="0.9.0.0"
   language="en-us"
   href="http://file:///C:\Samples\SimpleEmployee.xsn"?>
<?mso-application progid="InfoPath.Document"?>

<my:myFields xmlns:my="http://schemas.microsoft.com/office/infopath/2003/
    myXSD/2003-06-04T16:51:19" xml:lang="en-us">
    <my:name>Bob</my:name>
    <my:salary>1000</my:salary>
    <my:birthdate>1965-12-31</my:birthdate>
</my:myFields>

Starting from XML Schema or WSDL

When you design a form from an existing data source, you're essentially defining a mapping between the existing data source and the form template. InfoPath supports the following data sources: XML documents, XSD, databases (SQL Server™ or Microsoft Access), and Web Services. When you select New from the Data Source menu, the Data Source Setup Wizard appears and prompts you to select the type of data source you want to use.

Regardless of the type of data source you ultimately decide to use in your project, InfoPath reads the available metadata from the data source and displays it in the Data Source view. You can then drag and drop data source fields onto the form as you're designing it. This approach allows the data source to drive the forms creation in a very dynamic way.

For example, consider the schema in Figure 5 which represents author information in a publication system. If you created a new form based on this schema, InfoPath would display all of the author schema fields in the data source view to the right of the design surface. You could then drag and drop the schema fields onto the form and arrange them as you like (see Figure 6).

Figure 5 Author Schema (Close Figure 5)
<xs:schema id="Author"
   targetNamespace="http://www.develop.com/authors"
   elementFormDefault="unqualified"
   xmlns:tns="http://www.develop.com/authors"
   xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
    <xs:simpleType name="SSNumber">
       <xs:restriction base="xs:string">
          <xs:pattern value="\d{3}\-\d{2}\-\d{4}" />
       </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="PhoneNumber">
       <xs:restriction base="xs:string">
          <xs:pattern value="\d{3}\-\d{4}" />
       </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="TwoLetterState">
       <xs:restriction base="xs:string">
          <xs:minLength value="2" />
          <xs:maxLength value="2" />
       </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="ZipCode">
       <xs:restriction base="xs:string">
          <xs:pattern value="\d{5}" />
       </xs:restriction>
    </xs:simpleType>

    <xs:complexType name="NameType">
       <xs:sequence>
          <xs:element name="first" type="xs:string" />
          <xs:element name="last" type="xs:string" />
       </xs:sequence>
    </xs:complexType>

    <xs:complexType name="AddressType">
       <xs:sequence>
          <xs:element name="street" type="xs:string" />
          <xs:element name="city" type="xs:string" />
          <xs:element name="state"
              type="tns:TwoLetterState" />
          <xs:element name="zip" type="tns:ZipCode" />
       </xs:sequence>
    </xs:complexType>

    <xs:complexType name="AuthorType">
       <xs:sequence>
          <xs:element name="name" type="tns:NameType" />
          <xs:element name="phone" type="tns:PhoneNumber" />
          <xs:element name="address"
              type="tns:AddressType" />
          <xs:element name="contract"
              type="xs:boolean" />
       </xs:sequence>
       <xs:attribute name="id" type="tns:SSNumber" />
    </xs:complexType>

    <xs:element name="author" type="tns:AuthorType" />
</xs:schema>
Figure 6 New Author Form
Figure 6 New Author Form

When you fill out the form and save it, InfoPath creates an XML document that conforms to the existing schema definition that you started from. In this case, it would create an XML document that looks something like the document shown in Figure 7.

Figure 7 Saved Author Form (Close Figure 7)
<?mso-infoPathSolution
   solutionVersion="1.0.0.5"
   productVersion="11.0.4920"
   PIVersion="0.9.0.0"
   language="en-us"
   href="http://file:///C:\Samples\NewAuthor.xsn"?>
<?mso-application progid="InfoPath.Document"?>

<tns:author id="333-33-3333"
  xmlns:tns="http://www.develop.com/authors">
    <name>
        <first>Mary</first>
        <last>Smith</last>
    </name>
    <phone>555-0444</phone>
    <address>
        <street>123 Main</street>
        <city>Layton</city>
        <state>UT</state>
        <zip>84041</zip>
    </address>
    <contract>true</contract>
</tns:author>

Designing a form for a Web Service follows the same process, but you must also specify the location of the Web Service and provide the WSDL definition. InfoPath extracts the schema definitions from the WSDL definition in order to build the data source view for you to work with. You must also indicate how the form will interact with the Web Service, indicating options such as whether it receives and submits data to the service. You can build a form that only sends data, one that only receives data, or one that does both.

If you choose "Receive and submit data," InfoPath will create two views for your form: one view for submitting data to the service and another view for the data returned from the service. In this case, the data source view also contains two groups of fields. The queryFields contain the data that needs to be supplied when invoking the service. The dataFields contain the data returned from the Web Service. You drag the queryFields onto your Query view and the dataFields onto your Data view. When the user fills out the Query view and presses Calculate, InfoPath invokes the Web Service with the supplied information and displays the result in the Data view.

Saving and Publishing Forms

Once you've designed a form template, you need to save and/or publish it. You should note that clicking the Save button simply saves the form template to a file. You can return to the form template at any time and continue working on it. The form template is a standalone file that contains all the information necessary for another user to fill it out. You can extract the various files that constitute a form by selecting File | Extract Form Files. This will write various files to your hard drive including the XML Schema and XSLT files used internally.

Selecting Publish displays the Publishing Wizard, which allows you to distribute the finished form to a centralized location, accessible by other users. You can publish a finished form to a number of places: a network share, a SharePoint form library, or a virtual directory on a Web server, all of which users can easily browse to using their Web browser.

Filling Out Forms

The whole purpose of designing a form is to have users fill it out. Once a form has been published, users can access the form directly from InfoPath (by selecting Fill Out a Form) or by simply browsing to the file in Windows® Explorer or Microsoft Internet Explorer. When you browse to a form, InfoPath opens it in "fill out" mode and enables the user to enter data (see Figure 8). As I mentioned earlier, users can save forms to the local hard drive and return to work on them later even if disconnected from the network, then submit them when reconnected.

Figure 8 Filling Out a Form
Figure 8 Filling Out a Form

A saved form is just an XML document that conforms to the schema or Web Service it was designed from. For example, the XML document shown in Figure 7 is the saved form shown in Figure 8. As you can see, InfoPath injects a few processing instructions (PIs) into the XML documents that allow the loader to figure out that this document should be loaded back into InfoPath, not into the default application for XML documents (usually Internet Explorer). When you double-click an XML file that contains these PIs, Windows automatically launches InfoPath and allows the user to continue filling out the saved form.

One of the main benefits of using an InfoPath form over a traditional Web form is the rich functionality offered by the runtime environment. For example, InfoPath provides automatic spell checking while you fill out a form, very much like you would find available in Word. If InfoPath detects a spelling error while you're entering data, it highlights the word with a red squiggle and offers suggestions for possible changes.

InfoPath offers a range of validation features to help ensure the quality of your data. It provides real-time validation against the form's underlying XML Schema definition including custom simple type definitions like those used in Figure 5. If the user enters a value that doesn't conform to the control's underlying XML Schema type, it's also highlighted with a red squiggle and a helpful error message is provided in the control's tooltip. Validation against the schema also happens at submission time.

InfoPath also offers other advanced features like autocomplete, find and replace, drag and drop, and complete printing support. Generally speaking, working in InfoPath feels just like you're working in any other Microsoft Office product.

Submitting Forms

The final step is to submit the filled-in form. This step depends somewhat on how the form was designed. In some cases, the designer may have only intended for the form to be saved and placed in a shared directory somewhere. However, the form will usually need to contain some kind of Submit button. You can create your Submit button by dragging a button control onto the form and double-clicking it, and setting its action to Submit in its Properties window. Doing so displays a dialog that allows you to specify what should happen when the user presses the button.

Figure 9 Sending a Form Via E-mail
Figure 9 Sending a Form Via E-mail

The form can be submitted to a Web Service, to a virtual directory on a Web server, or via custom script code. It's also possible to submit the completed form via e-mail by selecting File | Send to Mail Recipient. When you do this, InfoPath attaches the XML document to the e-mail and adds the HTML of the view you sent to the body of the e-mail (see Figure 9).

Advanced Features

InfoPath has a variety other advanced features that I'll cover only briefly. One such feature is the rich text control that can be used to input formatted text (such as text annotated with different fonts, bold, or italic). Rich text controls are mapped to XHTML in the underlying XML document. The rich text control makes it possible to enter free-form text, like you would in Word, without sacrificing the benefits of working with XML or losing the formatting information provided by the user.

Another advanced feature enables optional and repeating sections. Optional sections can be hidden and displayed on demand, making it possible to simplify forms and decrease clutter. Repeating sections (or repeating lists/tables) let you repeat blocks of information that occur multiple times in a schema. InfoPath also enables you to bind repeating controls to external data sources for automatic population of the control.

Figure 10 Creating a Validation Rule
Figure 10 Creating a Validation Rule

Finally, InfoPath provides more advanced validation features than XML Schema provides. Pressing the Data Validation button on the Properties page for any form control gives you two options for additional validation. One option is to write custom validation code using script to respond to one of three events: OnBeforeChange, OnValidate, and OnAfterChange. The other option is to create a new validation rule using a set of predefined expressions and constraints, as illustrated in Figure 10.

Summing it Up

InfoPath is a new Microsoft Office product that facilitates the process of gathering information from heterogeneous sources through dynamic XML-based forms. Overall, InfoPath makes it easy for anyone to design, publish, and fill out electronic forms based on XML and Web Services technology, which offers many advantages over traditional techniques used today. By using InfoPath, organizations will find it much easier to share their previously inaccessible data with any application that supports XML.

For background information see:
The XML Files: A Quick Guide to XML Schema
The XML Files: A Quick Guide to XML Schema-Part 2
Understanding XML Namespaces
Understanding XML Schema

Aaron Skonnard is an instructor and researcher at DevelopMentor, where he develops the XML and Web Service-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.


.NET Remoting: Create a Custom Marshaling Implementation Using .NET Remoting and COM Interop

.NET Remoting

Create a Custom Marshaling Implementation Using .NET Remoting and COM Interop

Jim Sievert
This article assumes you're familiar with the CLR, C#, and COM
Level of Difficulty 1 2 3
Code download available at: CustomMarshaling.exe (4,233KB)
SUMMARY

The .NET Framework offers several methods for customizing the presentation of native .NET and COM object types. One such technique, custom marshaling, refers to the notion of specializing object type presentations. There are times, like when a legacy COM component needs to implement a new interface or when you need to make calls across process or machine boundaries, when custom marshaling saves the day. Elements of COM Interop permit the customizing of COM types while .NET Remoting offers the developer the ability to tailor native .NET types. This article examines these techniques.

Contents
Remoting Proxies
Extending the RealProxy
An Object with a Different View
COM Custom Marshaling
Redefining the Existing Interface
Defining the Adapter Class
Defining the Custom Marshaler
A More Sophisticated Example
More Advanced COM Interop
Conclusion

S
ometimes a data object must be transmitted to a receiver that exists in a separate, predefined context, such as a machine, process, thread, class, or managed/unmanaged space. During this transmission, the data may need to undergo a transformation to make it suitable for use by the recipient. This process of converting a data object between types when sending it across contexts is commonly called marshaling.

Marshaling is employed by several technologies within the Microsoft® .NET Framework, including Remoting and COM Interop. Within Remoting and COM Interop, the .NET Framework allows programmers to tailor or customize client object presentations through the use of custom marshaling. For example, programmers typically think of implementing interfaces statically at compile time; however, occasionally you need a class that can dynamically implement the public interface of another referenced class instance. Such situations typically arise when performing method interception—intercepting method calls to a target object in order to perform some behavior completely orthogonal to the target object itself. Many scenarios probably come to mind. Method tracing, authentication, authorization, and transaction semantics are just some examples. You can probably think of some other implementations, such as COM+ and the Universal Delegator. With the .NET Remoting technology, you can implement dynamic public interfaces through the use of specialized proxies.

Similarly, it may be useful in some .NET-based client applications that use COM Interop to change or extend the public interface of legacy or third-party COM components. This scenario can occur when a legacy COM component doesn't or can't implement a newer interface used by a .NET-based client. Extended interface support for these client apps can be accomplished by using the COM custom marshaling capabilities in the .NET Framework.

Finally, there are advanced cases in which a combination of extended .NET Remoting proxies and COM custom marshaling is in order. These situations exemplify the power and flexibility of the .NET Framework application environment. This article examines some of the advanced aspects of the customized marshaling capabilities available in .NET Remoting and COM Interop.

Remoting Proxies

Of all technologies that the .NET Framework encompasses, Remoting ranks as one of the most interesting, challenging, and flexible. Remoting is itself comprised of many individual technologies such as contexts, messages, channels, proxies, sinks, and configuration. Of all of these concepts, the flexibility that Remoting proxies provide might get the least attention, even though coverage of the topic of Remoting proxies is quite extensive.

As its name implies, a Remoting proxy is a stand-in for a remote object. To a caller, it looks and acts like the actual remote object. A Remoting proxy hides all of the details of transporting and executing a method call issued from a caller to the remote object.

For example, in an inter-AppDomain Remoting scenario, one that perhaps crosses machine boundaries, a Remoting proxy is the first step towards getting a method call forwarded to the remote object through the Remoting channel. The Remoting proxy obscures all details of such behavior from the caller. For instance, in Figure 1 the caller views method calls being executed directly on the target object. These method calls are actually executed against a proxy object, not the original. The proxy hides the underlying details of the method-call transport mechanism across message sinks and the channel to the remote object itself.

Figure 1 Marshal By Reference
Figure 1 Marshal By Reference

In the Microsoft .NET Framework there are two classes that work together to form the Remoting proxy: System.Runtime.Remoting.Proxies.TransparentProxy and System.Runtime.Remoting.Proxies.RealProxy. The client of a remote object instance always communicates with that remote object through an instance of a TransparentProxy, which gives the caller the appearance of the target object.

The TransparentProxy instance presents the interface of the remote object through close cooperation with a peer object—an instance of the RealProxy class. The RealProxy class is the source of information about and communication with the actual remote object, but it's an abstract class that must be extended. This override allows a derived class to properly configure the RealProxy. TransparentProxy, on the other hand, cannot be extended and derives its operation strictly from information obtained from and services provided by the associated RealProxy. A caller obtains an instance of the TransparentProxy class from the RealProxy in which the TransparentProxy is contained.

Extending the RealProxy

The following code defines the three basic elements that are involved in extending the RealProxy class: the constructor, the virtual CreateObjRef method override, and the abstract Invoke method override:

public abstract class RealProxy
{
    protected RealProxy( Type classToProxy );
    public virtual ObjRef CreateObjRef( Type requestedType );
    public abstract IMessage Invoke( IMessage message );
    •••
}

The key to extending a RealProxy is specifying the type of object that the resulting TransparentProxy will represent. A RealProxy-derived class accomplishes this type specification through the protected constructor of the RealProxy class. This constructor takes a single parameter, a System.Type instance called classToProxy. This parameter represents the type or class of object that is to be proxied. That is, the TransparentProxy will customize its behavior to act as an instance of the type specified in the classToProxy parameter. While the constructor of the RealProxy takes a generic System.Type object as a parameter, class ToProxy must be one of two permissible types or the RealProxy constructor will throw a System.ArgumentException.

The first permissible System.Type argument to the RealProxy constructor is an instance of a System.MarshalByRefObject type. MarshalByRefObject is the base class for any and all objects that can be marshaled by reference from one AppDomain to another. It goes without saying that remotely referenced objects require proxy support. This is especially true given all the implementation details necessary to get a method call from one AppDomain to another, as illustrated in Figure 1.

In general, the TransparentProxy for a System.MarshalByRefObject will appear to a caller as an actual instance of the remote type. In the case of remotely referenced objects, this makes perfect sense. Passing an instance of the System.MarshalByRefObject type into the RealProxy constructor introduces some subtle concerns, including propagation of the proxy to the new AppDomain and the possible performance implications of this action.

The second permissible type of RealProxy constructor argument is an interface. In this case, the resulting TransparentProxy will appear to a caller as an object implementing the interface type specified in the RealProxy constructor. Such a caller view presents some interesting implementation possibilities in extending the RealProxy class. The first obvious possibility is that this new public view can be applied to objects whose type is not derived from System.MarshalByRefObject. The other notable implementation possibility is that a TransparentProxy can expose a completely different public view than that exposed by the actual target class. This article will expand further on these implementation possibilities in subsequent sections.

The next task in extending a RealProxy is overriding the CreateObjRef method. The .NET Framework calls CreateObjRef whenever it attempts to remote an object reference. Considering only the case where the RealProxy constructor argument is an interface type, CreateObjRef should be overridden as follows:

public override ObjRef CreateObjRef( Type requestedType )
{
   throw new NotSupportedException();
}

As emphasized here, a class implementing the interface type exposed by the TransparentProxy does not necessarily inherit from System.MarshalByRefObject. Remoting of such a class is not possible. As a result, the creation of an ObjRef is not supported.

The final task in extending a RealProxy, which must be done for all derived classes, is to override the abstract Invoke method. Invoke initiates the transmission of a method call to a remote object. The type of this transmission may vary depending on the particular remoting situation. For example, when remoting a method call between AppDomains, Invoke transmits the method call through a Remoting channel. In other cases, a method call can reach the target object through other means, such as direct reflection.

The TransparentProxy forwards the actual method being called to the overridden Invoke method on a derived RealProxy implementation. The form or representation of an actual method call is that of a Remoting message. In the case of Invoke, a Remoting message contains all the information necessary to communicate a method call to a remote object, including the actual method being called. The goal of the Invoke implementation is to transmit the actual method call message to the remote object, receive the results, and form a return message that encompasses the return value and modified parameter values of the method call.

The code segment that is shown in Figure 2 combines all of the elements necessary to form a RealProxy-derived class called MyProxy. When using the techniques described here, this code segment provides a general template for the extension of the RealProxy class. You should note that the code segment contains a public static method called Marshal from which a caller can obtain a TransparentProxy instance that wraps a specified object instance. The Marshal method removes the burden of callers having to manually obtain the TransparentProxy instance by dealing directly with the derived RealProxy.

Figure 2 RealProxy Class Derivation (Close Figure 2)
internal class MyProxy : RealProxy
{
   public static IMySpecial Marshal( object instance )
   {
      MyProxy myProxy = new MyProxy( instance );

      return myProxy.GetTransparentProxy();
   }

   private MyProxy( object instance )
      :
      base( typeof( IMySpecial ) )
   {
      m_instance = instance;
   }

  public override ObjRef CreateObjRef( Type requestedType )
  {
     throw new NotSupportedException();
  }

      public override IMessage Invoke( IMessage message )
      {
         // Process message and form a return message, the details
         // of which are ignored for now.
         IMessage returnMessage = ...;

         return returnMessage;
      }

   private object m_instance;
}

The name of the static method, Marshal, is also significant. Calling the Marshal method represents a context boundary in which a caller's view of the original object may be different from that view presented by the resulting object. Herein lies one aspect of custom marshaling within .NET Remoting.

Because it contains the public static Marshal method, the constructor of MyProxy is private. Additionally, the constructor takes one parameter, an object instance against which the derived proxy will operate in the overridden Invoke method. The constructor of MyProxy initializes the base RealProxy with the interface type, IMySpecial. This action results in the TransparentProxy instance supporting the interface IMySpecial.

I've left out the details of the Invoke method for the time being, but I'll cover them in subsequent samples. It should be apparent to you that method call interception on the client or caller side begins in a RealProxy-derived class. The remainder of this article builds on this method call interception capability that is inherent in RealProxy-derived classes.

An Object with a Different View

As previously described, constructing a RealProxy with an interface type results in a TransparentProxy presenting that interface as the public view of the underlying object. What's more, this public view may be different than that of the underlying object. Imagine for a moment the possibilities. A RealProxy-derived class can intercept one method and potentially forward that call to the target object as another, entirely different method. A further possibility is that the RealProxy-derived class can intercept a method call and implement it locally without forwarding it to the target object at all. It is also entirely possible to swap out the target object held by the RealProxy-derived object while allowing current clients to hold onto the existing proxy.

A RealProxy-derived class can take its stealthiness one step further by implementing the System.Runtime.Remoting.IRemotingTypeInfo interface. When implemented by a RealProxy-derived class, IRemotingTypeInfo provides additional type information to the TransparentProxy. More specifically, the TransparentProxy calls IRemotingTypeInfo.CanCastTo to determine whether or not a caller can cast to a particular interface type.

The decision of whether or not to allow specific casts depends on the RealProxy-derived class. A RealProxy-derived class can allow a cast to any interface, intercept a method call on that interface, and direct that method call to another method on a completely different interface. Furthermore, a RealProxy-derived class could disallow some casts and restrict the methods that are accessible on the target object.

It should be noted that the capabilities I just outlined apply to both remotely referenced classes (those that derive from System.MarshalByRefObject) and non-remotable types.

The sample in Figure 3 wraps a target object in an Interposer. It follows the basic template for extending RealProxy, as shown in Figure 2. The TransparentProxy obtained from an Interposer supports the IInterposed interface, which allows a caller to retrieve the target object instance. The Interposer class implements the IInterposed interface locally within the Invoke method. Also, the implementation of IRemotingTypeInfo ensures that all caller-issued interface cast attempts against a TransparentProxy of an Interposer succeed only if the target object implements them. Even though I've already described some method interception implementations, the sample in Figure 3 forwards method calls to the target object using reflection. This call will either succeed or fail with the results returned. The Invoke method implementation in Figure 3 can be expanded to implement some of the more sophisticated method interception capabilities described previously.

Figure 3 Interposer Sample (Close Figure 3)
// This interface is supported directly by the Interposer class itself.
// This interface allows a caller to retrieve the object instance on
// which the Interposer performs method invocation through reflection.
public interface IInterposed
{
   object Target
   {
      get;
   }
}

public class Interposer : RealProxy, IRemotingTypeInfo
{
   // Standard Marshal method that returns a TransparentProxy instance
   // that can act as ANY interface.
   public static object Marshal( object target )
   {
      Interposer interposer = new Interposer( target );

      return interposer.GetTransparentProxy();
   }

   // The basic constructor initializes the RealProxy with the
   // IInterposed interface type and saves away the target instance.
   private Interposer( object target )
      :
      base( typeof( IInterposed ) )
   {
      m_target = target;
      m_targetType = m_target.GetType();
   }

   public override ObjRef CreateObjRef( Type requestedType )
   {
      // We don't support marshaling objects wrapped in a
      // Interposer, mainly because objects wrapped
      // by this proxy may not be marshalable.  That is they
      // may not extend System.MarshalByRefObject.
      throw new NotSupportedException(
         "Remoting of an interposed object is not supported." );
   }

   public override IMessage Invoke( IMessage message )
   {
      // The actual message coming in at this point should be
      // an IMethodCallMessage.  The MethodCallMessageWrapper
      // holds onto the arguments within the message for the
      // duration of the Invoke call.
      IMethodCallMessage methodMessage =
         new MethodCallMessageWrapper( (IMethodCallMessage) message );

      // Obtain the actual method definition that is being called.
      MethodBase method = methodMessage.MethodBase;

      object returnValue = null;

      // Handle IInterposed methods/properties here.  Right now, there is
      // only one property, so don't do any extensive checking to
      // determine the method or property being invoked.
      if ( method.DeclaringType == typeof( IInterposed ) )
      {
         returnValue = m_target;
      }
      else
      {
         returnValue = method.Invoke( m_target, methodMessage.Args );
      }

      // Finally, form a return message containing the return
      // value along with any and all parameters that may have been
      // modified by the method call (those that are out or ref).
      IMessage returnMessage =
         new ReturnMessage( returnValue, methodMessage.Args,
            methodMessage.ArgCount, methodMessage.LogicalCallContext,
            methodMessage );

      return returnMessage;
   }

   public bool CanCastTo( Type type, object o )
   {
      // Allow a cast to IIinterposed or to any type supported by the
      // underlying target object.
         bool valid =
            type == typeof( IInterposed ) ||
              type.IsAssignableFrom( m_targetType );

      return valid;
   }

   public string TypeName
   {
      get
      {
         throw new NotSupportedException();
      }

      set
      {
         throw new NotSupportedException();
      }
   }

   private object m_target;
   private Type m_targetType;
}

COM Custom Marshaling

COM custom marshaling allows you to customize the client view of objects across the managed/unmanaged code boundary. In his information-packed tome on COM Interop, .NET and COM: The Complete Interoperability Guide (SAMS, 2002), Adam Nathan devotes an entire chapter to the topic of COM custom marshaling. While this article is not meant to rehash the coverage contained in Adam's book, it may be helpful to summarize the premise behind COM custom marshaling.

In general, the .NET Framework COM Interop layer hides all that is COM. The Framework wraps COM objects that are accessed from the managed environment in what's called a runtime-callable wrapper (RCW). From the point of view of a .NET caller, an RCW makes a COM object look like any other .NET-based object. An RCW, in essence, acts as a proxy to a COM object. In a similar way, the .NET Framework wraps .NET-based objects in something called a COM-callable wrapper (CCW) to make them accessible to COM clients. The CCW exposes the public interfaces implemented on a given .NET object.

COM custom marshaling allows a programmer to tailor the view presented by an RCW or CCW. Such customization can be useful in situations where a programmer wants to enhance or change object functionality without affecting legacy code. As was the case when examining Remoting proxies, customized RCWs and CCWs can provide behavior completely orthogonal to the target object. Furthermore, COM custom marshaling permits customization of RCWs and CCWs at the method level, so customization can vary depending on the particular method called.

As an example, consider a legacy, COM-based arithmetic library. The library supports a number of calculator-like classes. One of these classes is an Adder, which returns the sum of two integers. The following IDL-based interface describes the function of the Adder:

[
   object,
   uuid(A67A7777-B16E-45D3-953B-F5915E3EBFBB),
   oleautomation
]
interface IAdder : IUnknown
{
   HRESULT Add( [in] int a1, [in] int a2, [out, retval] int *sum );
};

A COM client of the arithmetic library obtains an instance of an Adder through a calculator factory instance. The calculator factory has an IDL-based interface like the following:

[
   object,
   uuid(0A4043BF-D9D0-47EB-AC23-4D5E62154A7C),
   oleautomation
]
interface ICalculatorFactory : IUnknown
{
   HRESULT CreateAdder( [out, retval] IAdder **adder );
};

Now suppose you've been asked to change the Adder function itself when called via the common language runtime (CLR)—instead of returning the sum of two integers, the Adder is to return the sum for a vector of integers using the legacy COM library. In addition, suppose that a further requirement suggests that a change to the legacy library is not permitted.

You can implement the new Adder class through the use of COM custom marshaling, creating a customized RCW for the Adder. The implementation requires you to redefine the existing interface, define an Adapter class, and define the custom marshaler. In the following sections, I will describe these steps in detail.

Redefining the Existing Interface

The requirements for the new Adder class specify that an instance must return the sum for a vector of integers. So, given an instance of the calculator factory, the programmer's job is to provide a caller with an Adder whose public interface accepts an array of integers and returns the sum as an integer. Such a change to the calculator factory interface flies in the face of the legacy calculator factory interface provided by the type library import process.

The process of importing a type library for the legacy COM-based calculator library generates a COM Interop assembly. In this example, the COM Interop assembly is called CalculatorLibraryLib. This assembly provides a .NET-based client with all the metadata required to instantiate and use objects within the legacy library. However, the Adder returned by the calculator factory simply returns the sum of two integers. The first change requires that the calculator factory interface provide an Adder class that supports returning the sum for a vector of integers.

Because the other requirement for the new Adder class stipulates that the legacy library can't be changed, you have to resort to another technique to change the return type of the ICalculatorFactory.CreateAdder method. You can make this change through the use of the ComImport attribute, which lets you define the exact binary representation of a COM interface. So the interface for the calculator factory can now be defined within the .NET Framework environment as follows:

[
   ComImport,
   Guid( "0A4043BF-D9D0-47EB-AC23-4D5E62154A7C" ),
   InterfaceType( ComInterfaceType.InterfaceIsIUnknown )
]
public interface INewCalculatorFactory
{
   [return: MarshalAs(
       UnmanagedType.CustomMarshaler,
       MarshalTypeRef = typeof( AdderMarshaler ) )]
       NewAdder CreateAdder();
}

There are three important points of interest in this interface definition besides the use of the ComImport attribute. The first is that the interface Guid and IUnknown interface inheritance is the same as that specified in the original IDL definition of the interface. Note that the interface name is also different from the original IDL definition. Such a name change is not strictly necessary, but it helps to disambiguate references in this article pertaining to the original IDL definition versus the new C# definition.

The second point of interest is that the return type from CreateAdder is no longer IAdder. Instead, CreateAdder returns a new type called NewAdder. The MarshalAs attribute specified for the return type of CreateAdder facilitates the return of the NewAdder instance. The parameters to the MarshalAs attribute state that a COM custom marshaler instance of type AdderMarshaler will be used to generate the NewAdder instance.

The third point of interest is that a programmer specifies COM custom marshaling on a per-method-parameter basis. That is, custom RCWs and CCWs can only be used when required. In general, there is no way to specify the use of custom marshaling on a per-type basis.

Defining the Adapter Class

You can refer to the NewAdder type as an Adapter class. An Adapter class is an intermediate type that supports the new public view which the Adapter class can implement using any reasonable approach. Typically, it will use the functionality of the legacy object to implement the new public view. Such will be the case with the NewAdder implementation shown in Figure 4. In effect, the Adapter class is an intermediary or proxy that stands between a caller and the ultimate target CCW or RCW.

Figure 4 NewAdder Class (Close Figure 4)
public class NewAdder
{
   internal NewAdder( CalculatorLibraryLib.IAdder remoteAdder )
   {
      m_remoteAdder = remoteAdder;
   }

   public int Add( int[] ints )
   {
      int sum = 0;
      foreach ( int i in ints )
      {
         sum = m_remoteAdder.Add( sum, i );
      }

      return sum;
   }

   private CalculatorLibraryLib.IAdder m_remoteAdder;
}

The constructor for a NewAdder takes as input a single parameter that specifies the legacy instance responsible for generating the vector summation. The COM custom marshaler will supply this legacy instance. The following section describes the complete actions of the COM custom marshaler. The public Add method takes as input a vector of integers and uses the legacy COM Adder to do the real work.

Defining the Custom Marshaler

The definition of a COM custom marshaler is the final step in completing the implementation of the new Adder class. The custom marshaler is responsible for generating the proper transition environment between the managed and unmanaged worlds. This transition environment is either a CCW or RCW Adapter object.

A COM custom marshaler has two implementation requirements. The first is that a COM custom marshaler must implement the interface, ICustomMarshaler, which can be found in the System.Runtime.InteropServices namespace. The following code snippet describes the ICustomMarshaler interface:

interface ICustomMarshaler
{
   IntPtr MarshalManagedToNative( object ManagedObj );
   object MarshalNativeToManaged( IntPtr pNativeData );
   void CleanUpManagedData( object ManagedObj );
   void CleanUpNativeData( IntPtr pNativeData );
   int GetNativeDataSize();
}
The key methods in this interface are MarshalManagedToNative and MarshalNativeToManaged. The .NET Framework calls MarshalManagedToNative when a method parameter makes a transition from the .NET-managed environment to the unmanaged COM environment. A call to MarshalManagedToNative results in the creation of a CCW Adapter object. Likewise, the .NET Framework calls MarshalNativeToManaged when a method parameter makes a transition from the unmanaged COM environment to the .NET-managed environment. In this particular case, a call to MarshalNativeToManaged will result in the creation of an RCW Adapter object.

The .NET Framework calls the methods CleanUpManagedData and CleanUpNativeData after any of the previously described marshaling calls have been completed. The .NET Framework invokes CleanUpManagedData to release any resources that may have been allocated during the MarshalManagedToNative call. Likewise, the .NET Framework invokes CleanUpNativeData to release any resources that may have been allocated during the MarshalNativeToManaged call. In general, you should find marshalers to be essentially stateless. As a result, releasing managed or unmanaged resources should not be required.

Finally, GetNativeDataSize returns the size required to marshal a value type parameter. The current release of the .NET Framework (version 1.1) does not support the marshaling of value types. This method should simply return -1.

The second requirement for a COM custom marshaler is the implementation of the following method:

static ICustomMarshaler GetInstance( string cookie )

A typical implementation of GetInstance makes the custom marshaler a singleton. A singleton is used because the .NET Framework calls GetInstance only once per custom marshaler type, no matter how many parameters or return types specify the use of the particular custom marshaler type. A reasonable GetInstance implementation might look like this:

class Marshaler
{
   •••
   public static ICustomMarshaler GetInstance( string cookie )
   {
      return m_marshaler;
   }

   private static ICustomMarshaler m_marshaler = new Marshaler();
   •••
}

The cookie parameter can be specified in the public MarshalCookie field of the MarshalAs attribute. The cookie value can be used in any manner by the GetInstance method of the custom marshaler. Note that the cookie value can even be ignored as in the example that was previously shown.

The code in Figure 5 represents a complete implementation of a COM custom marshaler for the improved Adder class. The only method of interest is MarshalNativeToManaged. As presented earlier, MarshalNativeToManaged creates an RCW Adapter object that implements the new public view using the legacy COM Adder. MarshalNativeToManaged does this by first obtaining an RCW for the legacy Adder using the Marshal.GetObjectForIUnknown method. It then creates and returns a NewAdder Adapter instance to wrap the RCW.

Figure 5 COM Custom Marshaler Implementation (Close Figure 5)
internal class AdderMarshaler : ICustomMarshaler
{
   public static ICustomMarshaler GetInstance( string cookie )
   {
      // Return the static marshaler instance.
      return m_marshaler;
   }

   public object MarshalNativeToManaged( System.IntPtr pNativeData )
   {
      // Get a managed RCW corresponding to the native legacy
      // Adder instance.
      object remoteAdder = Marshal.GetObjectForIUnknown( pNativeData );

      // Wrap the legacy Adder in a NewAdder instance and return the
      // NewAdder instance.
      object newAdder = new NewAdder( (CalculatorLibraryLib.IAdder)
          remoteAdder );

      return newAdder;
   }

   public void CleanUpNativeData( System.IntPtr pNativeData )
   {
      // There is no data to release.  Do nothing.
   }

   public System.IntPtr MarshalManagedToNative( object managedObj )
   {
      // This method should never be called.  Marshaling a NewAdder
      // to unmanaged code is not supported.
      throw new NotSupportedException();
   }

   public void CleanUpManagedData( object managedObj )
   {
      // This should never be called.  Do nothing in any case.
   }

   public int GetNativeDataSize()
   {
      // This method should never be called.  Do nothing in any case.
      return -1;
   }

   private AdderMarshaler()
   {
   }

   private static AdderMarshaler m_marshaler = new AdderMarshaler();
}

At this point, a .NET-based client program can make use of the NewAdder instance with code like the following:

object rawFactory = new CalculatorLibraryLib.CalculatorFactoryClass();

INewCalculatorFactory factory = (INewCalculatorFactory) rawFactory;

NewAdder adder = factory.CreateAdder();

int[] ints = new int[] { 1, 2, 3, 4 };
int sum = adder.Add( ints );
Notice that the client program instantiates a legacy factory instance using the original Interop assembly. It then caches the legacy factory in the rawFactory variable. Following that, it casts the legacy factory instance into INewCalculatorFactory. The .NET Framework permits such a cast because the Guid attribute supplied on the INewCalculatorFactory definition corresponds to the one that's already supported by the legacy factory. From this point on, any reference to INewCalculatorFactory.CreateAdder will fire up the COM custom marshaling infrastructure in the .NET Framework, resulting in an instance of a NewAdder. With a NewAdder instance in hand, the client program can initiate the summation of an integer vector.

A More Sophisticated Example

The Interposer sample and COM custom marshaling together form the basis for more complex custom marshaling scenarios. It's worth considering one more example in which a RealProxy-based interception method coupled with COM custom marshaling may be useful. The sample in question goes beyond the typical uses of method interception and attempts to correct a shortcoming that's in the .NET Framework.

Let's consider a sample .NET-based application that is required to work with instances of .NET-based classes as well as instances of legacy COM IDispatch-based classes. From the application's point of view, it would be convenient to operate homogeneously across all instances, whether the instance type is based in COM or in the .NET Framework.

Assume that the basic operation of the sample .NET-based application uses reflection to determine the methods implemented by these various instances. Based on the methods that these instances implement, the application performs some internal event monitoring. When the application detects an internal event, the application calls the instance methods corresponding to the event.

The goal of this sample application appears simple enough; however, it should be noted that a reader familiar with COM Interop will discover an insidious problem. While the .NET Framework allows an application to invoke IDispatch-based COM object methods through reflection, strangely enough it does not permit an application to directly discover the method signatures of such COM objects using reflection.

Based on this information, it's clear that the sample application can't homogeneously reflect upon the various instances that it may encounter. Assuming that IDispatch-based COM instances can, and generally do, return type information through their IDispatch implementations, it is possible to overcome the inherent shortcomings of .NET Framework reflection with respect to IDispatch. Obtaining this type information is the first step toward accomplishing the application goal of treating all classes equally. Moreover, a RealProxy-derived class combined with COM custom marshaling is a key element in supporting this goal. This is because a TransparentProxy is the only type that can be dynamically configured as supporting a given interface.

More Advanced COM Interop

The IDL code shown in Figure 6 contains the canonical IDispatch interface declaration. The IDispatch GetTypeInfo method can be used in order to obtain an ITypeInfo interface. Given an ITypeInfo interface, it is possible for you to obtain instance method names and parameter information.

Figure 6 Canonical IDispatch (Close Figure 6)
[
    object,
    uuid(00020400-0000-0000-C000-000000000046),
    pointer_default(unique)
]
interface IDispatch : IUnknown
{
    HRESULT GetTypeInfoCount(
                [out] UINT *pctinfo
            );
    HRESULT GetTypeInfo(
                [in] UINT iTInfo,
                [in] LCID lcid,
                [out] ITypeInfo **ppTInfo
            );
    HRESULT GetIDsOfNames(
                [in] REFIID riid,
                [in, size_is(cNames)] LPOLESTR *rgszNames,
                [in] UINT cNames,
                [in] LCID lcid,
                [out, size_is(cNames)] DISPID *rgDispId
            );
    HRESULT Invoke(
                [in] DISPID dispIdMember,
                [in] REFIID riid,
                [in] LCID lcid,
                [in] WORD wFlags,
                [in, out] DISPPARAMS *pDispParams,
                [out] VARIANT *pVarResult,
                [out] EXCEPINFO *pExcepInfo,
                [out] UINT *puArgErr
            );
}

As stated earlier, the .NET Framework hides any notion of COM from .NET-based callers. This holds true even for an IDispatch-based COM instance where the RCW completely obscures the IDispatch interface. The problem then becomes how to obtain the IDispatch interface from a given RCW. This can be done by using the COM Interop method presented previously. Recall that a binary version of a COM interface can be defined in C# using the ComImport attribute. The code in Figure 7 uses this attribute to define a somewhat radical view of IDispatch compared to the definition in canonical IDL.

Figure 7 Revised IDispatch Definition (Close Figure 7)
[
   ComImport,
   Guid( "00020400-0000-0000-C000-000000000046" ),
   InterfaceType( ComInterfaceType.InterfaceIsIUnknown )
]
private interface IDispatch
{
   void Unused1();
   [PreserveSig]
   int GetTypeInfo( uint nInfo, int lcid,
      [MarshalAs(
         UnmanagedType.CustomMarshaler,
         MarshalTypeRef = typeof( TypeToTypeInfoMarshaler ) )]
      out System.Type type );
}

Because the .NET-based application will exclusively call IDispatch.GetTypeInfo, it's only necessary to define the binary version of IDispatch up to and including that method. Intervening methods can simply be tagged with names like "Unused1" that act as placeholders.

The C# IDispatch definition shown in Figure 7 also plays a trick quite similar to that presented in the Adder example. Notice that in the original canonical IDispatch IDL (shown in Figure 6), the last parameter of GetTypeInfo returns an ITypeInfo interface; however, the C# version defines that same parameter to return a System.Type. It just so happens that there is a custom marshaling routine called TypeToTypeInfoMarshaler that transforms an ITypeInfo interface into a System.Type object. The System.Type object is actually a dynamically generated interface whose metadata describes the methods of the underlying IDispatch implementation. This custom marshaling routine can be found in the CustomMarshaler.dll assembly located in the .NET Framework installation directory. As described previously, the use of the MarshalAs attribute applies TypeToTypeInfoMarshaler to the System.Type parameter in IDispatch.GetTypeInfo.

Calling IDispatch.GetTypeInfo as I've defined results in the return of an interface type that describes the methods and parameters of the particular COM instance implementing IDispatch. The resulting interface type is a great boon to the sample .NET-based application and can be applied to a RealProxy-derived class. Furthermore, the resultant TransparentProxy will present the public view defined by this interface. The bottom line is that the sample .NET Framework-based application will now be able to reflect on the resulting TransparentProxy (using Type.GetMethod) as it would any other .NET Framework-based object. In addition, the sample application will also be able to invoke methods in a consistent way across all classes.

Figure 8 presents the code for a RealProxy-derived class called GenericReflectionProxy that handles reflection against IDispatch-based COM object instances. Note that GenericReflectionProxy follows the basic template illustrated in Figure 2. In addition to the basic template, GenericReflectionProxy adds some code to the Marshal method that discriminates between .NET-based objects and COM objects. That is, the Marshal method only returns a TransparentProxy when the instance provided is COM-based.

Figure 8 GenericReflectionProxy (Close Figure 8)
internal class GenericReflectionProxy : RealProxy
{
   public static object Marshal( object instance )
   {
      // Any null references are simply returned back.
      if ( instance == null )
      {
         return null;
      }

      // Next, we'll see if the instance attempting to be wrapped
      // is a COM object.  If it's not, there's no need to put a
      // proxy around it because .NET objects support reflection.
      // Instead we'll just return the instance itself.
      Type type = instance.GetType();
      if ( !type.IsCOMObject )
      {
         return instance;
      }

      // At this point, we're dealing with a COM object.  We'll get the
      // default interface type and wrap the COM object in a proxy.
      Type interfaceType = GetInterface( instance );
      GenericReflectionProxy proxy =
         new GenericReflectionProxy( interfaceType, instance );

      return proxy.GetTransparentProxy();
   }

   public override ObjRef CreateObjRef( Type requestedType )
   {
      throw new System.NotSupportedException();
   }

   public override IMessage Invoke( IMessage message )
   {
      IMethodCallMessage methodMessage =
         new MethodCallMessageWrapper( (IMethodCallMessage) message );

      MethodBase method = methodMessage.MethodBase;

      object returnValue = null;

      // If the method that we're attempting to call is "GetType",
      // return the original interface type.
      if ( method.Name == "GetType" )
      {
         returnValue = m_interfaceType;
      }
      else
      {
         // We're attempting to execute some other method.

         // Because we're going to execute using the IDispatch interface
         // on the COM object, we need to set up parameter modifiers for
         // all arguments that are out or ref.  These modifiers allow the
         // COM Interop layer to pass the arguments as VT_BYREF.
         ParameterModifier[] modifiers = null;

         if ( methodMessage.ArgCount > 0 )
         {
            ParameterModifier modifier =
               new ParameterModifier( methodMessage.ArgCount );
            modifiers = new ParameterModifier[] { modifier };

            int index = 0;
            foreach ( ParameterInfo pi in method.GetParameters() )
            {
               modifier[ index++ ] = pi.ParameterType.IsByRef;
            }
         }

         // Select the correct binding flags for invoking a method. It
         // should be noted that as written, this code will not get or
         // set IDispatch-based properties.  This exercise is left to the
         // reader.  ;-)
         const BindingFlags flags = BindingFlags.InvokeMethod;

         // Call the actual method using the IDispatch interface.
         returnValue =
            m_instanceType.InvokeMember(
               method.Name, flags, null, m_instance,
               methodMessage.Args, modifiers, null, null );

         // We now have to look at all values to be returned.  We
         // make sure that if they are COM objects, that we wrap
         // them as necessary in a GenericReflectionProxy.
         returnValue = Marshal( returnValue );

         int j = 0;
         foreach ( ParameterInfo pi in method.GetParameters() )
         {
            if ( pi.ParameterType.IsByRef )
            {
               methodMessage.Args[ j ] =
                 Marshal( methodMessage.Args[ j ] );
            }
               j++;
         }
      }

      IMessage returnMessage =
         new ReturnMessage( returnValue, methodMessage.Args,
            methodMessage.ArgCount, methodMessage.LogicalCallContext,
            methodMessage );

      return returnMessage;
   }

   private GenericReflectionProxy( Type interfaceType, object instance )
      :
      base( interfaceType )
   {
      m_interfaceType = interfaceType;
      m_instance = instance;
      m_instanceType = instance.GetType();
   }

   private static Type GetInterface( object instance )
   {
      // The Marshal method has filtered out non-COM objects.  We
      // now have to determine whether or not the incoming COM object
      // supports IDispatch.  Furthermore if it does support IDispatch,
      // it must support type information.  If the incoming object
      // fails to support both, we'll return the type of interface as
      // "IDontSupportReflection".  This interface supports no methods.
      Type defaultType = typeof( IDontSupportReflection );
      Type type = defaultType;

      // Make sure that the incoming object supports IDispatch.  If it
      // doesn't we'll just return the default type.
      IDispatch dispatch = instance as IDispatch;
      if ( dispatch != null )
      {
         // The incoming object supports IDispatch.  Attempt to get
         // type information from the object.  This may fail.  If it
         // does fail, return the default type.
         int status = dispatch.GetTypeInfo( 0, 0, out type );
         if ( status != 0 || type == null )
         {
            type = defaultType;
         }
      }

      return type;
   }

   [
      ComImport,
      Guid( "00020400-0000-0000-C000-000000000046" ),
      InterfaceType( ComInterfaceType.InterfaceIsIUnknown )
   ]
   private interface IDispatch
   {
      void Reserved();
      [PreserveSig]
      int GetTypeInfo( uint nInfo, int lcid,
         [MarshalAs(
             UnmanagedType.CustomMarshaler,
             MarshalTypeRef = typeof( TypeToTypeInfoMarshaler ) )]
         out System.Type typeInfo );
   }

   private interface IDontSupportReflection
   {
   }

   private Type m_interfaceType;
   private object m_instance;
   private Type m_instanceType;
}

GenericReflectionProxy also adds a private static method called GetInterface. This method returns an interface type describing the COM object. If the COM object supports IDispatch, and that IDispatch implementation provides type information, GetInterface returns an interface type that describes the methods supported by the IDispatch implementation. If the COM object doesn't support IDispatch or the IDispatch implementation does not provide type information, GetInterface returns an internal interface type that supports no methods. In this way, a .NET-based application will find no supported methods when doing reflection.

Conclusion

This article has presented insight into important, yet little-known custom marshaling techniques available in .NET Remoting and COM custom marshaling. You can extend a RealProxy using interfaces to control the overall public view presented by the resulting TranparentProxy. In addition, COM custom marshaling provides capabilities to change object views across the managed/unmanaged boundary. In many cases, some powerful method interception behaviors can be achieved when combining these techniques.

Even though the custom marshaling techniques I've presented demonstrate the extreme flexibility of the .NET Framework, this flexibility doesn't come without a price. There is a significant performance penalty for using an Interposer-based solution. Compared with reflection, invoking a method on an Interposed object is an operation that turns out to be roughly 10 times slower. Likewise, COM custom marshaling adds about 50 percent more execution time to the marshaling process. For the designer, the usual consideration of performance versus flexibility arises; however, in many cases the flexibility of these marshaling techniques far outweigh any performance penalty.

For related articles see:
.NET Remoting: Design and Develop Seamless Distributed Applications for the Common Language Runtime
.NET Interop: Get Ready for Microsoft .NET by Using Wrappers to Interact with COM-based Applications
Extending RealProxy

For background information see:
Side-By-Side Execution for COM Interop
.NET and COM: The Complete Interoperability Guide by Adam Nathan (SAMS, 2002)

Jim Sievert is a professional consultant for Unisys Corporation. He'll often be found with the mud of new technologies all over his hands. His responsibilities include the pursuit of new technologies that enhance system control software for the Unisys ES7000 line of enterprise computer systems. He can be reached at james.sievert@unisys.com..


.NET Internals: Rewrite MSIL Code on the Fly with the .NET Framework Profiling API

.NET Internals

Rewrite MSIL Code on the Fly with the .NET Framework Profiling API

Aleksandr Mikunov
This article assumes you're familiar with the CLR and C#
Level of Difficulty 1 2 3
Code download available at: NETProfilingAPI.exe (2,901KB)
SUMMARY

In this article, the author shows how to dynamically rewrite Microsoft Intermediate Language code on the fly using the Profiling API of the CLR. Unlike approaches based on Reflection.Emit, this scheme works with the existing assemblies and doesn't require the creation of proxy or dynamic assemblies. The need for IL code rewriting emerges when you want to make your changes transparent to the client and preserve the identity of classes. This technique can be used for creation of interceptors, pre- and post-processing method calls, and code instrumentation and verification.

Contents
Internal Representation of Methods by the CLR
Basics of MSIL Code Rewriting
More Advanced Techniques
Profiler Example
Conclusion

A
s a developer you have probably encountered a scenario where one application needs to preprocess and post-process calls to another application (whose source code is not available) to perform validation of input and output parameters, enforce application-specific business rules, or do some call tracing. You may also have had to cache method calls when returning a cached result rather than calling the original method, presuming the method was called with the same parameter and within a certain period of time. In such cases, you could dynamically generate a new proxy assembly which does the preprocessing, calls the original assembly's method, and performs the post-processing.

The most obvious drawback with this scheme is that the client has to reference the proxy assembly instead of the original. Thus, generally speaking, the identity of the classes is not preserved and the client application has to be modified.

Another drawback is that you must work with dynamic assemblies that are created using Reflection.Emit. When a dynamic assembly is saved to disk and reloaded, it is no longer dynamic and therefore is treated like any other assembly. The infrastructure of Reflection.Emit will no longer allow you to modify its modules and classes nor change its intermediate language (IL) code. If you want to dynamically change a method's IL code within an already existing assembly or modify an assembly's types and methods, you need to find some other way.

In this article, I will describe a simple and powerful approach which allows you to overcome many (if not all) of the limitations of Reflection.Emit. This technique will allow you to modify existing assemblies and add new types and methods to already loaded assemblies. You will also be able to rewrite the IL code of a given method on the fly.

Unlike the traditional approaches based on the Reflection API and dynamically emitted assemblies, this scheme works with existing assemblies and doesn't require the creation of proxies or dynamic assemblies. Thus, you preserve the identity of the classes whose methods are instrumented and your changes are transparent to the client applications. This approach will give you further insight into the underpinnings of IL code and insight into how the Profiling API works, but note that its not suited for deployment in a production environment. The reason is that it's based on the Profiling API, which after being registered circumvents the CLR security infrastructure. Plus it prevents use of other profilers which are seeking to perform real performance analysis on the app.

The basic idea of this approach is as follows. When the CLR loads a class and executes its method, the method's IL code is compiled to native instructions during the just-in-time (JIT) compilation process. The Profiling API provided as part of the CLR allows you to intercept this process. Before a method gets JIT-compiled, you can modify its IL code. In the simplest scenario, you can insert the customized prologue and epilogue into the method's IL and give the resulting IL back to the JIT compiler. If you desire, the newly generated IL could do some additional work before and after the original method's code is called.

You can also add new local variables, guarded blocks, and exception handlers to the method. More advanced techniques could dynamically add new data members and methods to the loaded class or create new types.

Figure 1 Reflection API versus IL Code Rewriting
Figure 1 Reflection API versus IL Code Rewriting

Figure 1 summarizes the important differences between approaches based on the Reflection API and the approach based the dynamic IL code rewriting. The techniques I describe here were tested for both the Microsoft® .NET Framework 1.0 and Shared Source CLI (SSCLI) Beta Refresh ("Rotor").

Internal Representation of Methods by the CLR

Before I delve into the details of IL code rewriting, I want to review several important aspects of the CLR, such as the internal representation of methods and structured exception handling (SEH) tables. Let's start with a simple "Hello world!" program:

// C#
using System;

class MainApp
{
       public static void Main()
     {
          Console.WriteLine( "Hello World!" );
       }
}

What follows is the IL code for this code snippet, which I got from running the IL Disassembler utility (ildasm.exe). I've also included opcodes for the IL instructions:

// MSIL
.method public hidebysig static void  Main() cil managed
{
  /* 72 | (70)000001*/ ldstr "Hello World!"
  /* 28 | (0A)000002*/ call  void System.Console::WriteLine(string)
  /* 2A |           */ ret
}
When the program gets compiled to IL, information about classes and methods in the hello.exe assembly is stored in the metadata tables. In particular, the Method metadata table holds information about the assembly's methods.

Each entry in this table provides the CLR with important information about a method. This is the relative virtual address (RVA) of the method's IL code—the offset from the address where the image (EXE or DLL) was loaded. The entry describes the method's properties such as managed/unmanaged, private/public, static/instance, virtual, and abstract. It also contains the method's name, signature, and a pointer to the Param table, which specifies additional information about the method's parameters.

The image file also contains a special header called the runtime header, which stores information about the application entry point. To view this information, launch the IL Disassembler utility and then click the View / COR header menu. The runtime header for the "Hello world!" assembly should look like this:

CLR Header:
 72       Header Size
 2        Major Runtime Version
 0        Minor Runtime Version
 1        Flags
 6000001  Entrypoint Token
 207c     [208     ] address [size] of Metadata Directory:
•••

Now let's see how the CLR handles method-specific metadata during loading and JIT compilation. When the assembly is loaded, the CLR examines the runtime header to determine the application entry point and will recognize that this is a method coded by the metadata token 0x06000001 (tokens of the form 0x06XXXXXX are used for methods). It will also recognize that the method's information is stored in the first row of the Method table that is shown in Figure 2. Using this record, the CLR will be able to locate the memory address that holds the actual method body and also obtain the method's description.

Figure 2 Main Method Stored in the Method Table (Close Figure 2)

RVA | ImplFlags | Flags | Name | Signature | ParamList
0x2050 (offset from the start of the EXE file) | 0x0 (IL = 0x0, Managed = 0x0) | 0x96 (Public =0x6, Static = 0x10, Hide by name + signature = 0x80) | Some value (an index pointing to the "Main" string) | 0x00 0x00 0x01 (no params, return type is void) | Some value

If you added the method's RVA to the load address (the address where the executable file for the "Hello World!" assembly was loaded), you would see a memory dump layout similar to the one shown in Figure 3. On my machine, the load address was 0x06EA1000, so the physical address for the method body is 0x06EA1000 + 0x2050, or 0x6EA3050. You could also open the hello.exe file (provided in the code download for this article) with any binary editor and locate the method's body at the offset of 0x250.

Figure 3 Memory Dump
Figure 3 Memory Dump

Although the offset of a method in the assembly file (file pointer) usually differs from its RVA, you can follow these steps to locate a method's body within the file on disk. First, run ildasm.exe with the /Adv switch. Next, open the assembly and select the View | MetaInfo | Show! menu item (or press Ctrl-M). The tool will generate an output window with the assembly's metadata information (Modules, Types, Methods, and so on). You can now find the method you are interested in and read its RVA.

Run the dumpbin.exe utility with the /ALL switch (the utility is located in the Vc7\bin folder under your Visual Studio® .NET installation) and in the output produced by the tool you should find information about the section named .text (this section contains the IL code of the assembly). You will see both the value of the section's file pointer and the section dump called "raw data." Finally, use the method's RVA provided by ILDASM to find the method's body within the dump.

As you can see in Figure 3, the RVA column points to the method body, which consists of a method header, IL code, and possibly some additional sections, as shown in Figure 4. Let's look closely at these structures.

Figure 4 IL Method Body Layout
Figure 4 IL Method Body Layout

Currently, there are two types of method headers: tiny and fat. The tiny header is used when the method is smaller than 64 bytes, when its stack depth won't exceed 8 slots (one slot for each item on the stack regardless of the item's size), and when it contains no local variables or SEHs.

The structure of the tiny header is declared in the CorHdr.h file, which is located in the \FrameworkSDK\include folder under your Visual Studio .NET installation (see Figure 5). The Flags_CodeSize field has the following binary form: XXXXXX10b. The upper 6 bits are used to store the IL code size in bytes (the header size is not counted) and the lower 2 bits hold the tiny header type code (0x02). If you need to calculate the size of a tiny method, you would read the Flags_CodeSize byte and then shift it right by 2 bits.

Figure 5 Tiny and Fat Headers (Close Figure 5)
// CorHdr.h
-----------------------------------------------------------------------
// tiny method header
typedef struct IMAGE_COR_ILMETHOD_TINY
{
    BYTE Flags_CodeSize;
} IMAGE_COR_ILMETHOD_TINY;
-----------------------------------------------------------------------
// fat method header
typedef struct IMAGE_COR_ILMETHOD_FAT
{
    unsigned Flags    : 12;     // Flags
    unsigned Size     :  4;     // size in DWords of this structure
                                // (currently 3)
    unsigned MaxStack : 16;     // maximum number of items (I4, I, I8,
                                // obj ...),
                                // on the operand stack
    DWORD   CodeSize;           // size of the code
    mdSignature  LocalVarSigTok;// token that indicates the signature of
                                // the local vars (0 means none)
} IMAGE_COR_ILMETHOD_FAT;

typedef union IMAGE_COR_ILMETHOD
{
    IMAGE_COR_ILMETHOD_TINY       Tiny;
    IMAGE_COR_ILMETHOD_FAT        Fat;
} IMAGE_COR_ILMETHOD;

The fat header has a more complex structure and must be DWORD aligned. Unlike the tiny header, the fat header has a special field to store the size of the IL code. The CorHdr.h file also defines a convenient union IMAGE_COR_ILMETHOD (also shown in Figure 5). Using this union, you can easily write these two functions to determine the type of a given header (see Figure 6).

Figure 6 IsTinyHeader and IsFatHeader Functions (Close Figure 6)
// these flags are declared in CorHdr.h, CorILMethodFlags enum
// •••
// CorILMethod_FormatShift= 3,
// CorILMethod_FormatMask = ((1 << CorILMethod_FormatShift) - 1),
// CorILMethod_TinyFormat = 0x0002,
// CorILMethod_FatFormat  = 0x0003,
// •••

BOOL IsTinyHeader( const IMAGE_COR_ILMETHOD* pMethodHeader )
{
    return ( BOOL )( (pMethodHeader->Tiny.Flags_CodeSize &
        (CorILMethod_FormatMask >> 1)) == CorILMethod_TinyFormat );
}

BOOL IsFatHeader ( const IMAGE_COR_ILMETHOD* pMethodHeader )
{
return ( BOOL )( (pMethodHeader->Fat.Flags & CorILMethod_FormatMask)
    == CorILMethod_FatFormat );
}

Now let's go back to the Main method's memory layout. As was shown in Figure 3, the Main method has a fat header:

// method header
0x13 0x30 0x01 0x00
0x0b 0x00 0x00 0x00 0x00 0x00 0x00 0x00
This gives the layout shown in Figure 7 (don't forget about Intel's reversed byte-order).

Figure 7 Fat Header of the Main Method (Close Figure 7)

Fat Header Entry and Its Size | Value | Note
Header type, Flags, and header size (WORD) | 0x3013 (0011000000010011) | The upper 4 bits (0011) hold the header size in DWORDs; that is, 3. The next 10 bits (0000000100) hold the Flags value (0x4), which means that local variables must be initialized. The lower 2 bits (11) indicate the header type (Fat).
MaxStack (WORD) | 0x1 | Maximum stack size in slots (items).
CodeSize (DWORD) | 0x0b | IL code size in bytes (without method header).
LocalVarSigTok (DWORD) | 0x0 | Token of the local variables signature. It's equal to zero since no local variables are presented.

The method's IL code, which consists of a sequence of IL instructions, is located immediately after the method's header:

// IL code of the Main method
0x72 0x01 0x00 0x00
0x70 0x28 0x02 0x00 0x00 0x0a 0x2a // <== this the last
                                   // "ret" instruction
As you can see, the code size is 11 bytes (0x0b), as specified by the method header. You may also notice that the last byte is 0x2a, which is the IL opcode for the return instruction.

Using IL opcode values and the IL code parsing technique, which I'll explain later in this article, you could easily generate a listing for the Main method similar to the one produced by ILDASM. Since the size of the fat header in double words is 3 (12 bytes), a fat method's IL code is always DWORD aligned. This is obviously not the case for tiny headers.

So far, you can see that the internal representation of methods is relatively simple. The picture gets more complicated, however, when a method uses exception handing. In this case, there should be a way to make this information available to the execution engine. For this purpose, a source code compile to IL (using the C# compiler in this case) generates special SEH tables. The compiler also sets the Flags field in the method header to 0x02 (the CorILMethod_MoreSects value from CorHdr.h) to tell the runtime that the method body has extra sections.

Although I won't go into all the details of SEH tables until later when I discuss IL code rewriting, it's important to understand that these tables simply contain DWORD-aligned sections that follow the method's IL code body, beginning with the section header. This is then followed by a sequence of exception handling clauses, as was shown in Figure 4. Each exception handling clause is also DWORD aligned. Because of the alignment, there is usually some gap (up to 3 bytes) between the last IL instruction and the first SEH section header. The section header and each exception handling clause can have either a small or fat format. They are described in CorHdr.h and are also shown in Figure 8.

Figure 8 Small and Fat SEH Section Headers (Close Figure 8)
// CorHdr.h
typedef struct IMAGE_COR_ILMETHOD_SECT_SMALL
{
    BYTE Kind;
    BYTE DataSize;
} IMAGE_COR_ILMETHOD_SECT_SMALL;

typedef struct IMAGE_COR_ILMETHOD_SECT_FAT
{
    unsigned Kind : 8;
    unsigned DataSize : 24;
} IMAGE_COR_ILMETHOD_SECT_FAT;

typedef struct IMAGE_COR_ILMETHOD_SECT_EH_CLAUSE_FAT
{
    CorExceptionFlag    Flags;
    DWORD               TryOffset;
    DWORD               TryLength;      // relative to start of try block
    DWORD               HandlerOffset;
    DWORD               HandlerLength;  // relative to start of handler
    union {
        DWORD           ClassToken;     // use for type-based exception
                                        // handlers
        DWORD           FilterOffset;   // use for filter-based exception
                                        // handlers
(COR_ILEXCEPTION_FILTER is set)
    };
} IMAGE_COR_ILMETHOD_SECT_EH_CLAUSE_FAT;

typedef struct IMAGE_COR_ILMETHOD_SECT_EH_CLAUSE_SMALL
{
#ifdef _WIN64
    unsigned            Flags         : 16;
#else // !_WIN64
    CorExceptionFlag    Flags         : 16;
#endif
    unsigned            TryOffset     : 16;
    unsigned            TryLength     : 8;  // relative to start of try
                                            // block
    unsigned            HandlerOffset : 16;
    unsigned            HandlerLength : 8;  // relative to start of
                                            // handler
    union {
        DWORD       ClassToken;
        DWORD       FilterOffset;
    };
} IMAGE_COR_ILMETHOD_SECT_EH_CLAUSE_SMALL;

Let's take a look at the SEH section header. The Kind field is always 1 byte in size and it holds a set of binary flags which are defined by the CorILMethodSect enumerator (see CorHdr.h for details). The IL compiler always sets the CorILMethod_Sect_EHTable flag (0x01) which tells the runtime that this is an SEH header. For the fat section header, the Kind byte also contains the CorILMethod_Sect_FatFormat (0x40) value. The rest of the flags are either optional or not currently used. The most typical values for the Kind field are 0x41 and 0x01.

The DataSize holds the total size in bytes of the section header and any related exception handler clauses. If, for example, you have one fat SEH section and a sequence of 14 fat clauses, DataSize will be set to the following value:

sizeof(FAT section header) + 14 * sizeof(FAT exception handler clause)

Each exception handler clause provides a complete detailed description of the exception handler, such as the offsets from the beginning of the method body of both the protected code block (the try block) and the exception handler itself, along with the sizes of the try and handler blocks.

The types of exception handler clauses are described by the Flags field, which can have one the following values declared in the CorExceptionFlag enumerator (see CorHdr.h for details):

  • The COR_ILEXCEPTION_CLAUSE_NONE value (0x0) corresponds to the try/catch blocks.
  • The COR_ILEXCEPTION_ CLAUSE_FILTER value (0x1) corresponds to the filters.
  • COR_ILEXCEPTION_CLAUSE_ FINALLY (0x2) is used for the try/finally blocks.
  • COR_ILEXCEPTION_CLAUSE_FAULT (0x4) is used for the finally blocks that are called inside exception handlers.

Basics of MSIL Code Rewriting

Now let's turn to the basic technique of IL code rewriting; namely, the insertion of prologues and epilogues into methods. This approach makes use of the Profiling API and the internal CLR structures that I've described so far. The application I'll develop is a profiler DLL, which is integrated with the CLR through the ICorProfilerCallback and ICorProfilerInfo interfaces. The runtime uses ICorProfilerCallback to notify the profiler of the various events such as class loads and unloads, JIT compilation, garbage collection, and threading. The profiler calls the ICorProfilerInfo interface (the profiler host) to get detailed information about the runtime's internal structures and to modify them if needed.

I'm particularly interested in the events related to the JIT compilation process. Before a method gets JIT-compiled, the runtime calls the profiler's ICorProfilerCallback::JITCompilationStarted method, allowing you to make changes in the method's IL.

Now let's take a closer look at the profiler's implementation details. Some of these are common to all profilers. For example, at initialization time, I store a pointer to the ICorProfilerInfo interface (implemented by the CLR) and register for the events I'm interested in. The Profiling API also requires that you implement standard COM entry points such as DllGetClassObject, DllCanUnloadNow, DllRegisterServer, and DllUnregisterServer functions. Note that although profilers are implemented as COM DLLs, the CLR itself doesn't use the COM API.

Those of you who have the SSCLI can look at the Rotor implementation of the method that loads a profiler DLL. You'll see that the runtime implements its own version of the CoCreateInstance function. The code loads the profiler DLL, gets the class object, and calls the IClassFactory interface to instantiate the object, which then implements the ICorProfilerCallback interface. (This implementation is located in the file \sscli\clr\src\profile\ee\profile.cpp.)

When looking at the Rotor implementation, you will also notice that unlike the .NET Framework, Rotor uses the environment variable COR_PROFILER_DLL to find the complete path to the profiler DLL. Before you can use the DLL in the Rotor environment you should set this variable. Now let's see how the profiler handles the JIT compilation events.

The CLR calls ICorProfilerCallback::JITCompilationStarted to notify the code profiler that the JIT compiler is starting to compile a function. Here's the description of this callback:

HRESULT ICorProfilerCallback::JITCompilationStarted( FunctionID
    functionId, BOOL fIsSafeToBlock )
The CLR passes the ID of the function being JIT-compiled to the profiler. In Rotor this ID is actually a pointer to the internal runtime's structure called MethodDesc, and can be used to query the profile host to get information about the function, but it should only be used as an opaque identifier. The second parameter tells you whether it's safe to perform a time-consuming operation in the code. The Profiling API documentation states that ignorance of this parameter won't harm the runtime. In addition, I've noticed in the Rotor source code that this parameter is always set to TRUE, so I'll ignore it.

The JITCompilationStarted event is a safe place to modify the method before it gets JIT-compiled. For this purpose, the core profiler host, ICorProfilerInfo, provides two very powerful functions, GetILFunctionBody and SetILFunctionBody:

HRESULT GetILFunctionBody(
    ModuleID moduleId,       // ModuleID of the given module.
    mdMethodDef methodId,    // Metadata token for method.
    LPCBYTE *ppMethodHeader, // Pointer to the IL method body.
    ULONG *pcbMethodSize     // Pointer to the size of the method);

HRESULT SetILFunctionBody(
    ModuleID moduleId,     // ModuleID of the given module.
    mdMethodDef method,    // Metadata token for method.
    LPCBYTE pbNewILMethod, // Pointer to the new IL method body.
    ULONG cbNewMethod      // Size of the new method.);

The parameters are self-explanatory. The first method (GetILFunctionBody) returns a pointer to the method body for a given method's metadata token and a module ID (metadata is created on a per-module basis). The second function allows you to modify the existing method using a newly created method body. The Profiling API also requires that you allocate memory for the new method using the special IMethodMalloc interface, which can be obtained from the CLR through an ICorProfilerInfo::GetILFunctionBodyAllocator call.

My implementation of ICorProfilerCallback::JITCompilationStarted performs the following main steps. It modifies a method header (both fat and tiny). It then adds a prologue, which implies that you have to add a set of new IL instructions to the beginning of the method so the original IL code gets shifted. It adds an epilogue to the end. Since the original IL code changes, you also have to modify SEH tables, which store the relative values of offsets and lengths of guarded blocks and exception handlers.

Note that the JIT compilation process also performs a verification process that examines the code and attempts to determine whether or not the code is safe and incapable of performing any hidden hacks. As long as you add verifiable IL instructions, the original code also remains verifiable and will thus pass the verification process. Let's take a closer look at these four steps.

Changes in the method header are relatively simple—since you modify the size of the IL code, you'll have to change the CodeSize field accordingly. Epilogues and prologues may also use the method's stack, declare their own local variables, and add SEH sections. Thus, three fields—Flags, MaxStack, and LocalVarSigTok—will have to be modified as well.

Inserting a prologue is a straightforward procedure. The only requirement is that the method stack should be empty before the old IL gets executed. This rule has a very simple explanation—the original method won't know anything about your code and assumes that its stack is empty. You also have to restore the values of the method's arguments and initial values for any local variables you've used. In other words, the evaluation stack of the method and local variables should be restored to their initial state.

Insertion of an epilogue requires a lot of work and, in the very generic case, is extremely complicated. First, you have to identify all the points where the method may return. Typically, there are three return scenarios:

  • The ret (return) instruction, which causes the method to return to the call site.
  • The throw instruction, which pops an exception object from the stack and throws it as a managed exception.
  • The rethrow instruction, which throws the already-caught exception again and can only be used within SEH handlers.
Combinations of these three methods are also possible, such as in the example shown in Figure 9. In this particular case there's no easy way to emit the epilogue. Even if you replaced the ret statement with a branching instruction, you would still have to modify the original error handler (which starts at line IL_0012). The less obvious problem is that since the ret is at least 1 byte shorter than any of the branching instructions, you will also have to modify the bne.un.s instruction at IL_0008 (the bne.un.s instruction takes two values from the stack and branches if the first value is not equal to the second one; the "s" postfix means that it is a short form of the instruction).

Figure 9 Method Using Ret and Throw to Exit (Close Figure 9)
// MSIL
.method ••• SomeMethod( ••• ) cil managed
{
        •••
  IL_0008:  bne.un.s   IL_0012 // go to the error handler
        •••
  IL_0011:  ret // normal exit
  IL_0012:  ••• // error handler begins here

        •••
  // create an exception object
  IL_0048:  newobj instance void System.Exception::.ctor(•••)
  // and throw it
  IL_004d:  throw
} // end of SomeMethod

For the sake of simplicity, I will consider only those cases in which a method returns via ret. I will have to perform a few steps such as parsing the IL code and identifying all the return instructions, then replacing them with my branching instructions in order to transfer control to the epilogue. Since the new branching instructions I've added are at least 1 byte longer, I will have to correct the original branching instructions as well. As you can see, the simplest scenario is when the IL code has only one ret instruction at the end. In this case, I could replace the ret opcode with the nop instruction, causing the method's execution flow to be passed to the epilogue. In order to do this, I have to be able to parse, restructure, and modify the IL code. This will require some familiarity with the basics of IL code parsing.

IL instruction opcodes are declared in the \FrameworkSDK\include\opcode.def file, which also contains information about instruction sizes and the intermediate language parameters they take. Initially you set the instruction pointer (IP) to point to the very first byte of the IL code. Next, you try to identify the first IL instruction using IL opcodes and add the size of instruction and its parameters to the IP. After doing this, the process will repeat until you reach the end of the method. It looks like a straightforward procedure, except for one thing: the number of IL instructions is about 300, so you must have a huge switch statement with more than 250 cases. Let's see how it all works for this simple input:

// IL code (DWORD aligned)
0x14 0x0E 0x00 0x28
0x01 0x00 0x00 0x0A 0x26 0xDE 0x0D 0x26
0x72 0x73 0x00 0x00 0x70 0x28 0x02 0x00
0x00 0x0A 0xDE 0x00
0x2A

The first byte of this code is 0x14, corresponding to the ldnull instruction, which loads a null object reference on the stack. This instruction doesn't take any parameters and its size is 1 byte. You will need to increase the IP value by 1 to get the next opcode, which is 0x0E. This is the opcode for the ldarg.s instruction, which loads a method argument value on the stack. The number of this argument is specified by the instruction's parameter. Since it's in short-parameter form (as noted by the .s suffix), this parameter is a 1-byte integer in the range 0 through 255 (or -128 thru 127, depending on the parameter type). In other words, you have to skip the next byte (0x00) in order to move to the next instruction, whose opcode is 0x28. This is the call instruction, which takes a 4-byte token as a parameter. In order to read the next IL instruction, I'll skip the next 5 bytes (one for the call opcode, 0x28, and 4 bytes for the token, 0x0A000001).

The full implementation is provided by the VerifyReturn function and can be found in the source code which accompanies the article. SEH tables also require some extra work.

In a typical situation you have to modify the sizes of the try blocks and their relative offsets since the original IL code and its size are changing. Then the offsets of the handlers have to be shifted accordingly. Depending on the application's rewrite logic, the handler sizes themselves may also need to be changed.

The simplest rewrite scenario is when you add a prologue before the very first try block and an epilogue right after the very last handler. In such cases, all you have to do is modify the offsets of the try and handler blocks. To illustrate this, let's take a look at Figure 10. (The complete source code and IL code can be found in the cc.il file provided with the article.) Note that I'm using the short-parameter form of the leave instruction to exit the guarded and handler blocks. According to the CLR exception handling rules, the leave instruction is the only correct way to leave SEH blocks. Branching into or out of the SEH block is illegal.

Figure 10 Method with a Simple SEH Block (Close Figure 10)
// IL assembly language
.class public auto ansi CCC extends [mscorlib]System.Object
{
•••
.method public static void TestException( int32 )
{
    .try
    {
        ldnull    // purposely!
        ldarg.s 0 // load param #1 on the stack
        call vararg int32 printf( string, •••, int32 )
        pop       // remove ret value from the stack
        leave.s brOK
    }
    catch [mscorlib]System.Exception
    {
        pop
        ldstr "Exception!"
        call void [mscorlib]System.Console::WriteLine( string )
        leave.s brOK
    }
brOK:
    ret
} // TestException
•••
} // CCC Class

// IL code and opcodes for TestException.
.method /*06000004*/ public static void  TestException(int32 A_0) cil managed
{
  // Code size 25 (0x19)
  .maxstack  8
  .try
  {
    IL_0000:  /* 14   |                  */ ldnull
    IL_0001:  /* 0E   | 00               */ ldarg.s    A_0
    IL_0003:  /* 28   | (0A)000001       */ call       vararg
int32 printf(string, •••, int32)
    IL_0008:  /* 26   |                  */ pop
    IL_0009:  /* DE   | 0D               */ leave.s    IL_0018
  }  // end .try
  catch [mscorlib]System.Exception/* 01000002 */
  {
    IL_000b:  /* 26   |                  */ pop
    IL_000c:  /* 72   | (70)000073       */ ldstr      "Exception!"
    IL_0011:  /* 28   | (0A)000002       */ call       void
[mscorlib]System.Console::WriteLine(string)
    IL_0016:  /* DE   | 00               */ leave.s    IL_0018
  }  // end handler
  IL_0018:  /* 2A   |                    */ ret
} // end of method CCC::TestException

If you located the TestException method body in the cc.exe file (this method's RVA is 0x000020a0) using the steps I described earlier, you should have seen the memory layout for the method header, shown in Figure 11. Pay attention to the Flags field in the method header, which is now set to 0x02 to tell the CLR that an SEH table is present. Now take a look at the method's IL code and SEH sections shown in Figure 12 (I've also added some comments to underscore SEH-related details).

Figure 12 IL Code and SEH Header (Close Figure 12)
// method header
•••
// IL code (DWORD aligned)
0x14 0x0E 0x00 0x28
0x01 0x00 0x00 0x0A 0x26 0xDE 0x0D 0x26
0x72 0x73 0x00 0x00 0x70 0x28 0x02 0x00
0x00 0x0A 0xDE 0x00
0x2A <== this the "ret" instruction
// SEH Header goes here immediately after the code and also DWORD aligned
0x01 0x10 0x00 0x00 <== padded with 2 bytes small SEH header
// small EH clause is located immediately after SEH header
0x00 0x00 0x00 0x00
0x0B 0x0B 0x00 0x0D
0x02 0x00 0x00 0x01
Figure 11 Method Header for TestException (Close Figure 11)

Fat Header Entry (Size) | Value | Note
Header type, Flags, and header size (WORD) | 0x300B (0011000000001011) | Flags field is set to 0x2 (10b) to tell runtime that method body has extra sections
MaxStack (WORD) | 0x8 | Maximum stack size in slots (items)
CodeSize (DWORD) | 0x19 | IL code size (without method header)
LocalVarSigTok (DWORD) | 0x0 | No local variables presented

Due to the DWORD alignment, there's a 3-byte gap (delta) between the last IL instruction and the SEH section header. The first byte of the section header (the Kind field) is 0x01, which means that this is an exception handler section in small format. The small exception handler clause is located right after the section header and is 12 bytes in size.

As you can see, the exception type metadata token is 0x01000002, which refers to the second row in the TypeRef table. This table contains a row for each class defined in another module. In the test application (cc.exe), this token corresponds to the System.Exception class (which is the base class for all CLR exceptions). This can be seen by generating metainfo output using the ILDASM tool.

Let's assume that I'm going to add a prologue, which simply loads the value of the method's argument (an int32) onto the stack and then removes it:

ldarg.0 /*0x02*/
pop /*0x26*/
The epilogue is also very simple—I replaced the ret instruction with a nop and then added a new return statement:
nop /*0x00*/
ret /*0x2A*/
So, the resulting IL code looks like this, where epilogue and prologue are highlighted:
0x02 0x26 0x14 0x0E
0x00 0x28 0x01 0x00 0x00 0x0A 0x26 0xDE
0x0D 0x26 0x72 0x73 0x00 0x00 0x70 0x28
0x02 0x00 0x00 0x0A 0xDE 0x00 0x00 0x00
0x2A

Since I added 2 more bytes to the beginning of the method, the try and catch blocks are both shifted. I also have to modify the TryOffset and HandlerOffset fields in the exception handler clause to let the SEH mechanism work correctly. The modified exception handler clause is shown in Figure 13. Note that the SEH header remains the same.

Figure 13 Modified Exception Handler Clause (Close Figure 13)

Exception Handler Clause Entry | Value | Note
Flags | 0x0000 | COR_ILEXCEPTION_CLAUSE_NONE typed handler from CorExceptionFlag enum (see corhdr.h for details)
TryOffset | 0x0 + 0x02 | Original offset + size of prologue
TryLength | 0xB | Size is the same
HandlerOffset | 0xB + 0x02 | Original offset + size of prologue
HandlerLength | 0xD | Our handler size remains the same
ClassToken | Exception type metadata token (in our example it's 0x01000002) | A System.Exception instance /*01000002*/

In my profiler I've implemented a generic function called FixSEHSections which does all this work (it can be found in the source code which accompanies this article). It analyzes SEH tables and fixes their offsets and sizes depending on the types of exception handler clauses. This function also uses several handy structures provided by the .NET Framework SDK and declared in corhlpr.h. In particular, I'm using the COR_ILMETHOD_TINY, COR_ILMETHOD_FAT, COR_ILMETHOD, and COR_ILMETHOD_DECODER structures, which encapsulate all the work with the method bodies.

Now I'm ready to return to the implementation details of the JITCompilationStarted method, which is outlined in Figure 14. The GetMethodInfoByFunctionID function is a helper that gets the method's information and fills out the CMethodInfo info structure. This structure holds a complete set of data for a method. In order to get the method information, GetMethodInfoByFunctionID calls the core profiler's GetTokenAndMetaDataFromFunction function. This function takes the function ID (provided by the runtime via the JITCompilationStarted call) and returns the corresponding method's metadata token and a special IMetaDataImport interface. The metadata import interface can be used to query method details through the GetMethodProps function. More information about these functions can be found in Profiling.doc, located in the \FrameworkSDK\Tool Developers Guide\docs folder.

Figure 14 JITCompilationStarted Method (Close Figure 14)
extern CMethodInfo* pmi;
extern GetMethodInfoByFunctionID(FunctionID functionID,
    CMethodInfo* pmi );

HRESULT COurProfiler::JITCompilationStarted( FunctionID functionID, BOOL
    fIsSafeToBlock )
{
    // 1. get old method info
    GetMethodInfoByFunctionID( functionID, pmi );

    // 2. generate prologue and epilogue based on method info
    PrepareNewMethodInfo( pmi );

    // 3. get IL code of the old method
    hr = m_pCorProfilerInfo->GetILFunctionBody( pmi->m_moduleID,
        pmi->m_tkMethodDef, &pMethodHeader, &cbMethodSize );

    // 4. create new function's body by adding prologue and epilogue
    void* pNewMethodBody = NULL;
    ULONG cbNewMethodSize = 0;
    hr = CreateILFunctionBody( m_pCorProfilerInfo,
                               pmi,           // method info
                               pMethodHeader, // old method info
                               cbMethodSize,  // old method size
 /* [out] */(void**)&pNewMethodBody,          // newly generated IL body
 /* [out] */ cbNewMethodSize                  // new method size);

    // 5. Set new ILFunctionBody
    hr = m_pCorProfilerInfo->SetILFunctionBody( pmi->m_moduleID,
        pmi->m_tkMethodDef, (LPCBYTE)pNewMethodBody );

    return S_OK;
}

The most difficult part of the JITCompilationStarted implementation is hidden in the CreateILFunctionBody function, which does most of the real work. The part that creates a new method body with the tiny header is outlined in Figure 15. It's self-explanatory. First, it uses prologue and epilogue sizes to calculate the new method size and then it allocates memory for the new method body. Then it copies the prologue and the old method, after which it replaces the last return instruction in the old IL with the nop code (causing the method's execution flow to be passed to the epilogue). Finally, it adds the epilogue.

Figure 15 CreateILFunctionBody Method (Close Figure 15)
static ULONG g_cbFatHeaderSize = 0xC; /*12 bytes = WORD + WORD + DWORD +
                                       DWORD */
static ULONG g_cbTinyHeaderSize = 0x1; /*1 byte*/
static ULONG g_cbMaxTinySize = 0x40; /* 64 bytes */
static WORD     g_wDefMaxStack = 0x8; /*default stack depth in slots*/

HRESULT COurProfiler::CreateILFunctionBody( ICorProfilerInfo*
    pCorProfilerInfo,
    CMethodInfo* pmi,         // method info
    LPCBYTE pMethodHeader,    // old method info, points to method header
    const ULONG cbMethodSize,          // old method size
    /* [out] */void** ppNewMethodBody, // newly generated IL body,
                                       // points to method header
    /* [out] */ULONG& cbNewMethodSize  // new method size
)
{
    // 1. get malloc interface
    IMethodMalloc* pMethodMalloc = NULL;
    hr = pCorProfilerInfo->GetILFunctionBodyAllocator( pmi->m_moduleID,
        &pMethodMalloc );

    // 2. calculate sizes and allocate memory for new function
    // 2.1. Get size of our prologue
    ULONG cbPrologSize = GetPrologSize(•••)
    // 2.2. Get epilogue size
    ULONG cbEpilogSize = GetEpilogSize(•••);

    // 2.3. Calculate new method size (including header)
    cbNewMethodSize = g_cbTinyHeaderSize; // TINY HEADER
    cbNewMethodSize += ( cbPrologSize + cbEpilogSize );
        // our prologue + epilogue
    cbNewMethodSize += ( cbMethodSize - g_cbTinyHeaderSize );
        // old method size w/o TINY HEADER
    ULONG cbNewMethodSizeWithoutHeader = ( cbNewMethodSize -
        g_cbTinyHeaderSize );
    ULONG cbMethodSizeWithoutHeader = ( cbMethodSize  -
        g_cbTinyHeaderSize );
    // 2.4. allocate memory
    void* pNewMethodBody = pMethodMalloc->Alloc( cbNewMethodSize );
    pMethodMalloc->Release();

    // 3. copy over new function
    // 3.1. "Create" a header and specify the new method size
    BYTE tinyHeader = (BYTE)cbNewMethodSizeWithoutHeader;
    // shift left 2 bits ( so, XXXXXXXXb becomes XXXXXX00b )
    // and set first 2 bits holding tiny header type ( XXXXXX10b = 0x2 )
    tinyHeader = ( tinyHeader << 2 ) | CorILMethod_TinyFormat;
    // 3.2 Copy header
    memcpy( pNewMethodBody, (void*)&tinyHeader, sizeof(BYTE) );
    // 3.3 copy our prologue
    memcpy( (BYTE*)pNewMethodBody + g_cbTinyHeaderSize, (void*)Prolog,
        cbPrologSize );
    // 3.4. copy old method body
    memcpy( (BYTE*)pNewMethodBody + g_cbTinyHeaderSize + cbPrologSize,
        (BYTE*)pMethodHeader + g_cbTinyHeaderSize,
        cbMethodSizeWithoutHeader );
    // 3.5. replace last 'RET' statement with 'NOP'
    BYTE OpCodeNop = g_ILcodes[ CEE_NOP ].OpCode2;
    memcpy( (BYTE*)pNewMethodBody + g_cbTinyHeaderSize + cbPrologSize +
        cbMethodSizeWithoutHeader -1, (void*)&OpCodeNop, sizeof(BYTE) );
    // 3.6. copy our epilogue
    memcpy( (BYTE*)pNewMethodBody + g_cbTinyHeaderSize + cbPrologSize +
        cbMethodSizeWithoutHeader, (void*) Epilog, cbEpilogSize );
    // everything went OK
    *ppNewMethodBody = pNewMethodBody;
    return S_OK;
}

The same function for a method with the fat header is much more complicated. First, you need to take care of the method header—the MaxStack and CodeSize fields have to be modified, which is the easiest part. Second, you have to walk through all SEH headers and clauses and make sure they still contain valid offsets and sizes, despite the fact that the original IL code gets shifted. You also have to correctly calculate the gap between the IL and the first SEH header and make sure that all the SEH sections are properly aligned. You can find the complete version of the function in the source code which accompanies this article.

More Advanced Techniques

I will now consider more complex techniques such as adding local variables and exception handlers to a method and creating new methods at run time. I'll start with local variables.

The local variables of a method are coded as a signature and the corresponding record is stored in the StandAloneSig metadata table as 0x11XXXXXX formatted tokens. The table has only one column which stores offsets into the special metadata stream called the Binary Large Object (BLOB) heap. (StandAloneSig also holds signatures for indirect calls that use the IL instruction calls.)

For example, consider the following method in IL assembly language with two local variables of type int32:

// IL assembly language
.method public static int32 SomeFunction( int32, int32 )
{
.maxstack 10
// information about those variables is stored in StandAloneSig
// "init" means that all local variables
// must be initialized
    .locals init ( int32 nParam1, int32 nParam2 )
    •••
}
When the method gets compiled to IL, the compiler will create a signature based on the number of local variables (2) and their types (both int32). At the very beginning of the signature the compiler will also add a special byte to identify the type of the signature, which is 0x07 (IMAGE_CEE_CS_CALLCONV_LOCAL_SIG) for local variables.

Taking into account that the int32 type is represented by the ELEMENT_TYPE_I4 value (0x08), the resulting signature would look like this:

0x07 0x02 0x08 0x08
Next, the signature will be stored in the BLOB stream and the corresponding record will be created in the StandAloneSig table. The compiler will also update the LocalVarSigTok field of the method header to reference the newly created record. Since I've specified the init keyword in the local variables declaration, the Flags value in the method header should be set to 0x04. At run time it will indicate to the JIT compiler that local variables have to be initialized by calling their default constructors. Otherwise, the code won't pass the runtime verification process. Thus, in order to add local variables to a method you must modify the StandAloneSig table. This is also true when you modify methods at run time. Before you change the method and call ICorProfilerInfo::SetILFunctionBody you have to generate a new local signature and dynamically add a new record to the StandAloneSig table.

Fortunately, the CLR supports an interface called IMetaDataEmit, through which the profiler can modify the existing metadata at run time (unlike Reflection.Emit). To obtain a pointer to this interface, you'll need to call the ICorProfilerInfo::GetModuleMetaData method with dwOpenFlags of ofRead | ofWrite and an riid of IID_IMetaDataEmit. Here's the function's format:

HRESULT GetModuleMetaData( ModuleID moduleId, DWORD dwOpenFlags,
                           REFIID riid, IUnknown **ppOut )
It will give you back a writeable metadata interface (Metadata Emitter), which is quite powerful. It's defined in Cor.h and has several important methods (see Figure 16).

Figure 16 IMetaDataEmit Interface (Close Figure 16)
// Cor.h
DECLARE_INTERFACE_(IMetaDataEmit, IUnknown)
{
•••

    STDMETHOD(DefineTypeDef)(         // S_OK or error
        LPCWSTR     szTypeDef,        // [IN] Name of TypeDef
        DWORD       dwTypeDefFlags,   // [IN] CustomAttribute flags
        mdToken     tkExtends,        // [IN] extends this TypeDef or
                                      // typeref
        mdToken     rtkImplements[],  // [IN] Implements interfaces
        mdTypeDef   *ptd) PURE;       // [OUT] Put TypeDef token here

    STDMETHOD(DefineMethod)(          // S_OK or error
        mdTypeDef   td,               // Parent TypeDef
        LPCWSTR     szName,           // Name of member
        DWORD       dwMethodFlags,    // Member attributes
        PCCOR_SIGNATURE pvSigBlob,    // [IN] point to a BLOB value of
                                      // CLR signature
        ULONG       cbSigBlob,        // [IN] count of bytes in the
                                      // signature BLOB
        ULONG       ulCodeRVA,
        DWORD       dwImplFlags,
        mdMethodDef *pmd) PURE;       // Put member token here

    STDMETHOD(GetTokenFromSig)(       // S_OK or error
        PCCOR_SIGNATURE pvSig,        // [IN] Signature to define
        ULONG       cbSig,            // [IN] Size of signature data
        mdSignature *pmsig) PURE;     // [OUT] returned signature token

•••
};      // IMetaDataEmit

Using this interface, you will be able to create new metadata tokens on the fly. The entire process of the dynamic creation of local variables for a given method can be summarized as follows:

  1. Get the IMetaDataEmit interface by calling ICorProfilerInfo::GetModuleMetaData.
  2. Allocate (as a sequence of bytes) the local variable's signature and fill it out.
  3. Add a signature to the StandAloneSig table using the IMetaDataEmit::GetTokenFromSig method, which returns a metadata token for the newly added record.
  4. Modify LocalVarSigTok to point to the new token. The Flags field has to be set to 0x04 to initialize local variables.
The complete implementation is provided by the CreateILFunctionBodyWithLocalVariables function and can be found in the source code that accompanies the article.

Adding a new method requires some extra work and can be done by calling IMetaDataEmit::DefineMethod. Note that any changes to class metadata—adding new methods or whole new classes—should be done as early as possible in the run of the program. The ideal location is during the ModuleLoadFinished callback for the module into which you want to add the new class or method. This process can be described as follows:

  1. Obtain the IMetaDataEmit interface.
  2. Create the local variable signatures and add them to the metadata (as described in the previous section).
  3. Create the new method body (see CreateILFunctionBody for details) and modify its header to point to the correct signature (with the LocalVarSigTok and Flags entries).
  4. Calculate the method's signature, attributes, and implementation flags. Note that unlike the local variable signatures, the method's signature is not coded as a token. Therefore, you don't have to store it as a record in a separate table before you call DefineMethod.
  5. Call the IMetaDataEmit::DefineMethod function, which modifies the Method metadata table and does the actual work, storing the method's signature, attributes, and implementation flags in the Signature, Flags, and ImplFlags columns, respectively.
The source code for the CreateILFunctionBodyWithLocalVariables function will provide additional details.

To understand the process of adding new exception handlers to methods, first consider a generic method with no exception handler as shown here:

// IL assembly language
.method ••• void SomeMethod( ••• )
{
    •••
    // method body
    •••

ret
} // SomeMethod

I'm going to modify the method by adding the simplest possible catch handler (see Figure 17). Note that in this example I'm using the short-parameter leave instruction, which requires two bytes:

IL_01a7:  /* 26   |                  */ pop
IL_01a8:  /* DE   | 00               */ leave.s    IL_01aa
The long-parameter instruction would require 5 bytes:
IL_01a7:  /* 26   |                  */ pop
IL_01a8:  /* DE   | 00000000         */ leave        IL_01aa
Since I'm replacing the return statement in the original IL code with the short leave instruction, which is 1 byte longer, the guarded block will also grow by 1 byte.

Figure 17 Method with Dynamically Added SEH Handler (Close Figure 17)
// IL assembly language
.method ••• void SomeMethod ( ••• )
{
    .try
    {
        •••
    // original method body without return statement
        •••

        leave.s brOK // short form of SEH Block exiting instruction
    }
    catch [mscorlib]System.Exception
    {
        pop // get rid of the reference to the intercepted exception
            // (in our case it's a System.Exception instance
        leave.s brOK
    }
brOK:
    ret
} // SomeMethod

In order to add an exception handler you will need to:

  1. Set the Flags field in the method header to 0x02, since the method has extra sections.
  2. Find the last return instruction (as you did in the regular case of emitting epilogues and prologues) and replace it with the leave instruction (which exits SEH blocks).
  3. Add the exception handler block (for example, the catch [mscorlib]System.Exception {...} block).
  4. Add a new SEH header if the method doesn't have one.
  5. Add either a new small (if the SEH header is small) or fat exception handler clause and set its field.
If you also added a prologue and an epilogue, the corresponding exception handler clause would have the layout shown in Figure 18.

Figure 18 Generic Exception Handler Clause (Close Figure 18)

Exception Handler Clause Entry | Value
Flags | 0x0000
TryOffset | Prologue size + 0x0
TryLength | Short form of leave instruction: Prologue size + Size of the original code + 1 Long form of leave instruction: Prologue size + Size of the original code + 4
HandlerOffset | Short form of leave instruction: Prologue size + Size of the original code + 1 Long form of leave instruction: Prologue size + Size of the original code + 4
HandlerLength | The handler size
ClassToken | Exception type metadata token

Profiler Example

The profiler example provided with the article inserts a simple prologue and an epilogue for each method of a given class. The code illustrates most of the techniques I've discussed in the article. It also produces a log file, so you can see what's going on in the CChecker.log file. The profiler requires the following environment variables to be set:

set Cor_Enable_Profiling=0x1
set COR_PROFILER={97D8965A-8686-4639-9C24-E1F6D13EE105}
set COR_PROFILER_DLL=<your complete path>\CChecker.dll
set CC_PROFILER_ROTOR_APP=YourApp.exe
The COR_PROFILER_DLL and CC_PROFILER_ROTOR_APP variables are specific to Rotor. The first variable sets the full path to your profiler DLL and makes it available for Rotor. The second is used by the profiler itself and specifies the assembly name being instrumented. For example, if you wanted to profile the hello.exe application in the Rotor environment, you would have to set CC_PROFILER_ROTOR_APP like this:
set CC_PROFILER_ROTOR_APP=hello.exe

Conclusion

This article showed some of the advanced features found in the common language runtime. I've explained how to use the Profiling API and the internal structures of the CLR to dynamically modify intermediate language code, preserving the identity of the classes whose methods are instrumented. These techniques demonstrate how to instrument code for gathering metrics that aren't possible with the events provided in the rest of the Profiling API. In addition, the example provided gave you some insight into the internal workings of the CLR and will allow you to tweak your own advanced .NET Framework-targeted applications.

For related articles see:
The Implementation of Model Constraints in .NET
Inside Microsoft .NET IL Assembler by Serge Lidin (Microsoft Press, 2002)
Avoiding DLL Hell: Introducing Application Metadata in the Microsoft .NET Framework
Under the Hood: The .NET Profiling API and the DNProfiler Tool
The Microsoft Shared Source CLI Implementation
Rotor: Shared Source CLI Provides Source Code for a FreeBSD Implementation of .NET
Using Reflection Emit to Cache .NET Assemblies

Aleksandr Mikunov is a senior software engineer at Compuware Corporation. His background is in developing relational databases and n-tier applications on Windows platforms. Reach him at Aleksandr.Mikunov@compuware.com.


Coroutines: Implementing Coroutines for .NET by Wrapping the Unmanaged Fiber API

[ Editor's Update - 9/16/2005: The solution described in this article relies on undocumented functionality that is not supported by Microsoft at this time. This article is intended for informational purposes only, and its contents should not be used to create production code.]
Coroutines

Implementing Coroutines for .NET by Wrapping the Unmanaged Fiber API

Ajai Shankar
This article assumes you're familiar with Managed C++ and .NET
Level of Difficulty 1 2 3
Code download available at: CoroutinesinNET.exe (135KB)
SUMMARY

Coroutines are a powerful feature of many programming languages including CLU, Scheme, Python, Ruby, and ICON. Coroutines can save processor overhead and reduce redundancy because they allow you to stop execution of a procedure midstream, return a value, and resume exactly where the procedure left off.

This article shows how coroutines can be implemented for the .NET Framework by using the Fiber API and Managed Extensions for C++, and how they can be easily used with other .NET-compliant languages. This article also shows a sophisticated use of the runtime host for running multiple managed threads on a single OS thread.

Contents
Coroutines and the .NET Framework
Fibers and Threads
Implementation Details
A Generator and Tree Matching Example
Conclusion

F
ibers—lightweight threading objects that can be used from 32-bit versions of Windows®—are useful in many scenarios. Since threads are precious resources, sometimes you don't want to dedicate an entire OS thread to a simple task. Fibers allow you control the scheduling of tasks more closely than threads because the OS isn't in charge of them, you are. Since they have less overhead, they're also faster when you switch contexts. In addition, because you control fibers, you can often track down synchronization problems more easily with them.

Coroutines are a code-based concept, similar to fibers, that have been implemented in several modern languages like CLU, Scheme, Python, Ruby, and ICON. Simply put, a coroutine is a method that can stop in mid-execution and provide a value to the caller without returning program flow. Since coroutines and fibers are parallel concepts, the two should complement each other.

Coroutines are not currently implemented natively under the Microsoft® .NET Framework. (A C# version is planned for future release.) However, with a bit of clever coding involving the Win32® Fibers API and Managed Extensions for C++, you can create your own coroutine libraries today and take full advantage of this advanced programming concept.

Coroutines can be thought of as methods which, instead of returning to the caller, stop dead in the middle of processing and yield a value to the caller. The next time the coroutine is called, it resumes where it left off until it yields another value. For example, here is a tiny program, written without coroutines, that iterates over the elements of an array:

class Iter {
    public object Next() {
        return array[ndx++];
    }
    object[] array = new object[] {1, 2, 3, 4};
    int ndx;
}

The array this code iterates over and the current index into the array are fields of the Iter class. The Next method returns the next element of the array. If a language supports coroutines, the same program could be written as:

class CorIter {
    public void Next() {
        object[] array = new object[] {1, 2, 3, 4};
        for(int ndx = 0; true; ++ndx)
            Yield(arr[ndx]);
    }
}

The important thing to note here is that each time the Next method is called, rather than returning, it saves its current execution state (locals like array, ndx, and the next intermediate language instruction) and calls Yield to provide the next element of the array to the caller. The first time the method is called, the index is 0 and the function yields the first element of the array. As the method is called subsequently, it resumes where it left off, incrementing the index and yielding the next element.

Coroutines don't really accomplish anything in a trivial example like this. But what if you need to write a recursive method that does an in-order traversal of a binary tree in order to return the nodes one at a time? Without coroutines this would be rather difficult to implement. Either you would have to remove the recursion manually and write an iterator similar to the one in the C++ Standard Template Library, or resort to the use of callbacks as each node is encountered. Both options put a burden on the programmer. In addition, the first option creates more work for the coder who implements the iterator while the callback option has its own problems for the user.

The presence of coroutines in a language is a convenient alternative to both of these options because they let you yield the nodes as they are encountered. You can see a similar use of coroutines for processing the XML DOM at Streamlining DOM XML processing with Python. The State Threads library from Netscape (http://state-threads.sourceforge.net) implements coroutines in C using setjmp and longjmp calls.

These two coroutine examples also show the concept of Python generators, which yield the next value of an array or binary tree as they are called. Using coroutines, you can enumerate the nodes of a binary tree from within C# with a simple foreach statement:

foreach(Node node in tree) {
    // process the node
}

One of the real advantages of coroutines is that they can be aborted if the rest of the computation is deemed unnecessary. For example, to see if two trees contain the leaves in the same order (fringe-matching), I can create a coroutine that yields the leaves of the first tree and another that yields the leaves of the second tree. The moment I find a point where the leaves don't match, I can abort both of the coroutines and avoid walking the rest of the trees.

Coroutines and the .NET Framework

I made a number of major design and implementation decisions when implementing coroutines in the .NET Framework. On the presentation level, the interface presented to the user should be very clean, similar to those in other languages. Users should always have the ability to abort coroutines. A coroutine should never cause an application to abort, even if it generates an uncaught exception.

On the implementation level I used the Win32 Fiber API to implement coroutines, since the common language runtime (CLR) does not provide a way to create separate execution paths or to save the current execution state and resume it later. Because of the additional resource overhead, I decided against using threads to implement coroutines. I also wanted to ensure that switching between coroutines would not cause any problems within the CLR and that there would be proper cleanup of internal CLR states.

I used Managed Extensions for C++ for the core implementation since it allows you to switch easily between managed and unmanaged code. Also, as I will explain later, I had to hook in the CLR host from unmanaged code to notify the runtime of the presence of fibers.

Fibers and Threads

Fibers in Windows NT® can be thought of as lightweight threads that have to be manually scheduled. When a fiber is created, it is passed a fiber-start function. The OS then assigns it a separate stack and sets up execution to begin at this fiber-start function. To schedule this fiber, you need to switch to it manually. The OS saves the execution state of the currently running fiber, then starts executing the new fiber. When this fiber switches to another one, its execution state is similarly saved so that it can be resumed later. Many fibers can be scheduled on the same thread, and a fiber can also move between threads. If the currently running fiber exits, then the thread running that fiber also exits. The Platform SDK documentation explains fibers in more detail (see Fibers).

Each fiber has a separate stack and a set of registers. If a fiber is running managed code, then it is possible to save the managed execution state of a fiber, pass program control to another managed fiber that called it (a yield), and resume this state later. This is how coroutines can be implemented as a fiber in the .NET Framework.

I created a class called Fiber that contains an abstract method named Run and the methods shown in Figure 1. The Fiber constructor sets up execution to begin at a function which calls the abstract method Run. A call to the Resume method switches to this fiber, which will in turn call Run. A coroutine inherits from the Fiber class and provides the implementation of the Run method. Within the Run method and other methods called by it, the coroutine yields to the previous fiber.

Figure 1 The Fiber Class (Close Figure 1)
enum FiberStateEnum {
    FiberCreated, FiberRunning, FiberStopPending, FiberStopped
};

__gc __abstract public class Fiber : public System::IDisposable {
public:
    Fiber() {}
    System::Object* Resume() {}
    void Dispose() {}
protected:
    virtual void Run() = 0;
    void Yield(System::Object *obj) {}
private:
    void *fiber; // this fiber
void *previousfiber; // the fiber that scheduled this fiber
    FiberStateEnum state; // whether the fiber is running or has stopped.
    System::Object *retval; // the value that Resume returns
};

To add some syntactic sugar to this class, I created a delegate called Coroutine, which wraps the Resume method of a Fiber object. I also added an implicit conversion operator in the Fiber class:

public __delegate System::Object* Coroutine();

static Coroutine* op_Implicit(Fiber *obj) {
    return new Coroutine(obj, &Fiber::Resume);
}

Using the Fiber class and this Coroutine delegate, the code to iterate over an array can now be written as:

class CorIter : Fiber {
    protected override void Run() {
        object[] array = new object[] {1, 2, 3, 4};
        for(int ndx = 0; true; ++ndx)
            Yield(arr[ndx]);
    }
}

Coroutine next = new CorIter();
Object o = next();
Each time the next coroutine is called, the fiber is resumed and execution continues at the state where it left off. Once the fiber yields the next array element, that value is then passed back to the calling fiber.

Implementation Details

Before I explain the code for the Fiber class in detail, let's review the interaction between fibers and the CLR. At first I had a very tough time trying to use fibers directly from within the .NET Framework, especially with exception handling. Then I discovered that the Platform Abstraction Layer (PAL) and the runtime make heavy use of thread local storage (TLS), as does the .NET exception-handling mechanism. The list of exception handlers for a managed thread is stored in a TLS slot. For more information on the PAL, see Rotor: Shared Source CLI Provides Source Code for a FreeBSD Implementation of .NET.

Since fibers running managed code all exist on the same thread, switching between them causes problems for the exception handlers on a "managed" fiber. To avoid this I hook in the CLR host from unmanaged code, and using four undocumented APIs I inform the runtime about the presence of fibers, and notify it just before switching between fibers. Here is the static constructor that binds to the CLR host:

#pragma unmanaged

ICorRuntimeHost *corhost;
void initialize_corhost() {
    CorBindToCurrentRuntime(0, CLSID_CorRuntimeHost,
        IID_ICorRuntimeHost, (void**) &corhost);
}

#pragma managed

static Fiber() { initialize_corhost(); }
The following shows the fiber constructor and the unmanaged fiber-start function:
Fiber() : retval(0), state(FiberCreated) {
  void *objptr = (void*) GCHandle::op_Explicit(GCHandle::Alloc(this));
  fiber = CreateFiber(0, unmanaged_fiberproc, objptr);
}
The constructor for the Fiber object creates a new fiber, which once scheduled begins executing the unmanaged_fiberproc function. When a fiber is created, Windows allows you to pass context information to the fiber procedure. When the fiber is scheduled, it can access the information that was passed to it. The constructor passes a pointer to the Fiber object to the unmanaged function. Since it is not possible to pass managed pointers directly to unmanaged code, I used the GCHandle class in the System:: Runtime::InteropServices namespace to get an unmanaged pointer to the managed Fiber object (see Figure 2).

Figure 2 Passing a Pointer (Close Figure 2)
#pragma unmanaged

VOID CALLBACK unmanaged_fiberproc(PVOID objptr) {
    corhost->CreateLogicalThreadState();
    void *previousfiber = fibermain(objptr);
    corhost->DeleteLogicalThreadState();
    SwitchToFiber(previousfiber);
}

#pragma managed

void* fibermain(void* objptr) {
    System::IntPtr ptr = (System::IntPtr) objptr;
    GCHandle g = GCHandle::op_Explicit(ptr);
    Fiber *fiber = static_cast<Fiber*>(g.Target);
    g.Free();
    return fiber->main();
}

Now comes the tricky part. Once I switched to the fiber and the unmanaged code started executing, I needed to warn the runtime about the newly created fiber. The undocumented function ICorRuntimeHost::CreateLogicalThread tells the runtime to set up a managed thread, complete with its own exception list and other data that is internal to the CLR for this fiber. This means that there is a managed thread corresponding to each fiber. Since many fibers can be scheduled on the same OS thread, it implies that multiple managed threads can be run on the same OS thread. Also, as a fiber can be scheduled later on another OS thread, it is possible for a managed thread to move between different OS threads. The sample application, threads.cs, which is available with the code download for this article, demonstrates this concept (see the link at the top of this article).

The unmanaged function then calls the managed fibermain function, passing it the pointer to the managed Fiber object. Once fibermain returns, the CLR is out of the picture. Control returns to the unmanaged code, where I deleted the managed thread state that I created for this fiber. I then switch back to the fiber that last scheduled this fiber.

The managed fibermain function simply gets the Fiber object and calls its private main method:

void* main() {
    state = FiberRunning;
    try {
        Run();
    } catch(System::Object *x) {
        System::Console::Error->WriteLine(
            S"\nFIBERS.DLL: main Caught {0}", x);
    }
    state = FiberStopped;
    retval = 0;
    return previousfiber;
}
The main method sets the state of the fiber to "running," and sets up an exception handler to catch any exceptions. Unlike in C#, Managed C++ lets me catch exceptions that are not derived from the Exception class. I will put this feature to good use when I implement the Dispose method. Run could return normally or throw an exception. In either case, fibermain gets control, sets the state to stopped, and returns the fiber that last scheduled this fiber.

From managed code, the user would switch between fibers using the Resume and Yield methods. As shown earlier, the CLR needs to be informed before switching between fibers. You have to tell the CLR to save the managed thread state for the currently running fiber before doing an OS-level SwitchToFiber. The unmanaged function CorSwitchToFiber does this, as you can see:

#pragma unmanaged

void CorSwitchToFiber(void *fiber) {
    DWORD *cookie;
    corhost->SwitchOutLogicalThreadState(&cookie);
    SwitchToFiber(fiber);
    corhost->SwitchInLogicalThreadState(cookie);
}

CorSwitchToFiber saves the thread state in a local variable using the ICorRuntimeHost::SwitchOutLogicalThreadState method, which can be found in the Shared Source CLI(remember, each fiber has its own stack). After saving the state, it does an OS-level fiber switch. Later, when this fiber is switched in, either through a Yield or a Resume, the CLR thread state is restored:

System::Object* Resume() {
    if(!fiber || state == FiberStopped)
        return 0;
    previousfiber = GetCurrentFiber();
    CorSwitchToFiber(fiber);
    return retval;
}
The Resume method first checks to see if the fiber has already stopped; otherwise it sets the previousfiber field to the currently running fiber and does a CorSwitchToFiber.

Yield can be called either from Run (which the coroutine implements) or from a method that Run calls. Yield switches to the fiber that scheduled the current fiber, passing it the value to be yielded, as shown in the following code:

__gc private struct StopFiber {};

void Yield(System::Object *obj) {
    retval = obj;
    CorSwitchToFiber(previousfiber);
    if(state == FiberStopPending)
        throw new StopFiber;
}
Yield sets the retval field and switches to the previous fiber (which was paused in the call to Resume), ready to execute the return retval statement. Note that there is no need for synchronization primitives, as fibers are cooperatively scheduled and run on the same thread. If Run simply returns without yielding a value, or if there is an exception, a null value is returned from fibermain.

The only pieces remaining in this implementation of Yield are the check for FiberStopPending and the throw of StopFiber. One of my design considerations was that a coroutine could be aborted and its remaining computation abandoned. It is not possible to simply delete the fiber because the CLR will still hold some thread and execution state info for the fiber. Because I needed the unmanaged fiber start procedure to get control (so that it could delete the CLR thread state created for the fiber), I implemented the IDisposable::Dispose method:

void Dispose() {
    if(state == FiberRunning) {
        previousfiber = GetCurrentFiber();
        state = FiberStopPending;
        CorSwitchToFiber(fiber);
    }
    DeleteFiber(fiber);
}

Dispose first checks to see if the fiber has run to completion. If not, it sets the state to FiberStopPending and switches to the fiber. Because this fiber was running earlier, it has obviously yielded. So Yield is all set to execute the statement:

if(state == FiberStopPending)
    throw new StopFiber;
Since this exception is not derived from the Exception class, it cannot be caught from C#, and so gets propagated to the main method that called run. As shown earlier, the main method sets the fiber state to stopped and also clears out the retval. In this way, control comes back cleanly to the unmanaged fiber function, which deletes the CLR thread state for this fiber and switches back to the fiber that called Dispose.

Since the CLR is no longer in the picture when control comes back to the unmanaged fiber-start function, I can be sure that the internal CLR state for this fiber has been properly cleaned up. But the use of an exception (derived from System.Object) to ensure proper cleanup imposes a small restriction on the use of this class. If this implementation is used from a .NET Framework-compliant language that supports catching an exception not derived from the Exception class, then the overridden Run should not have a generic catch clause.

A Generator and Tree Matching Example

Now that all the hard stuff has been done in Managed Extensions for C++, I can use the power of coroutines from other languages. I used the tree-matching example from "Teach Yourself Scheme in Fixnum Days" by Dorai Sitaram and implemented it in C#. As you will see, it is also easy to write a generator that enumerates the leaves of the tree. The two trees have matching fringes, regardless of depth, if two leaves are in the same order, as shown in Figure 3.

Figure 3 Two Trees
Figure 3 Two Trees

To see if the fringes match without using coroutines, I would have to flatten the trees by traversing them and putting their leaves in a list or an array. After that, I could loop through the lists in order to see if they matched. Even if the first leaves didn't match, I've already incurred the processing overhead of walking each tree completely to flatten it.

A coroutine that walks the first tree could yield each leaf as it is encountered. A second coroutine could yield the leaves of the second tree. At the first instance where the two leaves do not match, you are finished with the computation and both coroutines are aborted. The following code, which uses the Visitor design pattern, illustrates this point:

interface TreeVisitor {
    void Visit(Branch branch);
    void Visit(Leaf leaf);
}

interface Tree {
    void Accept(TreeVisitor visitor);
}
This uses an interface called Tree with a single method which accepts a TreeVisitor. The Branch and Leaf classes implement the Tree interface. A Branch is a recursive data type that has a left and right node of type Tree, either of which could be another Branch or a Leaf, as shown here:
class Branch : Tree {
    public Branch(Tree l, Tree r) { left = l; right = r; }
    public void Accept(TreeVisitor visitor) {
        visitor.Visit(this);
    }
    public Tree left, right;
}

class Leaf : Tree {
    public Leaf(Object val) { value = val; }
    public void Accept(TreeVisitor visitor) {
        visitor.Visit(this);
    }
    public Object value;
}
As C# does not support multiple dispatch, I simulated it by making both Branch and Leaf override the Accept method and called the Visit(Branch) or Visit(Leaf) methods of the TreeVisitor.

The tree visitor that I have is a coroutine. For a branch it recursively visits the left and right subtree, while for a leaf it just yields its value, as shown in Figure 4.

Figure 4 Traversing a Tree (Close Figure 4)
class LeafGenerator : Fiber, TreeVisitor, IEnumerator {

    public enum Order { Left, Right };

    public LeafGenerator(Tree _tree, Order _order) {
        tree = _tree;
        order = _order;
    }
    public void Visit(Branch branch) {
        if(order == Order.Left) {
            branch.left.Accept(this);
            branch.right.Accept(this);
        } else {
            branch.right.Accept(this);
            branch.left.Accept(this);
        }
    }

    public void Visit(Leaf leaf) {
        Yield(leaf.value);
    }

    protected override void Run() {
        tree.Accept(this);
    }
    private Tree tree;
    private Object curobj;
    private Order order;
}

The Visit(Branch) method in the code I just showed recursively visits the left and right nodes in the order specified by the user. If the left or right node happens to be a Leaf, then its value is yielded. If the node is a Branch, the Visit(Branch) method will be recursively called again. The recursion stops once a Leaf is encountered.

The Run method that the coroutine implements kicks off the recursive process of walking the tree by asking the tree to accept itself as the visitor. This coroutine can be used as a generator in a statement such as this:

foreach(Leaf leaf in tree) {
}

I also implemented the System::Collections::IEnumerator interface:

public Object Current { get { return curobj; } }
public bool MoveNext() {
    curobj = Resume();
    return curobj != null;
}
MoveNext calls Resume, which yields either the next leaf or null if the tree is exhausted. This value is stored in the field curobj, and is returned when the Current property is called.

Compilers automatically generate code to call the IDisposable::Dispose method if the enumerator supports it. As the Fiber class implements the disposable interface, proper cleanup is made even if the user breaks out of the enumeration.

The tree-matching code is very simple. If I have two coroutines (leaf generators) that yield the leaves of the first and the second tree, I call each coroutine in succession, then check the values that they yield. If both return null (indicating that all the nodes of the trees are exhausted), then the trees match. If one coroutine yields null and another yields a Leaf it means that the first tree has been fully traversed and the second tree still has branches or leaves to be visited, so the trees do not match. If the current leaves are the same, then I recursively call the procedure to check the next leaves. If they are not the same, I immediately return a nonsuccessful match, as shown here:

static bool MatchFringe(Coroutine nexta, Coroutine nextb) {
    Object a = nexta();
    Object b = nextb();
    if(a == null && b == null)
        return true;
    if(a == null || b == null)
        return false;
    if(a.Equals(b))
        return MatchFringe(nexta, nextb);
    else
        return false;
}

If I find that the trees do not match, I can abort both fibers by calling the Dispose method. The complete code for the tree matching example can be found in the code download for this article in the file tree.cs.

Conclusion

The implementation of fibers and coroutines I have demonstrated in this article could easily be used to port event-driven applications to a multithreaded model that more closely matches the underlying OS, but without the associated overhead of extra threads. For example, a complex application that implements asynchronous callbacks (like an Internet download app) could be transformed into a simpler coroutine-based application. This implementation could also be used internally by a compiler that targets the CLR as the runtime for a language that provides coroutine support.

I have not touched upon all possible angles of fibers as used with the .NET Framework. There are potential issues when fibers are used with serviced components, and the interaction between fibers and .NET exceptions can be tricky. In unmanaged structured exception handling, a fiber switch within a catch or finally block could cause problems because these blocks are run on a separate shared stack.

One other issue I've found is that coroutine switches can deadlock from within the garbage collector thread. The CLR freezes all managed threads when garbage collection is running, so I did not implement a Finalize method to automatically delete the fiber. If a fiber is still running, the burden falls on the programmer to call Dispose once a coroutine is no longer needed. Since each fiber has a corresponding managed thread, there's no way to switch to the fiber and schedule it from within the garbage collection thread. I did not implement a Finalize method that calls Dispose because Dispose switches to the fiber to stop it. Finally, note that this technique uses internal calls that may change in the future. If you use these techniques, you're on your own!

For related articles see:
Streamlining DOM XML processing with Python

For background information see:
Teach Yourself Scheme in Fixnum Days
Threads and Threading

Ajai Shankar is a consultant for IRIS Software Inc. in NJ and has designed and implemented projects for Wal-Mart, Ford, and MasterCard in a variety of programming languages. He can be reached at ajai_shankar@hotmail.com.


Visual C++ 6.0: Don't Let Memory Allocation Failures Crash Your Legacy STL Application

New information has been added to this article since publication.
Refer to the Editor's Update below.

Visual C++ 6.0

Don't Let Memory Allocation Failures Crash Your Legacy STL Application

James Hebben
This article assumes you're familiar with C++, STL, and MFC
Level of Difficulty 1 2 3
Code download available at: LegacySTLFix.exe (111KB)
SUMMARY

Most C++ developers make extensive use of the Standard Template Library (STL) in their code. If you are one of them and are using STL and Visual C++ 6.0 directly out of the box, your application is at high risk of crashing under low memory conditions. The problem arises because checking for failure of operator new is such an uncommon practice. To make things worse, when new does fail, the response is not standard. Some language compilers return NULL while others throw an exception.

In addition, if you are using STL in an MFC project, be aware that MFC has its own set of rules. This article discusses these problems, explains how the default behavior has changed in Visual C++ .NET 2003, and outlines the changes you must make if you're using Visual C++ 6.0 so that you can safely use the STL when operator new fails.

Contents
Background
Operator New Returns NULL
Standard Template Library
Visual C++ 6.0 and Operator New
Fixing Operator New
When Operator new(std::nothrow) Throws an Exception Anyway
Conclusion

H
ow many developers check for the failure of operator new? Is there a need to always check for failure? I have seen large and complex C++ projects, written using Visual C++® 6.0, where not a single check in the entire code base is made to see if new returns NULL. Note that I said checking for new returning NULL. The default behavior of operator new in all versions of Visual C++ through version 6.0 is to return NULL rather than throw an exception when it fails. (See Knowledge Base article 167733 for more information, but do not implement the solution given there. I will explain why you shouldn't later in this article.)

The default behavior has changed for Visual C++ .NET, both for version 7.0 (Visual C++ .NET 2002) and 7.1 (Visual C++ .NET 2003), which throws an exception when operator new fails. While this new behavior under the Microsoft® .NET Framework adheres to the C++ standard and should be welcomed, be aware that it can break the runtime behavior of all migrated Visual C++ 6.0-style code that does not anticipate operator new throwing an exception. If you are developing with Visual C++ .NET, you will find that the issues raised here have been fixed. If you're not yet using one of the .NET Framework versions, this article looks at the serious implications and incompatibilities of operator new returning NULL, a problem that applies to all versions of the Visual C++ compiler up to and including version 6.0.

[ Editor's Update - 2/28/2005: In Visual C++ .NET 2002 and Visual C++ .NET 2003, whether operator new throws an exception is decided dynamically by the linker based on which order libraries are passed to it. If a project includes only C++ headers and definitions of operator new, the linker will use an operator new that throws exceptions. Conversely, if a project only uses C headers and definitions of new, the linker will select to use a non-throwing new. However, if a project is mixed, the version of new that is used depends on which the linker comes across first. To override this behavior and to force the use of throwing new, explicitly link to thrownew.obj. Note that in Visual C++ 2005, new always throws unless you explicitly link to nothrownew.obj. Second, note that none of the behaviours described relate to managed code or the .NET Framework. These behaviours are those of native C++ code compiled with Visual C++ .NET 2002 and Visual C++ .NET 2003.]

Background

When Microsoft released the first version of its Visual C++ compiler, its primary role was to support the MFC framework. For all practical purposes, Visual C++ and MFC were seen as one product. Over the years, both MFC and the Visual C++ compiler have matured. At the same time, the Visual C++ compiler has become a product in its own right, not necessarily dependent on MFC and supporting other technologies such as the Active Template Library (ATL), the Standard Template Library (STL), and a host of various other technologies. Today, MFC is just one of the many libraries that the Visual C++ compiler supports. As a result of this, it is now quite common to come across development projects that use Visual C++ without MFC.

I started writing this article when I observed unusual behavior with my STL code after operator new failed. To my surprise, I realized that Visual C++ 6.0 (and all prior versions that support STL) was incompatible with STL when operator new fails. The project I was working on did not use MFC, so my observations were based solely on non-MFC code. When I started looking at MFC-based examples, I discovered that MFC defines a very different behavior for operator new. Before diving into this article, I want to summarize the behaviors of operator new when a memory allocation fails. Just for good measure, I will describe the behavior under Visual C++ .NET since it's different from previous versions.

The C++ standard states that operator new should throw an exception on failure. Specifically, the exception thrown should be std::bad_alloc. That may be the standard, but the behavior under Visual C++ 6.0 depends on how you use it and what version you are using. Figure 1 shows the Visual C++ behavior of operator new when a memory allocation fails.

Figure 1 Behavior of Operator New on Failure (Close Figure 1)

Version | Non-MFC | MFC
Visual C++ 6.0 | Return NULL | Throw CMemoryException
Visual C++ .NET | Throw std::bad_alloc | Throw CMemoryException

As you can see, only non-MFC code under Visual C++ .NET adheres to the standard. If you are using MFC then new will throw an exception, but its type is incorrect. If the implementation of the STL that you are using contains catch (std::bad_alloc) statements to handle failed memory conditions then the only combination that will work out of the box is Visual C++ .NET without MFC. The STL implementation that ships with Visual C++ 6.0 uses catch(...) to handle operator new failures, so if you are using MFC, the STL implementation that ships with Visual C++ 6.0 will operate correctly when operator new fails.

Given that the operator new implementation provided with MFC throws an exception (CMemoryException) and also that under Visual C++ .NET the non-MFC operator new throws an exception (std::bad::alloc), I will assume that for all practical purposes, these cases do not pose a problem. What then of the common scenario of a Visual C++ 6.0-based project that does not use MFC but does make use of the STL? This is what I will focus on for the remainder of this article.

Operator New Returns NULL

Back to the question at the beginning of this article, there are generally one of two reasons given for not checking the pointer value returned from operator new for NULL: either operator new will never fail or operator new throws an exception.

Even if you think operator new will never fail, it is poor coding practice to not check its return value. While a desktop application is less likely to suffer from out-of-memory conditions, a user pressing F9 on their 100MB Excel spreadsheet may well cause your application to run out of memory. As for server-based applications that are expected to be up running and processing 24 hours a day, out-of-memory conditions are much more likely, especially on a shared application server. The chances of a memory failure are increased if you can't guarantee that your application does not leak even a single byte periodically. How many applications, especially those that are developed in-house, can deliver that kind of guarantee?

If your reason for not checking the returned pointer value for NULL is that operator new throws an exception, you can be forgiven. After all, the C++ standard stipulates that new should throw an exception on failure. This is not the default implementation for all versions of Visual C++ through 6.0 (when MFC is not used), which return NULL on failure. While this is fixed in Visual C++ .NET, the prior implementation can cause problems, notably with the STL. The implementation of the STL assumes that new will throw an exception on failure, regardless of the compiler that's used. In fact, if new does not exhibit this behavior and a memory allocation fails, returning NULL, the STL behavior will be undefined and will, in all likelihood, cause your application to crash. I will show a concrete example of this shortly.

Standard Template Library

More and more developers rely on the STL for their C++ development. The STL provides a wealth of classes and functions based on C++ templates. There are several benefits to using the STL in your application. First, the library presents a consistent interface for numerous general-purpose tasks. The second benefit is that the code has been extensively tested and can be assumed to be reasonably free from bugs. Finally, the algorithms are considered best-of-breed.

In order for the STL to work, the hosting compiler must implement the C++ standards that the STL was written to. The Visual C++ compiler comes prepackaged with an STL implementation, and a variety of other vendors' implementations can also be used.

If you are using the Visual C++ compiler (up to version 6.0), the STL will work as anticipated with one notable difference—low memory scenarios that cause operator new to fail.

Visual C++ 6.0 and Operator New

When operator new returns NULL on failure, it can be considered a bug because it is contrary to the standard which states that operator new should throw an exception on failure. All STL implementations that I am aware of, including the version that ships with Visual C++, anticipate that operator new will throw an exception on failure.

While it is possible to alter the behavior of operator new to throw an exception on failure, which I will demonstrate later, this itself results in a further anomaly. First I will show what happens when operator new fails in a default environment. All of my testing was carried out using Visual C++ 6.0, SP4 and SP5 running under Windows NT® 4.0 Service Pack 6a and Windows® 2000 SP2. To the best of my knowledge, this behavior affects code built with any version of the Visual C++ compiler, up to and including version 6.0 SP5, on all operating systems. The same code has also been tested using Visual C++ versions 7.0 and 7.1 and both of these versions show standard-compliant behavior when operator new fails. The STL libraries used were the version that comes with Visual C++ and the STLPort (http://www.stlport.org) implementation.

For the STL walkthrough, I will use what is probably the most common STL class, std::string, although the behavior described applies to any STL function or class that allocates heap memory through operator new. In this example, let's assume that I'm attempting to construct a new string object with some data and that the heap allocation will fail. The following code snippet will suffice:

#include <string>
void Foo()
{
std::string str("A very big string");
}

The code in Figure 2 is taken from the STL string class code that ships with Visual C++ 6.0. The constructor eventually lands in std::basic_string<>::_Copy. An extract of this is shown in Figure 2, with some code removed. In the try block, the local variable _S is assigned the return from allocator.allocate that, in turn, makes a call to operator new. With the default Visual C++ 6.0 behavior, operator new will return a NULL value on failure, resulting in a NULL value being assigned to the local _S variable. Critically, no exception will be thrown.

Figure 2 STL basic_string Code (Close Figure 2)
void _Copy(size_type _N)
{
  // Code omitted

  _E *_S;
    _TRY_BEGIN
        _S = allocator.allocate(_Ns + 2, (void *)0);
    _CATCH_ALL
        _Ns = _N;
        _S = allocator.allocate(_Ns + 2, (void *)0);
    _CATCH_END

    // Code omitted

    _Ptr = _S + 1;
    // ACCESS VIOLATION
    _Refcnt(_Ptr) = 0;

    // Code omitted
}

The next line of code to be executed is the assignment of _S + 1 to the member variable _Ptr. As the value of _S is NULL, the value 0x00000001 is assigned to the _Ptr variable. The following line, _Refcnt(_Ptr) = 0, effectively returns _Ptr - 1 (actually _Ptr[-1]), which evaluates against the original NULL pointer value returned from operator new. The _Refcnt member function returns a reference to the first element of a NULL pointer and the subsequent assignment of 0 to that reference (essentially *NULL= 0) will fail with an immediate access violation. Beyond the rather draconian measure of installing a structured exception handler to catch this error, there is no way of preventing this access violation from terminating your application. Although this behavior might seem like it's the result of a bug, the STL code is in fact correct, but in order to get this correct behavior it requires that operator new throw an exception on failure.

Now let's take a look at the flow of execution when operator new throws an exception on failure. As shown before, the initial call to allocator.allocate will be executed. When operator new fails, a std::bad_alloc exception is thrown and the code lands in the catch(...) (_CATCH_ALL) handler where the allocation is retried, possibly with a smaller request. If this second allocation fails, a further std::bad_alloc exception is thrown and this will propagate all the way back to your code, leaving the std::string object in a defined, empty state.

You should be aware that this overloaded string constructor could throw an exception. If you build classes that have std::string member variables whose overloaded constructor is called in the initialize section of your constructor, you should carefully read "Handling Exceptions, Part 10" of Robert Schmidt's Deep C++ column.

Fixing Operator New

Earlier, I stated that the implementation given in Knowledge Base article 167733 for fixing operator new to throw an exception should not be followed. The article presents two code samples, shown in Figure 3 and Figure 4. The first sample correctly installs a new_handler (for operator new failures), which I will refine to present an automatically installing handler. The second sample is why I gave the warning. This sample installs a new_handler and calls _set_new_mode(1) to indicate that malloc should throw an exception on failure.

Figure 4 KB Article 167733 Sample #2 (Close Figure 4)
#include <new>
#include <new.h>
#pragma init_seg(lib)
int my_new_handler(size_t) {
   throw std::bad_alloc();
   return 0;
}
struct my_new_handler_obj{
   _PNH _old_new_handler;
   int _old_new_mode;
   _tag_g_new_handler_obj() {
      _old_new_mode = _set_new_mode(1);
      _old_new_handler = _set_new_handler(my_new_handler);
   }
   ~_tag_g_new_handler_obj() {
      _set_new_handler(_old_new_handler);
      _set_new_mode(_old_new_mode);
   }

} _g_new_handler_obj;
Figure 3 KB Article 167733 Sample #1 (Close Figure 3)
#include <new>
#include <new.h>
int my_new_handler(size_t) {
    throw std::bad_alloc();
    return 0;
}
int main () {
    _PNH _old_new_handler;
    _old_new_handler = _set_new_handler(my_new_handler);
    /* ... application processing ... */
    _set_new_handler(_old_new_handler);
    return 0;
}

If you set malloc to throw an exception on failure, all code that uses malloc may be somewhat surprised at this behavior. Operator new(std::nothrow) is also implemented in terms of malloc (for debug builds at least) and this change will cause operator new(std::nothrow) to throw an exception on failure. This is definitely not the behavior that you want.

Figure 5 shows the code for a new handler that is installed automatically, which is included in the NewHandler.cpp source file available with this article (see the link at the top of this article). It is essentially the same as the second sample code listing from the Knowledge Base article (shown in Figure 4), with the syntax fixed and the _set_new_mode call removed. You can use this code by adding the NewHandler.cpp file to your project.

Figure 5 Self-installing new_handler (Close Figure 5)
#include <new>
#include <new.h>

// 4073: initializers put in library initialization area
#pragma warning(disable: 4073)

#pragma init_seg(lib)

namespace
{

int new_handler(size_t)
{
    throw std::bad_alloc();
    return 0;
}

class NewHandler
{
public:
    NewHandler()
    {
        m_old_new_handler = _set_new_handler(new_handler);
    }

    ~NewHandler()
    {
        _set_new_handler(m_old_new_handler);
    }

private:
    _PNH m_old_new_handler;

} g_NewHandler;

}   // namespace

The test harness for this handler is trivial, as Figure 6 shows. The allocation in the test code should result in immediate failure on most computers. The sample code results in an invalid heap allocation size, leading to an immediate failure without any attempt to actually perform the allocation. Unfortunately, if you continuously allocate large blocks of memory until an actual heap allocation failure occurs, you will not be able to debug the code because the Visual C++ IDE invariably crashes under low memory conditions. To build the test, download the sample files. Open the Testnew_throw.cpp file in Visual C++ 6.0, and select Build from the Build menu. Accept the prompt to create a default workspace. If you step through the code you can verify that operator new now throws an exception of type std::bad_alloc on failure.

Figure 6 New Throws std::bad_alloc (Close Figure 6)
// Testnew_throw.cpp Test new throws std::bad_alloc on failure
#pragma warning(disable: 4307)
#include "../NewHandler.cpp"
#include <iostream>
struct Big{ double m_d[99999]; };
void main()
{
    try
    {
        // Force failure
        Big *p= new Big[99999];
    }
    catch(std::bad_alloc& er)
    {
        std::cout << er.what() << std::endl;
    }
}

When Operator new(std::nothrow) Throws an Exception Anyway

Toward the end of the Knowledge Base article, a note about operator new(std::nothrow) and Visual C++ 5.0 indicates that new(std::nothrow) will throw an exception if a new handler is installed. This problem still exists under Visual C++ 6.0, but the behavior is subtler. Using Visual C++ 6.0, operator new(std::nothrow) will behave as expected and return NULL on failure when linking against the Debug runtime libraries only. If you link against the Release version of the runtime library, then operator new(std::nothrow) will throw an exception anyway. This is certainly not the behavior you want in your applications. The test harness for operator new(std::nothrow) is trivial, but demonstrating the full behavior is not so straightforward due to a further quirk, this time with the optimizing compiler. The crucial part of this test is that the only code in the try block is the call to operator new(std::nothrow), shown in Figure 7.

Figure 7 new(std::nothrow) Throws bad_alloc (Close Figure 7)
// Testnew_nothrow.cpp Test new(std::nothrow) returns NULL on failure
#pragma warning(disable: 4307)
#include "../NewHandler.cpp"
#include <iostream>
struct Big{ double m_d[99999]; };
void main()
{
    Big *p;
    try
    {
        // Force failure
        p= new(std::nothrow) Big[99999];
    }
    catch(std::bad_alloc& er)
    {
        std::cout << "Error " << er.what() << std::endl;
        return;
    }
    std::cout << "p= " << p << "\n";
}

Build the Win32® Debug configuration of this test, using the same technique as for Testnew_throw.cpp. Running the resulting executable yields the following expected result:

p= 00000000
Now build the Win32 Release configuration and run the resulting executable. The output may surprise you:
abnormal program termination

One thing is certain here: operator new(std::nothrow) is definitely not returning NULL. Exactly what is happening is not so clear. Try moving this line inside the try block:

std::cout << "p= " << p << "\n";
The results change:
Error bad allocation
Now the catch handler is called, confirming the release build behavior I alluded to earlier. The question still remains: why did I previously get an abnormal program termination? To see exactly what is going on here, first revert back to the original code shown in Figure 7 (with only the call to operator new(std::nothrow) in the try block). Next, change the project settings for the Win32 Release configuration to disable compiler optimizations (Project | Settings | C/C++ | General | Optimizations = Disable (Debug)). Build and run the executable. The program's output is expected, though still not quite what you're after:
Error bad allocation

This behavior is due to the optimizing compiler, which is in fact generating valid compiled code. The documentation for Visual C++ 6.0 states that while the compiler supports the syntax for exception specifications, it ignores them. As it turns out, this is not strictly true. To see exactly what is going on, you have to delve into the generated assembly code (very lightly). Retaining the disabled optimizations and ensuring that the only line of code in the try block is the call to operator new(std::nothrow), turn on mixed assembly code (Project | Settings | C/C++ | Listing Files | Listing file type = Assembly with Source Code). Build the executable and open the generated file Testnew_nothrow.asm that is in the Release folder. In this file, search for the string "try", ensuring you have "Find whole word only" checked to avoid partial matches with other instances. You should be able to see the mixed source/assembly for the try block, including a reference to a variable called __$EHRec$. This is part of the generated code for the try/catch exception mechanism.

Next, turn optimizations back on, rebuild, and locate the "try" source line in the Testnew_nothrow.asm file. The reference to the __$EHRec$ variable is gone. What has happened is that the optimizing compiler has detected that operator new(std::nothrow) is declared as throwing no exceptions and has correctly concluded that the entire try/catch block is redundant code. The result is that the entire try/catch block has been optimized away, leaving the code that was wrapped by the try block to run without the unnecessary exception handling support. While this is technically correct, you are left with the discrepancy of the compiler subsequently allowing an exception to be thrown from a non-throwing function, which may be impossible to catch.

Having discovered exactly what is going on with operator new(std::nothrow), I now present the fix to this problem in the form of a user-supplied version of operator new(std::nothrow), taken from Knowledge Base article 167733, which is included in the NewNoThrow.cpp file source file available with the code download for this article. This version correctly returns NULL on failure:

void *__cdecl operator new(size_t cb, const std::nothrow_t&) throw()
{
    char *p;
    try
    {
        p = new char[cb];
    }
    catch (std::bad_alloc)
    {
        p = 0;
    }
    return p;
}

This requires a word of warning, however. If you are linking against a DLL version of the runtime library (Debug Multithreaded DLL or Multithreaded DLL) then the new_handler is effectively installed into the runtime DLL. This means that any DLL file that loads into your process address space that also links against a matching version of the runtime library DLL will be affected by this handler (new will throw an exception on failure). The implication of this depends wholly on whether the client DLL anticipated new returning NULL or assumed that it would throw an exception (ATL supports both modes).

The fix for operator new(std:nothrow) is only required if new is changed to throw an exception, a change that is mandatory for using STL correctly. However, the fix is local to the project into which the source file is inserted. In this case, any third-party DLL that uses operator new(std::nothrow) and builds against a compatible release runtime DLL as I've shown will now be at the peril of new(std::nothrow), throwing an exception on failure. This happens because the global scope of the new handler is mismatched with the local scope of the replacement operator new(std::nothrow). The only practical solution is to either link against one of the static runtime libraries or verify that third-party code does not call operator new(std::nothrow) or does not link to the DLL version of the runtime library. If there is a possibly redeeming feature to this unfortunate state of affairs, it is that operator new(std::nothrow) is a rarely used overload.

The final gotcha is that the STL provided with Visual C++ 6.0 does in fact use this overload in a single place, namely the get_temporary_buffer template function. It is quite unlikely that your code calls this function directly. However, a search through the STL source code shows that the internal _Temp_iterator template class calls the get_temporary_buffer template function. The _Temp_iterator template class itself is called, indirectly, by the following public functions defined in algorithm: stable_partition, stable_sort, and inplace_merge. The behavior of these functions is undefined if operator new(std::nothrow) does throw an exception. If you are using these functions and you have the new handler installed, consider using a different implementation of the STL, many of which are available. I successfully use STLPort (http://www.stlport.org) in my work. To the best of my knowledge, this implementation does not call operator new(std::nothrow) anywhere in its implementation.

Conclusion

I have demonstrated that if you're using the STL with Visual C++ up to version 6.0 in a non-MFC project, its out-of-the-box behavior can cause the STL to crash your application under low memory conditions. The version of operator new provided with Visual C++ 6.0 is not compatible with the STL. Even with the fixes provided here, it is still possible that trouble may ensue with third-party code or with a few functions in the version of the STL that's supplied with Visual C++ 6.0. The current mismatch in Visual C++ 6.0 between operator new, operator new(std::nothrow), and the STL cannot be completely fixed. However, if you are using STL in your code and you do not include the fixes I've recommended in this article, you run a real risk of your application crashing in STL code under low memory conditions.

For MFC-based projects, whether or not STL will survive a failure within operator new depends wholly on how the STL implementation that you use handles exceptions through this operator. Most implementations seem to use catch(...) rather than catch(std::bad_alloc) when handling failed allocations, but this is not a requirement.

Finally, as I stated at the beginning of this article, both currently available versions of Visual C++ .NET have fixed all of the problems I have described, with the exception of the MFC behavior.

For related articles see:
Handling Exceptions in C and C++, Part 1
Visual C++ Developer Center
http://www.stlport.org

James Hebben is a UK-based consultant specializing in Visual C++ financial and trading systems (http://www.extraview.uk.com). He can be reached at James.hebben@extraview.uk.com.


Editor's Note: Tales of a GPS Demo

Editor's Note
Tales of a GPS Demo

Code download available at: Ednote0309.exe (1,016KB)

One of the most annoying things about the Internet is that no matter how hard you try to make something permanent, it can be gone tomorrow with no forwarding address. We found this out the hard way earlier this year, and we're hoping to remedy the situation with this Editor's Note. Let us explain.

Back in 2001, we presented an article called "Migrating a GPS App from the Desktop to eMbedded Visual Basic 3.0," which set the standard for articles about migrating GPS apps to the Pocket PC. A year and a half later, the .NET Compact Framework was introduced in beta, and we decided to update the GPS app to fix a few bugs, modularize it, and bring the power of the .NET Framework to it. We planned to present it at the ASP.NET Connections conference in October 2002.

There were several interesting challenges in porting a program like this. The first big obstacle was accessing the COM port, since a GPS device communicates with the host machine through an RS-232 serial port. If you'll take a quick look at the .NET Framework, you'll find no classes to do native serial comm! There were two choices here. The first was to use the MSComm ActiveX control on a form—but the .NET Compact Framework doesn't support embedded ActiveX components. So we could either spend valuable minutes of our life wrapping MSComm and hoping for the best with a direct port or we could create a wrapper class.

About the time we were working on this project, we published an article called "Use P/Invoke to Develop a .NET Base Class Library for Serial Device Communications" by John Hind. The ideas in this piece were all we needed to create a .NET Compact Framework version of a serial comm library (it's not exactly the same as the desktop version since it uses different DLL exports). Putting this serial comm class together with a bit of simple threading, we created a more robust serial engine for the program.

There were a number of other challenges we faced in creating the session, which we documented for the anticipated assembled throng at the conference. Everything was going great. We had a stable beta of Visual Studio .NET 2003 running on the demo machine, but about a month before the conference, terror hit. The trusty laptop decided it no longer recognized a key peripheral—the power cord. Within days the only solution was to use the machine with its docking station attached at all times. Then the keys started popping off. We put in an emergency order for a new laptop, which arrived just in the nick of time.

Everything was reinstalled and working just fine as we got to Orlando. We got the project up and running with sample data, so there was only one step left—the live test. We went to plug in the trusty GPS unit and...and...where's the nine-pin adapter? When did they start making laptops without a nine-pin serial port, and why did we order the first one ever produced? What a mess. Fortunately, we had a backup PC Card GPS receiver.

The talk itself went pretty well (it was erroneously listed as GPRS, instead of GPS, so we only managed to get a dozen or so people in attendance, but they were all treated to a memorable mid-morning event). Now here's the problem. After the conference, we put the code up on the free Web space given to us by an ISP and told everyone to download the code from there. A few months later, the nice folks at this satellite company got out of the DSL business. We switched over to another national ISP, but can't figure out how to use the shell account. With nearly a dozen people clamoring for this .NET Compact Framework project, and another quarter-handful looking for it on various newsgroups, we thought we'd give it a more permanent home. The PowerPoint presentation and associated sample code are available as the download for this Editor's Note (see the link at the top of this article). We think that's a pretty permanent location.

Now for this year's ASP.NET Connections conference, we're working on another program that will draw maps directly from data provided by the U.S. Census Bureau. This is going to be even cooler than last year's demo, and we hope to see at least two dozen of you there this October.

—J.T.

BizTalk, JScript, Microsoft, MSDN, Outlook, Visual Basic, Visual C++, Visual SourceSafe, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Search and Retrieve
Test a Page for Privacy
Modify Database Files Using ASP Pages
Embed Coordinate Conversion in Windows-based Apps
Test .NET Classes and Assemblies
Managing Your Enterprise Issues
Visual Studio Code Generator
Create Web Pages from a Database
The Bookshelf

Search and Retrieve

ISYS/Odyssey Development Inc. now supports the Microsoft® .NET Framework with the availability of the ISYS:web.asp 6.0 search engine product. Developers using ASP.NET can interface directly with the core ISYS engine, powering their Web applications with the search and retrieval features of ISYS:web.asp 6.0. ISYS search software enables rapid search and retrieval of information from large volumes of data contained across disparate sources and file types.

ISYS supports more than 125 file formats and 30 languages. It offers natural-language search capabilities, as well as query searching for advanced searches. The technology provides searching of structured, unstructured, and semi-structured information in a single query, and provides polymorphic functionality across diverse formats, including XML, HTML, Microsoft Word, Microsoft Excel, PDF, ZIP, and OEM-defined formats.

http://www.isysusa.com

Test a Page for Privacy

Watchfire Corporation recently rolled out Watchfire WebXACT, an online service that allows users free testing of single pages of Web content for quality, privacy, and accessibility issues. WebXACT is based on technology from the company's enterprise solution.

The Quality Status Report in WebXACT explains the page's quality issues and indicates whether it has defects such as broken links or anchors, warnings, or other issues. The WebXACT Accessibility Report summarizes the accessibility issues on the page to help determine if that page can be accessed by individuals using screen readers, and facilitates compliance with the U.S. Government's Section 508 and the W3C Web Content Accessibility Guidelines.

The WebXACT Privacy Report indicates whether the page has a link to a privacy statement, identifies information collection, visitor-tracking technology like cookies and Web beacons, P3P compliance, and third-party links. WebXACT privacy reports are a good first step to help companies understand some of the potential privacy issues that need to be addressed on their Web sites.

http://webXACT.watchfire.com

Modify Database Files Using ASP Pages

ASPRunner, published by XlineSoft, creates sets of ASP pages to access and modify data from Microsoft Access, SQL Server™, Oracle, DB2, MySQL, FileMaker database, or any other ODBC data source. You can then use the resulting ASP pages, created using ASPRunner's wizard-like interface, to search, edit, delete, and add data to your database.

ASPRunner allows you to provide online access to your database. It lets you create password-protected ASP pages, improve the functionality of your Web site, and add, view, edit, and delete pages. You can sort data, perform searches, and use the software for data entry and Web-based reporting.

http://www.xlinesoft.com/asprunner/download.htm

Embed Coordinate Conversion in Windows-based Apps

Blue Marble Geographics has released GeoCalc 5.2, a DLL used for embedding real-time coordinate conversion in Windows®-based applications. GeoCalc users can match their custom coordinate systems to a known geodetic coordinate system for on the fly conversion. This new function allows a user to create an affine transformation and move data from a previously unknown coordinate system to a useable predefined coordinate system. GeoCalc contains over 130 predefined ellipsoids, 630 datum transformations, and over 30 projections.

New features include the implementation of Australian National Transformation version 2 that supports internal subgrids within a master grid file. Blue Marble has also added a WGS72 ellipsoid definition for use in MGRS grid conversion.

http://www.bluemarblegeo.com

Test .NET Classes and Assemblies

Parasoft has announced the release of .TEST 1.0, a unit testing tool that automatically tests classes written with the Microsoft .NET Framework without requiring you to write test scenarios or stubs. .TEST automatically performs a range of static and dynamic testing on .NET Framework assemblies. The program performs static analysis on assemblies, enforcing over 200 coding rules that verify code construction and also reduce the possibility of introducing errors into applications.

.TEST reads code and determines how to unit test it. All unit tests generated by .TEST are fully customizable. The output provides detailed coverage information for both intermediate language and source code statements, and maintains code integrity through automated regression testing. .TEST works with any programming language that targets the .NET Framework, including C#, Visual Basic® .NET, and Managed C++. .TEST is available either as a standalone tool or as part of Parasoft's Web Services Solution.

http://www.parasoft.com

Managing Your Enterprise Issues

Elsinore Technologies Inc. has released Visual Intercept 3.5, an enterprise issue management solution designed to integrate with Microsoft development and productivity tools. One of the many enhancements to Visual Intercept 3.5 is its Web-based integration to Microsoft Visual SourceSafe®, which allows you to work on source files within the context of their software development incidents.

Other enhancements include spell checking, automatic login, additional custom fields, and new custom data types. Version 3.5 also includes new standard fields for build numbers, version dates and attached document annotation, reverse sorting on columns, precedence value for setting parameter sort orders, and the ability to clone users and groups.

An electronic signature feature supports 21 CFR Part 11 and other regulatory compliance applications. You'll also find a mass update capability, macro-based queries, and a refined GUI.

http://www.elsitech.com

Visual Studio Code Generator

Workstate has announced the availability of Codify 1.2, a code generation add-in for Visual Studio®. Multiple code generators (CodeBuilders) can be embedded within any source code file in a project. CodeBuilders can generate anything from a few lines of code (micro code generation) to entire class hierarchies (macro code generation).

CodeBuilder parameters are specified as XML and can be edited either textually or visually using the CodeBuilder editor. Codify templates can be created or customized using the built-in template editor, which supports two distinct scripting styles: CodifyScript (similar to ASP.NET scripting) and XSLT.

Codify also includes templates that can be used as-is or can be customized. Built-in templates include Simple SQL Server Data Access Layer, which generates a class that contains a strongly typed method for each stored procedure in your database, and Strongly Typed Collections, which generate strongly typed collection and dictionary classes. The Simple Property Builder template generates simple properties within a target class that get and set their values from a private field. The Windows Forms Menu Builder template lets you create custom MainMenu and ContextMenu classes that encapsulate all of the tedious code required to create and initialize Windows Forms menus.

http://www.workstate.com/en/codify

Create Web Pages from a Database

TheDevShop Ltd. has released dbQwikSite 2.5, which creates Web pages using database information. Version 2.5 adds the ability to generate PHP Web scripting language in addition to the existing ASP and static pages site options. dbQwikSite produces Web catalogs, lists, online stores, and simple reports. You can produce database maintenance pages to add, edit, and delete data.

Generating Search pages allows your visitors to find information using a variety of field combinations. The program connects to any database including Microsoft Access, SQL Server, MySQL, Oracle, DB2 or any ODBC-compliant database.

The dbQwikSite interface includes a visual query builder and a page designer. The program splits n records per page, generates navigators, and can place database fields into both page content and hyperlinks. Images can be added as links to GIF or JPG files.

dbQwikSite supports HTML templates for record layouts and comes with several standard templates for Web catalogs and lists. The quick preview feature allows you to see how the final generated pages will appear populated with data. The integrated FTP function assists you in deploying your Web site to the server.

dbQwikSite is available in three editions: Sponsored, Standard, and Pro. The Sponsored Edition allows you to generate static HTML and limited ASP and PHP sites. Standard Edition allows production of static and dynamic Web sites with search capabilities. Pro Edition adds remote database management with the generation of add, edit, and delete pages. New Pro features include basic security using user login pages. When you register Standard or Pro you can select either ASP or PHP support. If you need support in both languages, you can opt for the Plus option. dbQwikSite runs under Windows 95 and above.

http://www.dbQwikSite.com

The Bookshelf

Programming C#, Third Edition by Jesse Liberty, published by O'Reilly, reflects the release of Visual Studio .NET 2003 and the .NET Framework 1.1 SDK. You'll find a new chapter demonstrating various Web Forms and Web Services applications, and expanded coverage of events and delegates in response to numerous reader requests. The author has added tips for programmers coming from Java, Visual Basic, and C++ backgrounds.

The first part of this book introduces C# fundamentals, then goes on to explain classes and objects, inheritance and polymorphism, operator overloading, structs and interfaces, arrays, indexers and collections, string objects and regular expressions, exceptions and bug handling, and delegates and events.

Part II focuses on development of desktop and Internet applications, including Windows Forms, ASP.NET, and ADO.NET. Part III gets to the heart of the .NET Framework, focusing on attributes and reflection, remoting, threads and synchronization, and streams. Part III also illustrates how to interoperate with COM objects.

http://www.oreilly.com/catalog/progcsharp3

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: InfoPath Back End, WSH Script Signing, and More

Web Q&A
InfoPath Back End, WSH Script Signing, and More
Edited by Nancy Michell

Q I've read a little about InfoPath lately, but I'm confused about what I need on the back end. Do I need to have a database, or does it make sense to store everything in XML files? From what I've read it seems like you can do that, but how efficient is it and is it recommended? Otherwise, would I use Microsoft® Access? Do I need Microsoft SQL Server™?

A InfoPath can handle any of the options you mention. Looking at the question from a best practices perspective, however, there is not always a single correct solution. You should evaluate the solutions for both online and offline situations.

Online solutions are typically connected to the network all the time. Ideally, these are best served using XML-based Web Service as the submission point. From within a Web Service you can push the data directly into a SQL database or a BizTalk® process. Even though you can use SQL Server or Access, you'll end up with a very tightly coupled solution when you send data directly to a database without a Web Service in between. Try to avoid this whenever possible, as Web Services offer better flexibility and control.

Offline solutions may never be connected to the network or may be connected only occasionally, typically in scenarios where people are using mobile devices and roaming around. For these situations, you should use either a local Access database, the Microsoft Data Engine (MSDE), or even local files. The data is really in a transitory state until it is pushed to its final resting place through an online solution. For example, replication or Microsoft Message Queue (MSMQ) can be used as a way of transmitting the data to the back end.

You should really think about this question in light of a services oriented architecture (SOA). SOA defines components as services, and InfoPath is simply a consumer of these services from the front end. People are often confused about where InfoPath stands in comparison with ASP.NET. They think they don't need an InfoPath solution because they're already doing everything as a Web Form. That is fine, but there are limitations. A better solution is to treat all services as a provider and then use InfoPath, ASP.NET, or Windows® Forms as the consumer. Once you abstract the back end, you need to look at how to get the data into a location that can be used. For example, you could push collection information into a SQL database for reporting or analysis.

For more information on InfoPath, see Aaron Skonnard's article in this issue of MSDN® Magazine.

Q I would like to use the signature verification policy available with Windows Script Host (WSH) 5.6 (see Signature Verification Policy). On MSDN Online, I found information on using signature verification policy in a Windows 2000 environment (in Windows 2000, the signature verification policy is set through the Local Security Policy editor), but I'm running Windows NT® 4.0. Can I use the signature verification policy of WSH 5.6 in a Windows NT 4.0 environment?

A The short answer is that you can install WSH 5.6, which features code signing, on Windows NT 4.0. The signature verification policy works wherever you can install WSH 5.6. Windows NT 4.0 is still supported, and there are setup packages available for it. Use the one labeled for Windows NT, Windows 98, and Windows Me.

Now, let's get down to specifics. What exactly do you want to do? If you want to use Software Restriction Policies (SRP), then you will have to use an operating system that supports it.

If you don't want to use SRP, there are some registry keys you can set. Under either HKLM or HKCU, take a look at the \Software\Microsoft\Windows Script Host key. Four values are relevant here: Enabled, TrustPolicy, UseWINSAFER, and IgnoreUserSettings. Here are their effects:

  • If Enabled is set to zero, then WSH will not run at all.
  • If TrustPolicy is set to zero, then all scripts will run. If it is set to one, then all signed, trusted scripts will run; unsigned scripts will prompt the user. If TrustPolicy is set to two, then only signed, trusted scripts will run and there will be no user prompt.
  • If UseWINSAFER is set, then there is first an attempt to use SRP if it is installed. If it is not, then this flag is ignored and the TrustPolicy flag is followed.
  • By default, both the HKLM and HKCU settings are checked and the user settings are followed rather than the machine settings. If you want the machine settings to be the default, set IgnoreUserSettings in the HKLM path to one.

You can buy a certificate from VeriSign or install your Public Key Infrastructure (PKI), produce your own certificate, add your Certificate Authority (CA) to the list of trusted CAs on the machines, and so on. Next, sign the scripts (as described in Windows Script Host) using the certificate. Finally, you'll be able to turn on the Enabled, TrustPolicy, UseWINSAFER, and IgnoreUserSettings registry keys on the machines.

Q I need to know the best way to iterate over a large XML document. Let's say, for example, that I have a huge XML document that's basically organized like the following:

<Root>
   <ForestNode>
      <Node>
         <...>
            <...>
               ...
   <ForestNode>
      ...
Let's also assume that I have some node that is regularly repeated at some relatively high level. Think of a forest as a collection of a specific type of node. In my sample, there is a forest of Contact nodes (see Figure 1). Each forest node may be extremely complex, and I may not need to get all the information contained in every node. For example, say I have an XML representation of all the domains in my company. This representation may drill all the way down to the details of computers and users on the domain, and so on. But what if I'm only interested in getting a list of domain names in the company, or even just domain names that fall within certain criteria (domains in North America, for instance)? How can I retrieve only the information I need without walking through every node of the XML?

Figure 1 Contact Nodes (Close Figure 1)
<Sample>
    <Contacts>
        <Contact>
            <Name>Some Guy</Name>
            <Address>1234 Third Street</Address>
            <City>Bedford</City>
            <Age>457</Age>
        </Contact>
        <Contact>
            <Name>Other Guy</Name>
            <Address>333 Fifth Ave</Address>
            <Age>25</Age>
        </Contact>
    </Contacts>
</Sample>

A The answer to this problem is a simplified mechanism that works with one main loop that goes over the forest nodes. The following while loop iterates the nodes, extracting just the information you need and processing it.

XPathNodeIterator ni = xdoc.CreateNavigator().Select(forestNode);
while (ni.MoveNext())
{
      XPathNavigator nav = ni.Current;

      string XPathStmt = "Name";
      myContact.Name = GetXPathValue(nav, XPathStmt);
      XPathStmt = "Age";
      myContact.Age = GetXPathValue(nav, XPathStmt);

      ProcessMyContact(myContact);
}

This code is extracting just the name and age from the contacts. Figure 2 shows the entire process. Obviously, this is a very simple example. You could generalize this so that you can provide a list of XPath expressions and the location to put the data so you don't need to hardcode the data assignment.

Figure 2 Iterating the Nodes (Close Figure 2)
using System;
using System.Xml;
using System.Xml.XPath;

namespace XNITest
{
    /// <summary>
    /// Summary description for Class1.
    /// </summary>
    class Class1
    {
        /// <summary>
        /// The main entry point for the application.
        /// </summary>

        public struct SimplifiedContact
        {
            public string Name;
            public string Age;
        };

        [STAThread]
        static void Main(string[] args)
        {
            string forestNode = "/Sample/Contacts/Contact";
            SimplifiedContact myContact;

            XPathDocument xdoc = new
                XPathDocument("sample.xml");

            //Loop over the individual nodes of the forest
            XPathNodeIterator ni =
                xdoc.CreateNavigator().Select(forestNode);
            while (ni.MoveNext())
            {
                XPathNavigator nav = ni.Current;

                string XPathStmt = "Name";
                myContact.Name = GetXPathValue(nav, XPathStmt);
                XPathStmt = "Age";
                myContact.Age = GetXPathValue(nav, XPathStmt);

                ProcessMyContact(myContact);
            }
        }

        public static string GetXPathValue( XPathNavigator XPathNav,
            string XPathStmt)
        {
            XPathNodeIterator ni = XPathNav.Select(XPathStmt);
            if(ni.MoveNext())
                return ni.Current.Value;
            else
                return "";
        }

        public static bool ProcessMyContact(SimplifiedContact contact)
        {
            Console.Write("Simplified Contact\n");
            Console.Write("—————————\n");
            Console.Write("Name: {0}\n", contact.Name);
            Console.Write("Age: {0}\n", contact.Age);
            Console.Write("\n\n");

            return true;
        }
    }
}

Q I am writing an ASP page that deals with four different types of objects that have common features. It is a very simple design and the classes make it easy for me to make polymorphic calls. I have a case in which I would like to reuse some functionality and one option is to make it using inheritance. Of course, I can always make it a separate function. What is the correct way to implement object inheritance in JScript®?

If I have Base and Derived classes and their constructors

function Base()
{
// do something
}

function Derived()
{
// do something else
}
how can I call the constructor of Base from the constructor of Derived (and use the this operator)?

A The correct way is to rewrite your program in a language that supports class inheritance. Classic JScript does not support class inheritance, only prototype inheritance. However, JScript .NET supports both.

Why do you want to use class hierarchies at all? Class hierarchies were invented to provide abstraction and encapsulation to developers working in large teams on complex software. JScript was designed to provide a quick, easy way to write simple scripts on Web pages. The fact that you're asking at all makes me think that you're using the wrong tool for the job.

If you don't want "is a" relationships but do want code sharing, that's when you use prototype inheritance. However, there is no easy way to do a recursive descent on constructor functions, which is what you asked. Languages like C# do recursive descent on constructors because class inheritance strongly expresses an "is a" relationship. A Giraffe is a Mammal, so you'd expect the Giraffe ctor to call the Mammal ctor (which calls the Animal ctor, and so on) to ensure that the polymorphism works.

Prototype inheritance languages do not express "is a" relationships nearly so neatly. Instead, they keep a prototypical object and share implementation details through one. However, if all you want to do is share some methods, that's easy:

function Feed() {}
function Animal() {}
Animal.prototype.Feed = Feed;

var Bob = new Animal(); // create a new Animal, set its prototype to
                        // Animal.prototype
Bob.Feed(); // Bob has no Feed method, but Bob's prototype does,
            // so we call it.
Now you can make this a little deeper by making Bob's prototype itself have a prototype (see Figure 3).

Figure 3 Sharing Methods (Close Figure 3)
function Feed() {}
function Slither() {}
function Animal() {}
function Snake(){}
Animal.prototype.Feed = Feed;
Snake.prototype = new Animal();  // Snake.prototype's prototype
                                 // is Animal.prototype.
Snake.prototype.Slither = Slither;

var Bob = new Snake();           // Bob's prototype is Snake.prototype
Bob.Slither() // Bob has no slither method, but Bob's prototype does.
Bob.Feed();   // Bob has no Feed method.  Neither does Bob's prototype.
              // But Bob's prototype's prototype does, so we call it.

The Animal constructor is not called once per instance of Snake. Rather, it is called once for all instances of Snake, and the result is the "prototypical Snake."

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Svet Bonev, Marco Crippa, John Lambert, Paul Leach, Eric Lippert, Mike Lonergan, Thom Robbins, Ben Smith, Michael Whalen.


Data Points: Developing Apps with the .NET Compact Framework, SQL Server CE, and Replication

Data Points
Developing Apps with the .NET Compact Framework, SQL Server CE, and Replication
John Papa

Code download available at: DataPoints0309.exe (150KB)
Contents
Configuration
SQL Server CE Server Agent
Replication Publication
The Application
The Code
Exception Handling
Conclusion

Handheld and other portable devices are shipping with ever growing feature sets. These devices can now communicate on private networks and over the Internet through Wi-Fi and built-in wireless modems that are combined with cell phones. Handheld devices are popular not only for personal use, but also many businesses are taking advantage of the mobility they offer. The growing demand for better portability means an increasing demand for mobile applications. One example could be an application that lets employees update inventory, enter orders, update customer profiles, and perform other data-centric tasks from the field.

In this column, I'll develop a mobile application that can be used to review and update a business' inventory. The application, targeted at handheld Windows® CE-based or Microsoft® Windows Powered Pocket PC devices, can operate as a standalone application or as part of an enterprise-wide application. I will use the Microsoft .NET Compact Framework as it exists in Visual Studio® .NET 2003 to develop the application for Pocket PC devices. One of the key ingredients in this application is that the entire inventory of products can be synchronized and stored on the handheld device within an instance of SQL Server™ CE.

The SQL Server CE database on the Pocket PC device will contain a replicated version of the Northwind database from a back-end SQL Server database. Since the database exists in two places, I will use the merge replication feature of SQL Server in order to maintain the data integrity of both databases.

I'll begin by explaining how to set up the prerequisites for the merge replication scenario including many of the ways that it can be customized. After discussing how to configure the environment, I'll demonstrate how the application functions and takes advantage of both SQL Server CE and the disconnected nature of ADO.NET. As in any application, exception handling is critical. However, when developing for mobile devices, the need for solid exception handling is amplified tremendously. Finally, I'll wrap up by explaining how to trap different errors that can occur between the handheld and the back-end database and how to report them.

Configuration

Before going any further, it is important to understand the prerequisites for developing an application using the .NET Compact Framework and SQL Server CE on a handheld device. The .NET Compact Framework is now installed with Visual Studio .NET 2003. SQL Server CE device components are installed with Visual Studio .NET 2003. SQL Server CE server components can be downloaded from the Microsoft Web site along with any service packs, which I highly recommend that you install. Once the .NET Compact Framework, SQL Server CE, and the latest version of the SQL Server and SQL Server CE service packs are installed, you are well on your way.

While installation is easy, the configuration of merge replication takes some careful consideration. When designing the application, it is important to take a step back and examine how the various moving parts interact with each other. For example, one simple but often forgotten step is to make sure that the SQL Server and SQL Server Agent services are both running on the back-end server. In the sample inventory application, the Northwind database on a back-end SQL Server instance is the central data store containing all information including products, orders, and customers. The application allows handheld devices to store the inventory information from the Northwind database on the back-end database server. The two databases communicate through the merge replication process, which uses the SQL Server CE Server Agent to pass information back and forth.

SQL Server CE Server Agent

The SQL Server CE Server Agent is an ISAPI DLL that is set up in a virtual directory under Microsoft Internet Information Services (IIS). Essentially, it allows the two databases to communicate with each other using the HTTP protocol to relay messages. To set up the SQL Server CE Server Agent, it is easiest to use the SQL Server CE Virtual Directory Creation Wizard. A more advanced approach is do it manually. I created a folder on my server called "C:\Projects\SQLCE" and copied the agent DLL (sscesa20.dll) from the SQL Server CE setup folder under \Program Files\Microsoft SQL Server CE 2.0\Server and registered it there. I then created a virtual directory in IIS with HTTP execute rights called SQLCE and pointed it to the SQL Server CE Server Agent's new content folder location.

When planning for merge replication via the SQL Server CE Server Agent, establishing the appropriate permissions on both the NTFS folder and the virtual directory are vital. The NTFS folder permissions and the virtual directory should allow the handheld application to access the agent. You should use some authentication other than anonymous mode, but for the simplicity of my sample I allowed anonymous access to the virtual directory through the IUSR_machine name account. Because files will be written when merge replication synchronizes, I gave the IUSR_machine name account write access to the folder as well. Again, anonymous access is great for sample applications, but you'll need a more secure authentication method in real apps. For more information on connectivity and security see Configuring Security for Connectivity.

Replication Publication

Once the means has been established for the SQL Server and SQL Server CE instances to communicate with each other, you need something for them to talk about. Initially, a publication must be created on the SQL Server back end to define the parameters of the merge replication process. The publication defines the data that will be available, and how to resolve any potential replication conflicts. The publication access list defines who's allowed to subscribe to it. The type of authentication used to subscribe is defined by the SQL Server configuration. A publication can be created by starting the Create Publication Wizard, which can be accessed by right-clicking a database and selecting New | Publication from the pop-up menu. Most of the wizard is intuitive, so I will concentrate on the more critical settings.

There are three types of publications that can be created: snapshot, transactional, and merge (see Figure 1). Since the inventory app must allow either the back-end database or the handheld's database to update the data and synchronize with each other, a merge publication best fits the bill. In addition, merge replication is the only form of replication supported by SQL Server CE.

Figure 1 Three Types of Publications
Figure 1 Three Types of Publications

The next step of the wizard is to indicate what type of subscribers are allowed to subscribe to the publication. Be sure to select at least "Devices running SQL Server CE." Now you must indicate which articles will be published. An article can be a table, view, or even a stored procedure. For the sample application I chose to publish all of the information for simplicity; however, a subset of the tables would be adequate since the only information that I intend for the handheld device to use is related to inventory.

One side effect of setting up a publication to use merge replication is that all articles that are selected to be published, and which do not already have a unique or primary key constraint with the ROWGUIDCOL property set, will have an additional column added called rowguid. An index along with the rowguid column, defined as type uniqueidentifier, will be added when the first snapshot of the publication is taken. The uniqueidentifier value is automatically generated for new rows in the article. The uniqueidentifier value is used to link rows of the back-end database's article to those of the SQL Server CE article.

Publications can also filter the article vertically (limiting columns in the article) or horizontally (limiting rows in the article). This can be done set either through the Create Publication Wizard or through the properties of the publication. For this sample application, I did not create any filters and I also specified that all tables become published articles. Once the wizard completes, it then creates the publication.

When the publication is created, SQL Server also creates several other objects behind the scenes to support the replication process. For example, triggers are created on all published articles that will track changes made to the articles on the server. The changes are sent through views to the MSmerge_contents and MSmerge_tombstone tables, which are also created by the publication. Inserts and updates are tracked in the MSmerge_contents table while deletes are tracked in the MSmerge_tombstone table. These views, triggers, and tables are created to support the replication process that is defined by the publication. SQL Server CE uses a similar process.

Once I created my publication, which I named NorthwindPub, I assigned the SQL Server accounts that could access the publication through the Publication Access List. For my sample, I created a SQL Server login called SqlTestUser and granted it rights to the Northwind database. Then I went to the Publication Access List tab in the properties of my NorthwindPub publication. There, I added the SqlTestUser to the list of valid accounts that can access the publication (see Figure 2).

Figure 2 Northwind Publication Properties
Figure 2 Northwind Publication Properties

The final step is to create the first snapshot. Before a new subscriber can accept changes, it must contain all of the tables and data contained in the publication's defined articles. This means that the first time you create a snapshot and then replicate it to the subscriber, all database tables will be created and filled in the subscriber's database (in this case, the SQL Server CE database). On subsequent synchronizations, only modifications are sent between the publisher and subscriber databases.

Once the SQL Server CE Server Agent is set up, the publication has been created, all types of authentication have been defined, and the initial snapshot has been established, the replication process can begin. Now all I need is a subscriber. Enter the sample inventory application.

The Application

The Inventory application is a Smart Device application written in C#. Since I have a Pocket PC, I targeted the application towards the Pocket PC device, but this could easily have been adapted to the more generic Windows CE-based device option. The application's basic requirements include synchronizing the inventory data from the Northwind back-end database with the handheld device. When a product is selected, its price and inventory levels are displayed in textboxes so the user can edit them.

One of the great features of developing with the .NET Compact Framework is that you don't need a Windows CE-based device to develop against. Instead, you can develop and debug using the Pocket PC emulator. The figures shown in this column are of the application running in debug mode in the emulator.

Figure 3 Synchronization Outcome
Figure 3 Synchronization Outcome

When the application is started, it looks to see if the Northwind database exists on the SQL Server CE instance. The first time the application is run the data has not yet been replicated over to SQL Server CE, so the input boxes are empty. When the user presses the Synch button, the application initiates the merge replication process, bringing over the initial snapshot of the NorthwindPub publication. Since the initial snapshot contains all of the Northwind data, it will take longer to synchronize than subsequent synchronizations. Once the publication has been published to the subscriber, a message box is presented showing the outcome of the synchronization (see Figure 3). The number of rows that were changed by the publisher from the initial snapshot are displayed, along with zero conflicts from the publisher and zero subscriber changes. The initial synchronization also creates the SQL Server CE database file on the Windows CE-based device to store the Northwind data.

Figure 4 Data Displayed
Figure 4 Data Displayed

Once the database has been created, the application displays the list of categories in a combobox control. Figure 4 shows how in the category Beverages, the product Chai is selected and the Chai product's inventory levels and unit price are displayed. At this point, the user can browse and/or edit the products in the different categories. As the user makes changes to the inventory levels, the data is saved to the local DataSet. Basically, the DataSet is loaded from the local SQL Server CE database after a synchronization. The DataSet is then bound to the product's controls and tracks all data changes made by the user.

Figure 5 Changing Inventory Value
Figure 5 Changing Inventory Value

Once the user presses the Save button, all changes are sent to the local SQL Server CE database using ADO.NET. The changes stored in the DataSet are sent to the database through the Update method of a SqlCeDataAdapter. Then, if a user presses the Synch button, the changes made to the local SQL Server CE database (the subscriber) are sent to the back-end database (the publisher) and if any changes were made at the publisher, they are sent to the subscriber through merge replication. Figure 5 shows the outcome after a user changed a single inventory value and no data changes were made at the back-end database.

The Code

The code that makes this all click requires several namespaces to be included in the project. The basic ones required by a smart device application are System, System.Drawing, System.Collections, System.ComponentModel, and System.Windows.Forms.

The additional namespaces that I included are used to write the database file to the handheld device, to communicate with the SQL Server CE database, and to store the product data in a DataSet. Thus, the following namespaces are also required by this project:

using System.Data.SqlServerCe;
using System.Data;
using System.Data.Common;
using System.Text;
using System.Xml;

When the application first starts, the connection to the database is defined using a SqlCeConnection object. This object works just like the SqlConnection object, except that it is used to communicate with SQL Server CE databases. Its datasource and connection string are much simpler than the SqlConnection object's connection string since it only has to point to the database file. A SqlCeDataAdapter is also established, with its SelectCommand and UpdateCommand properties set with their corresponding SELECT and UPDATE SQL statements. The DeleteCommand and InsertCommand properties do not need to be set since this application will not allow the user to perform those actions on the inventory data.

Once the application is started, it tries to load the data from the local database. First, the list of categories is retrieved using a SqlCeCommand and a SqlCeDataReader. The class's constructor executes and sets up the ADO.NET objects that were used throughout the application. The SqlCeCommand defines the SQL statement to retrieve the categories. After the SqlCeConnection has been opened and its ExecuteReader method called, the SqlCeCommand returns a SqlCeDataReader that points to the category data. The SqlCeDataReader is then iterated through to add the category data to the cboCategory combobox (see Figure 6).

Figure 6 Loading the Categories (Close Figure 6)
// Get the categories via a DataReader
SqlCeCommand oCmd = new SqlCeCommand("SELECT CategoryID,
    CategoryName FROM Categories " +
    " ORDER BY CategoryName", this.m_oCn);
this.m_oCn.Open();
SqlCeDataReader oDR = oCmd.ExecuteReader();

// Clear the category list and fill it
cboCategory.Items.Clear();
while (oDR.Read())
{
    cboCategory.Items.Add(new
        Category((string)oDR["CategoryName"],
        (int)oDR["CategoryID"]));
}
oDR.Close();
this.m_oCn.Close();

// Select the first category
cboCategory.SelectedIndex = 0;

Notice that the items are added to the combobox control in Figure 6 by adding an instance of a Category class to the combobox. The Category class has CategoryName and CategoryID public properties. The CategoryName is displayed in the combobox and the CategoryID is used to identify a unique category. It also has a ToString method which overrides the default ToString method of an object to return the CategoryName. This is how the cboCategory combobox control knows which value to display.

The products are also retrieved when the application loads, storing the entire list of products in a DataSet. The SqlCeDataAdapter has already been set up in the class's constructor to retrieve all of the products, so calling the Fill method of the SqlCeDataAdapter loads the DataSet, as shown here:

this.m_oDA.Fill(this.m_oDs, "Products");

Notice how in Figure 6, after the categories have been loaded into the cboCategory combobox, the first category in the list is selected. This causes the FilterProducts method to be invoked, which filters the products DataSet to only show the products that belong to the selected category. Figure 7 shows the main code in the FilterProducts method, which first grabs the selected category. This is done by casting the cboCategory combobox control's SelectedItem property to the Category class. Then, a DataView object is created that filters the products DataSet by the selected CategoryID value. The oDV DataView does not remove the data from the DataSet, but rather filters the DataSet so only the desired products are accessible via the DataView.

Figure 7 Filtering the Products (Close Figure 7)
// Get the selected category and cast it back to the Category class
Category oCat = (Category)cboCategory.SelectedItem;

// Create a DataView that filters the Products DataTable by the
// selected category
DataView oDV = new DataView(this.m_oDs.Tables["Products"],
    "CategoryID = " +
    Convert.ToString(oCat.CategoryID), "ProductName",
    DataViewRowState.CurrentRows);

// Clear all bindings
cboProduct.DataBindings.Clear();
txtUnitPrice.DataBindings.Clear();
txtUnitsInStock.DataBindings.Clear();
txtUnitsOnOrder.DataBindings.Clear();
txtReorderLevel.DataBindings.Clear();

// Bind the DataView to the controls
cboProduct.DataSource = oDV;
cboProduct.DisplayMember = "ProductName";
cboProduct.ValueMember = "ProductID";
txtUnitPrice.DataBindings.Add("Text", oDV, "UnitPrice");
txtUnitsInStock.DataBindings.Add("Text", oDV, "UnitsInStock");
txtUnitsOnOrder.DataBindings.Add("Text", oDV, "UnitsOnOrder");
txtReorderLevel.DataBindings.Add("Text", oDV, "ReorderLevel");

// Select the first Product
cboProduct.SelectedIndex = 0;

Next, all controls' bindings are removed and reset to the new DataView object. This process allows the application to load all products into a DataSet at once and only show a filtered subset of the data to the user. Keep in mind that any pending changes made by the user to the data is still tracked, whether or not the data is visible in the current DataView. For example, let's assume that a user changes the reorder level for the Chai product under the Beverages category and then changes the category to Seafood. Even though the Chai product is no longer in the current DataView, it is still in the DataSet. The pending change still exists and will continue to exist until the user presses either Cancel, which calls the DataSet's RejectChanges method, or presses the Save button.

When the user presses the Save button, the first step is to make sure that the last edit that the user made was captured. This is ensured by grabbing the row number of the row that is currently visible in the bound controls. Then the row number is used to reference the current row on the DataSet's Products table so EndEdit can be invoked. This method tells the DataRow that editing of the row is complete, so if any changes were made, they are captured:

//— Make sure the last edit was captured.
int nPosition = BindingContext[this.m_oDs.Tables["Products"]].Position;
this.m_oDs.Tables["Products"].Rows[nPosition].EndEdit();

In the Inventory application, saving occurs at three different levels: locally to the DataSet, from the DataSet to the local SQL Server CE database, or between the local SQL Server CE database and the back-end SQL Server database through merge replication.

When the user presses the Synch button, the merge replication process is kicked off. Figure 8 shows the code that instantiates the SqlCeReplication object (part of the System.Data.SqlServerCe namespace) and sets all of its properties required for merge replication to take place between the back-end and local databases. The Publisher property is set to the name of the server where the publication exists. The PublisherLogin and PublisherPassword properties are set to the SQL Server accounts that I created and gave permissions to the publication. The InternetUrl, InternetLogin, and InternetPassword properties are used to tell the subscriber where to send its request for merge replication. For merge replication to occur, the subscriber has to talk through the SQL Server CE Server Agent, which in this application is located at http://lancelot/sqlce/sscesa20.dll.

Figure 8 Kicking Off Merge Replication (Close Figure 8)
//Set up the Replication properties
oRpl = new SqlCeReplication();
oRpl.Publisher = "lancelot";
oRpl.PublisherLogin = "SqlTestUser";
oRpl.PublisherPassword = "hmmm";
oRpl.InternetUrl = "http://lancelot/sqlce/sscesa20.dll";
oRpl.InternetLogin = "iusr_lancelot";
oRpl.InternetPassword = "";
oRpl.Subscriber = "CESubscriberTest";
oRpl.Publication = "NorthwindPub";
oRpl.PublisherDatabase = "Northwind";
oRpl.SubscriberConnectionString =
"Provider=Microsoft.SQLSERVER.OLEDB.CE.2.0;Data Source=" +
this.m_sDataSource;

The Subscriber property of the SqlCeReplication object is set to the name of the subscriber, which is used to identify which subscriber is requesting the merge replication process to be kicked off. There could be multiple subscribers, all running various Pocket PC devices. Therefore, the subscriber name can assist the publisher in tracking the subscribers' changes. The Publication and PublicationDatabase properties must be set to the name of the publication and database, respectively, that the device wants to replicate with. Finally, the SubscriberConnectionString property is set to point to the SQL Server CE database instance so the replication process can read changes from the SQL Server CE database and send any changes to the database. Once these properties have been set, the SqlCeReplication object's Synchronize method can be invoked and the data can be reloaded into the application:

// Perform the synchronization and load the data
oRpl.Synchronize();
LoadData(false);

Exception Handling

The work involved in developing a Smart Device Application project in Visual Studio .NET can be divided into a two main categories: configuring merge replication and writing the application. Although configuring merge replication requires careful planning, the process is straightforward. Developing the application for the handheld device is very similar to developing a Windows Forms app. However, debugging errors in the application may make you pull your hair out. The debugging tools are very useful, but just like any application, they are only as useful as the information that they yield from exceptions. A solid exception handling strategy is just as critical in developing applications for handheld devices as it is for any other project type.

When you consider all of the replication configuration settings, security requirements, authentication, installation prerequisites, and connectivity issues, there are many ways that things can go amiss. It is important to be able to determine what causes an error, and the SqlCeException object generally contains useful information on where to focus your attention.

Figure 9 shows the ShowSqlCeErrors method that iterates through the SqlCeError objects, which represent the different errors that occurred. The ShowSqlCeErrors method loops through the SqlCeError objects and several of its properties to retrieve the details of the errors. It then formats the error information into a string and displays each error within the exception to the user via the MessageBox object. This method is quite simple to implement and I recommend that it be used as it can save a ton of wasted time in chasing down the root cause of exceptions.

Figure 9 Displaying Exceptions Gracefully (Close Figure 9)
private void ShowSqlCeErrors(SqlCeException exSql)
{
    StringBuilder oSBMsg = new StringBuilder();

    int i = 0;

    foreach (SqlCeError err in exSql.Errors)
    {
        i++;

        oSBMsg.Append("\n Error # " + i + " of " +
            exSql.Errors.Count);
        oSBMsg.Append("\n Error Code: " + err.HResult);
        oSBMsg.Append("\n Message   : " + err.Message);
        oSBMsg.Append("\n Minor Err.: " + err.NativeError);
        oSBMsg.Append("\n Source    : " + err.Source);

        for (int j = 0; j < err.NumericErrorParameters.Length;
            j++)
        {
            if (err.NumericErrorParameters[j] != 0)
            {
                oSBMsg.Append("\n Numeric Parameter: " +
                    err.NumericErrorParameters[j]);
            }
        }

        for (int k = 0; k < err.ErrorParameters.Length; k++)
        {
            if (err.ErrorParameters[k] != string.Empty)
            {
                oSBMsg.Append("\n Error Parameter  : " +
                    err.ErrorParameters[k]);
            }
        }
        MessageBox.Show(oSBMsg.ToString());
        oSBMsg.Remove(0, oSBMsg.Length);
    }
}

Conclusion

In this sample I've demonstrated how to implement an application for a Windows CE-based device that allows users to take advantage of disconnected data. The data can be stored, viewed, and manipulated on the handheld device and then resynchronized with the back-end database through the local network or even the Internet. One of the fundamental players in this application is ADO.NET. Its disconnected nature and XML foundation allow the data to be very portable yet, at the same time, highly functional. Using these techniques and tools, developing an enterprise business solution that includes mobile devices is within the reach of developers.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive. You can reach him at mmdata@microsoft.com.


Cutting Edge: Managing Your Remote Windows Clipboard

Cutting Edge
Managing Your Remote Windows Clipboard
Dino Esposito

Code download available at: CuttingEdge0309.exe (1,286KB)
Contents
Setting Up a Remote Component
The .NET Clipboard API
Building a Remote Clipboard Handler
The Remote Clipboard Client
Wrap-up

When DCOM appeared on the scene a few years ago, one of the most commonly used examples to illustrate its capabilities was a remote clipboard manager. Using the DCOM programming model, a component could read and write the contents of the clipboard that was stored in another machine but connected to the same network. (Of course this only worked if the security settings allowed for it.)

But while DCOM provided the infrastructure to build remote access to a system component like the clipboard, neither Windows® nor DCOM provided an API for direct remote access to the clipboard. The trick that developers could implement was based on the interaction between a local proxy and a remote stub. The application calls into a local proxy, which in turn serializes the call over the network transportation layer and delivers it to a remote host. Next, the application host instantiates a local copy of the clipboard handler component to read from and write to the clipboard of the local copy of Windows.

The Microsoft® .NET Framework provides the Clipboard class to wrap the main operations on the system clipboard. The Clipboard class is declared in the System.Windows.Forms namespace as part of the Windows Forms infrastructure. The methods on the class allow you to get and set the current contents of the clipboard in the context of a single application.

A client of mine was building an ASP.NET e-commerce site that did not utilize the clipboard. However, one of the developers on the team realized that the guys responsible for the population of back-end tables were continuously transporting chunks of data (mostly plain text) from machine to machine. The fastest way they found to do this was to create temporary text files across network shares. While this specific trick was acceptable, the overall process was not especially smart. In particular, the text was first highlighted in Microsoft Word or Microsoft Internet Explorer, copied to the clipboard, and then pasted into a new Notepad document. Finally, the document was dragged and dropped onto the network folder. Armed with good will and warm DCOM memories, that insightful developer thought that a made-to-measure remote clipboard viewer and manager would do the job more quickly and effectively.

Figure 1 The Clipboard Viewer
Figure 1 The Clipboard Viewer

The Clipboard Viewer (see Figure 1) is an old Windows accessory that is no longer referenced in the Start menu, but is still available to you from within the System32 folder as clipbrd.exe. The Clipboard Viewer acts as the manager of the clipboards on all machines connected to a network. If the current security settings allow it, you can connect to a remote instance of Windows and spy into the machine's clipboard. The viewer, though, is just a viewer, so it displays the current content and lets you delete it, but it doesn't provide a user interface to enter new text to the clipboard. In addition, the Clipboard Viewer is based on a distributed technology that was declared obsolete decades ago—the Network Dynamic Data Exchange, or NetDDE for short. So my client decided to write a customized version of the Clipboard Viewer as a truly distributed application. Since he's working with the .NET Framework, he designed the utility with .NET Remoting in mind.

Setting Up a Remote Component

.NET Remoting is the mechanism that enables communication between components running in different AppDomains. All .NET Framework-based applications are made of at least one (primary) AppDomain, but more AppDomains can be created programmatically. The AppDomain represents a managed subprocess living in the context of the physical process managed by the operating system and the CPU. The common language runtime (CLR) guarantees that any data contained in an AppDomain is inaccessible from within another AppDomain. The mechanism of separation is identical, whether the two application domains are in the same application, in two distinct applications within the same machine, or running on physically separate computers.

Architecturally speaking, the role of .NET Remoting lies in the system's ability to marshal the call context from the client to the server and then send the results back to the client. A remote component is a class with public methods that inherits—directly or indirectly—from MarshalByRefObject. The class, compiled into an assembly, is deployed on the server machine and interacts with clients through a host application. The host application listens on a particular port for incoming calls, transforms the parameters it receives into a local call to the object, and marshals the return values back to the caller. Figure 2 shows the architecture of a remote component that accesses the clipboard of a network machine.

Figure 2 Remote Component Accessing the Clipboard
Figure 2 Remote Component Accessing the Clipboard

The client running on, say, Machine1 issues a remote call to a server-side object residing on, say, Machine2. The call is received by the application host listening on the agreed port and is then processed. The call context includes the name of the remote method to call and its parameters. A component that wants to encapsulate the functions of the clipboard will be implemented as shown in Figure 3. The marshal-by-ref class will expose a Copy method to let clients write text to the remote clipboard and a Paste method to bring the contents of the remote clipboard into the local context. The code in Figure 3 and the underlying pattern should be easy to understand, especially if you've read my .NET Remoting introductory article in the October 2002 issue (see .NET Remoting: Design and Develop Seamless Distributed Applications for the Common Language Runtime). So far, so good. However, as you may have noticed, the bodies of the Copy and Paste methods are empty. Originally I thought that would be the most trivial part of the code. Well, I was dead wrong. I solved the problem only by brushing up a bit on Win32® and Visual Studio® 6.0.

Figure 3 Encapsulating the Clipboard (Close Figure 3)
using System;
using System.Runtime.InteropServices;

namespace MsdnMag
{
    public class ClipboardHandler : MarshalByRefObject
    {
        public ClipboardHandler()
        {
        }

        // Copy the specified text to the local clipboard
        public void Copy(string text)
        {
            // TODO
        }

        // Return the object currently stored in the clipboard
        public string Paste()
        {
            // TODO
        }
    }
}

The .NET Clipboard API

As mentioned earlier, a .NET Framework-based application manages the clipboard using the Clipboard class in the System.Windows.Forms namespace. The class, which doesn't have to be instantiated, contains a pair of static methods—GetDataObject and SetDataObject. GetDataObject retrieves the data that is currently stored on the system clipboard; SetDataObject places the specified data object in the system's memory.

The clipboard supports a variety of formats for the data, including user-defined formats. The Win32 clipboard API defines a bunch of predefined formats and identifies them with a mnemomic constant (an integer value) such as CF_TEXT or CF_BITMAP.

Since the clipboard is a system component, the .NET Framework class that represents it only provides a wrapper for a bunch of low-level API functions. You may be surprised to learn that the .NET Framework implementation of the clipboard is not based on raw Win32 functions and messages. The Clipboard class available in the .NET Framework performs operations through the OLE Clipboard protocol. OLE you say? Yes, that's right. This column is going to revive some old technologies that set important milestones in the evolution of Windows.

About 10 years ago, Microsoft introduced OLE as an all-encompassing component technology. Quoting from Kraig Brockschmidt's Inside OLE (Microsoft Press, 1995), OLE is defined as a "unified environment of object-based services." Unlike with Win32, though, you can't write a complete application using just OLE (or its successor, COM). However, OLE takes some existing system features and exposes them in a more general and extensible way. So what's the OLE Clipboard protocol?

Basically, the OLE Clipboard is a set of interfaces aimed at generalizing the data exchange that takes place between Windows-based applications and the clipboard. The Platform SDK defines only a few base types of data (mostly text and bitmap). The OLE Clipboard protocol extends the model by providing support for any data format and storage medium. The OLE Clipboard protocol is not a strict requirement for Windows-based applications as long as the applications don't need to go beyond the data formats offered by the Platform SDK. The OLE-based protocol is more powerful than the Win32 clipboard API and was a much more reasonable choice when it came to building clipboard support in the .NET Framework.

The key interface for clipboard operations in both OLE and the .NET Framework is IDataObject. These two interfaces have different sets of methods but play the same role. IDataObject provides a format-independent mechanism for transferring data and is used by the Clipboard class and in drag and drop operations. (In the .NET Framework, the OLE IDataObject interface has been renamed to IOleDataObject.) Figure 4 lists the methods defined on the IDataObject interface.

Figure 4 IDataObject Methods (Close Figure 4)

Method | Description
GetData | Returns the data object that is associated with the specified data format; predefined data formats are defined in the DataFormats enum type
GetDataPresent | Indicates whether the data stored in the clipboard is of the specified format
GetFormats | Retrieves a list of all the data formats currently contained in the clipboard
SetData | Stores the specified data and its associated format in the clipboard

The following code snippet shows how a .NET Framework-based application can copy plain text to the clipboard:

Clipboard.SetDataObject(text);
The SetDataObject method takes an object and copies it to the clipboard as an IDataObject instance. If the parameter is an object that already implements the IDataObject interface, the copy is direct. Otherwise, the method packs the object into a dynamically created instance of the DataObject class, as shown in Figure 5.

Figure 5 Packing into a DataObject Instance (Close Figure 5)
// Use OLE methods and cast to IOleDataObject
if (data as IOleDataObject != null)
{
   IOleDataObject o = (IOleDataObject) data;

   // Call the OLE Clipboard API
   Win32Methods.OleSetClipboard(o);
}
else
{
   DataObject o = new DataObject(data);

   // Call the OLE Clipboard API
   Win32Methods.OleSetClipboard(o);
}

The DataObject class implements both the IDataObject and the IOleDataObject interfaces. The Clipboard's SetDataObject method accepts a generic object argument:

public static void SetDataObject(object);
public static void SetDataObject(object, bool);
This allows you to pass in simple data like text and bitmaps as native objects and leaves the burden of the data adaptation to the built-in infrastructure. The SetDataObject method also has an overload that requires an extra Boolean argument. It indicates whether the data placed in the clipboard should remain available after the current application terminates. If you use the one-parameter overload, the contents of the clipboard are flushed upon exit.

Reading data from the clipboard is a bit more articulated than writing to it because the data extrapolation cannot be delegated to the .NET Framework. The following code snippet shows how managed applications read from the clipboard:

IDataObject data;
data = Clipboard.GetDataObject();
The GetDataObject method returns a IDataObject package that the application must unwrap:
// Verify that the data object contains plain text
if (data.GetDataPresent(DataFormats.Text))
{
   // Extrapolate and display the text
   string text = data.GetData(DataFormats.Text);
   MessageBox.Show(text);
}
The GetDataPresent method on the IDataObject interface (see Figure 4) takes an argument that identifies a type of data—for example, plain text. The method returns true if the content of the data object matches the specified type. Note that the method returns true either way, whether the native type of the stored object matches or the object's type can be converted to the desired type. For example, if the data object contains HTML text and the user asks for plain text, GetDataPresent returns true.

Using the clipboard API from within a .NET Framework-based application is a no-brainer, but not when you try to use it from within a remotable object.

Building a Remote Clipboard Handler

A Windows Forms application normally runs in single-threaded apartment (STA) mode. C# projects make this clear up front by adding an [STAThread] attribute to the Main routine of the application. In Visual Basic® .NET, this setting is implicit. The STA mode is a strict requirement for many GUI applications as they rely on services exposed by operating systems that don't always support pure multithreaded environments. This is typical of OLE and COM services such as the clipboard and drag and drop operations.

As a matter of fact, you can't use the Clipboard object from within a thread picked up from an MTA pool. Try out the following minimal console application:

using System.Windows.Forms;
class Test {
  [MTAThread]
  static void Main() {
    Clipboard.SetDataObject("MSDNMag");
  }
}
It throws a thread-state exception, like the one shown in Figure 6.

Figure 6 Thread-state Exception
Figure 6 Thread-state Exception

Before OLE calls can be made in a .NET Framework-based application, programmers must ensure that the current thread operates in an STA. Managed objects, in fact, are responsible for exposing their shared data in a thread-safe way. The .NET Framework supports thread apartments for backward compatibility only. In contrast, COM components use apartments. For this reason, the CLR needs to create an apartment before any interoperation takes place with a COM object. The STAThread and MTAThread attributes are the declarative programming interface for choosing the threading model for the application. In the previous code snippet, the MTAThread attribute sets the console application to create a managed thread and configure it to enter an MTA apartment. As soon as the application calls into the OLE/COM world, the apartment discrepancy is detected and an exception is thrown. In general, though, console and Windows Forms applications run in STA mode unless otherwise specified.

When I first built a marshal-by-ref object to copy text to a remote clipboard, I left the default settings untouched. Yet the thread exception was raised as soon as the execution flow reached the Clipboard class. Remember that remoting calls are always resolved through MTA threads.

Before calling into Win32 OLE methods, the Clipboard class performs a preliminary check against the apartment mode of the current thread. It does this by issuing a call to the Application.OleRequired method. This method verifies that OLE is initialized on the current thread or initializes it if necessary. The method returns a value from the ApartmentState enumeration, which lists three feasible values—MTA, STA, and Unknown. You can check the current threading model by using the following code:

Console.WriteLine(Application.OleRequired().ToString());

The client application calling into the remote object was a single-threaded Windows Forms application. The .NET Remoting host was a console application explicitly marked as STA. But the apartment state of the thread serving the remoting call was an MTA. The error message that you see in Figure 6 was, at that point, a foregone conclusion.

I couldn't find any ideas on how to work around the issue in the .NET Remoting documentation and in the available literature (including the excellent http://www.dotnetremoting.cc). So I took the advice that Kraig Brockschmidt shares in his chapter on the OLE Clipboard. Kraig points out that you should actually use the OLE Clipboard (as the .NET Framework does) only if it gives you added value. As long as you need to exchange only text and bitmaps, you can stick to the Win32 clipboard API. And as a bonus, you save the threading hassles. Figure 7 shows a Win32 DLL that exports a couple of public functions to copy and paste plain text to the clipboard.

Figure 7 Exporting Public Functions to Copy/Paste Text (Close Figure 7)
BOOL APIENTRY DllMain( HANDLE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved)
{
    return TRUE;
}

BOOL APIENTRY CopyToClipboard(LPCTSTR text)
{
    // Open and own the clipboard
    OpenClipboard(NULL);
    EmptyClipboard();

    // Allocate some memory for the string to copy
    HGLOBAL hglbCopy;
    hglbCopy = GlobalAlloc(GHND, (lstrlen(text)+1));
    if (hglbCopy == NULL)
    {
        CloseClipboard();
        return FALSE;
    }

    // Copy the text to the object
    LPTSTR lpsz = (LPTSTR) GlobalLock(hglbCopy);
    lstrcpy(lpsz, text);
    GlobalUnlock(hglbCopy);

    // Copy to the clipboard
    SetClipboardData(CF_TEXT, hglbCopy);

    // Release the clipboard
    CloseClipboard();
    return TRUE;
}

BOOL APIENTRY PasteFromClipboard(LPTSTR buffer)
{
    if (!IsClipboardFormatAvailable(CF_TEXT))
        return FALSE;
    if (!OpenClipboard(NULL))
        return FALSE;

    // Get the handle from the clipboard
    HGLOBAL hglb = GetClipboardData(CF_TEXT);
    if (hglb != NULL)
    {
        LPTSTR lpsz = (LPTSTR) GlobalLock(hglb);
        if (lpsz != NULL)
        {
                    memcpy(buffer, lpsz, lstrlen(lpsz));
                    buffer[lstrlen(lpsz)] = 0;
                    GlobalUnlock(hglb);
        }
        else
        {
            CloseClipboard();
            return FALSE;
        }
    }

    // Release the clipboard
    CloseClipboard();

    return TRUE;
}

When coding the Win32 way, you must first open and empty the clipboard. You need a window handle to open the clipboard because the clipboard can be owned only by a window object. OpenClipboard is the API function to call. The EmptyClipboard function frees any handles of global data that are stored in the clipboard. After that, the window that currently has the clipboard open becomes the new owner. To copy data to the clipboard you must allocate a block of global memory, preferably marked with the GHND flag—which means moveable and initialized to zero. Any data copied to or read from the clipboard is packed into a handle of global memory. At the Win32 level, the kinds of data formats that can be packed into that storage medium is limited to a few such as HTML, plain text, or device-independent bitmaps. If you use OLE instead, more formats and storage media are allowed.

Using the Win32 DLL—and the P/Invoke platform for Win32 interoperability—limits the usability of the clipboard to a few formats, but doesn't require changes to the threading model and works just fine through remoting. Figure 8 presents the final code for the remote clipboard handler object. The two DLL public functions are mapped to static external members of the marshal-by-ref class. The signature of CopyToClipboard is not hard to translate to the CLR type system. You change LPCTSTR with String and you're finished. Importing the PasteFromClipboard function is a bit trickier. In Figure 7, the function is declared to accept a string by reference and return a Boolean value.

Figure 8 Remote Clipboard Handler Object (Close Figure 8)
using System;
using System.Text;
using System.Runtime.InteropServices;

namespace MsdnMag
{
    public class ClipboardHandler : MarshalByRefObject
    {
        public ClipboardHandler()
        {
        }

        // Copy the specified text to the local clipboard
        public void Copy(string text)
        {
            bool result = CopyToClipboard(text);
        }

        // Return the object currently stored in the clipboard
        public string Paste()
        {
            StringBuilder buf = new StringBuilder("");
            PasteFromClipboard(buf);
            return buf.ToString();
        }

        // Imports from the Win32 DLL
        [DllImport("clip32.dll")]
        private static extern bool CopyToClipboard(string
            text);
        [DllImport("clip32.dll")]
        private static extern bool
            PasteFromClipboard(StringBuilder text);
    }
}

An efficient way to translate a similar signature to .NET code entails the use of the StringBuilder object:

[DllImport("clip32.dll")]
private static extern bool PasteFromClipboard(StringBuilder text);
You first replace the reference to a string with an initialized StringBuilder object. Next, you use the ToString method on the StringBuilder class to obtain the string:
StringBuilder buf = new StringBuilder("");
PasteFromClipboard(buf);
return buf.ToString();
You should note that a StringBuilder object differs from a String object in that it is a growable object as opposed to the immutable nature of traditional strings. In other words, when you concatenate two strings, the .NET Framework creates a new one whose size is the sum of the two. When you add a string to a StringBuilder object, the input text is simply appended to the existing buffer.

The Remote Clipboard Client

The Remote Clipboard application is made of two elements—the client and the host server for the remote component. To deploy it, you must copy the host server (see Figure 9) and the assembly with the remote component to any machine you want to reach. The client application must be copied to all machines from which you want to copy or paste across the network.

Figure 9 Remote Clipboard App
Figure 9 Remote Clipboard App

The .NET Remoting infrastructure does not automatically start the server-side host to receive the incoming calls to a remote component. It's the user's responsibility to have such a stub program up and running. You can use Microsoft Internet Information Services (IIS) for this task or write your own application. Using IIS has some limitations (HTTP channels are required), but it saves you from administering the remoting host. Alternately, you can write a custom application that simply registers a server channel (TCP, HTTP, or a custom type) and associates the remote type with a well-known type through a server URI.

In Figure 10, the host is a console application that creates a TCP channel for port 2222 and marks the MsdnMag.ClipboardHandler type (see Figure 8) as well-known. The URI for calling the object is ClipboardHandler. The console application must be started and terminated manually. You can also decide to write it as a GUI application and provide a user interface to pause or terminate the port monitoring. To pause the monitoring, you unregister the channel. If you don't want to write a console application, you can opt to create a Windows Service, which offers the same functionality without requiring start/stop operations to be handled manually.

Figure 10 Console Application (Close Figure 10)
Imports System
Imports System.Runtime.Remoting
Imports System.Runtime.Remoting.Channels
Imports System.Runtime.Remoting.Channels.Tcp

' Console application acting as the remote host for the
' CLIPBOARDHANDLER component
Class RemoteClipboardServer

    Shared Sub Main()

        ' Create a TCP channel object used to receive calls
        Dim channel As TcpServerChannel = New TcpServerChannel(2222)
        ChannelServices.RegisterChannel(channel)

        ' Register the remotable object as a well-known type
        RemotingConfiguration.RegisterWellKnownServiceType( _
             GetType(MsdnMag.ClipboardHandler), _
             "ClipboardHandler", _
             WellKnownObjectMode.SingleCall)

        ' Display a minimal UI to stop the service
        ' Users are responsible for starting this application
        Console.Write("Press [Enter] to stop the service...")
        Console.ReadLine()
    End Sub

End Class

The client of a .NET Remoting component must accomplish a basic task—making the remote type recognizable by the rest of the application. This task is carried out by using one of the static members of the RemotingConfiguration class, the RegisterWellKnownClientType method. The following code snippet shows how to register a type as well-known:

RemotingConfiguration.RegisterWellKnownClientType(
  typeof(MsdnMag.ClipboardHandler),
  "tcp://expo-two:2222/ClipboardHandler");

The RegisterWellKnownClientType method takes two arguments. The first is the type in question. The second is a URI that includes the transportation protocol for the marshaling, the server name or IP address, the port to use, and the nickname of the remote object, as shown in the previous code. Of course, the port must match the port on which the remote host is listening. While the concept of a well-known type is easy to understand, a remote type is not defined locally and any reference to any methods or properties must be handled differently. In front of each invocation, the compiler must generate some vanilla code to marshal the call down to the remote server and back. For this reason, any reference to a remote object that is found in the source code must be recognized and flagged so that the just-in-time (JIT) compiler can properly generate dynamic code for it.

The internal implementation of the RegisterWellKnownClientType method is not rocket science. It caches the type information and adds it to a global hash table of well-known types. The type is used as the key, whereas the URI string is used as the value of the pair. In no way does the programming interface of well-known types allow you to unregister a type. If you try to redirect a well-known type to another URI, an exception is thrown.

As you can see, the RegisterWellKnownClientType (and similar methods such as RegisterActivatedClientType) works by establishing a one-to-one relationship between a type and a server URI. What if the application needs to invoke the same type from different servers? Well, this is exactly the problem I faced next in building a remote clipboard handler.

As Figure 10 shows, the client app must register the same remote type multiple times, once for each connected server. The remote clipboard client must be able to read and write the clipboard of all the machines available to the network. Each of these machines will run the same host and an instance of the same type—the MsdnMag.ClipboardHandler class from Figure 8. How can you register the same type to different URIs more than once? You have two options. The first requires that you bypass the built-in configuration mechanism. You don't flag the remote type as well-known, instead simply use one of the overloads of the Activator.GetObject method to create and get a remote instance of the object:

string uri = @"tcp:\\expo-two:2222\ClipboardHandler";
object o = Activator.GetObject(typeof(MsdnMag.ClipboardHandler), uri);
MsdnMag.ClipboardHandler clip;
clip = (MsdnMag.ClipboardHandler) o;
The GetObject method gets or creates a proxy for the object indicated by the specified type and URL. This solution provides you with the greatest flexibility as it doesn't depend on the number of servers and their location.

Second, if you know exactly what servers your client will always work with, you can go for a more rigid approach that extends the traditional well-known-type approach to the case of multiple servers. The idea is to rename the remote type by deriving a nearly identical new class. The new class is rooted in a namespace named after the machine. (This is arbitrary, though.)

The code in Figure 11 shows how to rename the MsdnMag.ClipboardHandler class using inheritance. The new classes in the ExpoTwo and ExpoStar namespaces add no extra code and no overhead is required if you recompile. To support a new server, you must add a new namespace declaration. You should note that this solution is less flexible because it hardcodes the names of the servers and any change (such as a new server added to the list) requires a recompile. The following code illustrates a remote call from within the client:

ExpoTwo.RemoteClipboardHandler rc;
rc = new
  ExpoTwo.RemoteClipboardHandler();
rc.Copy(TextToCopy1.Text);

Figure 11 Rename a Class Using Inheritance (Close Figure 11)
namespace MsdnMag {
  public class ClipboardHandler : MarshalByRefObject {
    :
  }
}

namespace ExpoTwo {
  public class RemoteClipboardHandler : MsdnMag.ClipboardHandler
  { }
}

namespace ExpoStar {
  public class RemoteClipboardHandler : MsdnMag.ClipboardHandler
  { }
}

When a method on the well-known type is called, the .NET Remoting infrastructure verifies that the host application is up and running. If this is not the case, a socket exception is thrown. The specific exception raised is SocketException; the class belongs to the System.Net.Sockets namespace. The sample project for the client application references the assembly that contains the remote object.

Although its functionality is unexceptional, this approach may not be optimal in real-world scenarios. Versioning, class dependencies, and even size are all reasons for figuring out an alternative approach. For example, you could avoid linking the original assembly by defining a base class with empty implementations for all the public methods or even interfaces. In the latter case, though, you can't just use the new operator on the client to get an instance of the remote object. By design, in fact, you can't call new on an interface. Instead, you can use Activator.GetObject to retrieve an instance of the object that implements the given interface at the specified URI. For more information on this specific point, and as a valuable resource for .NET Remoting-related work, have a look at http://www.ingorammer.com/RemotingFAQ.

Wrap-up

The client application shown in Figure 12 provides buttons to copy and paste plain text to and from specific machines. The physical operations on the system's clipboard are accomplished using a Win32 DLL that uses low-level API calls. A more powerful approach to the problem of sharing the clipboard across the machines in a network is presented in the article available at http://www.codeproject.com/dotnet/clipsend.asp.

Figure 12 Copy and Paste to Specific Machines
Figure 12 Copy and Paste to Specific Machines

For the purposes of my client, the Win32 base formats were enough. Hopefully this limitation will be removed in the future. My solution would be to mimic the distributed architecture of ASP.NET Session and introduce a Windows Service acting as the host and add a marshal-by-ref class that manipulates the clipboard remotely on an STA thread. Finally, I would add a couple of overloads to the methods of the existing Clipboard class.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Building Web Solutions with ASP.NET and ADO.NET and Applied XML Programming for .NET, both from Microsoft Press, he spends most of his time teaching classes on ASP.NET and speaking at conferences. Dino recently published Programming ASP.NET for Microsoft Press. Get in touch at dinoe@wintellect.com.


The XML Files: Introducing the Web Services Enhancements 2.0 Messaging API

[ Editor's Update - 6/16/2004: This article was written about the Technical Preview release of WSE 2.0, and some of the code and concepts changed for the final release. For up-to-date information on WSE 2.0, see Web Services Enhancements.]
The XML Files
Introducing the Web Services Enhancements 2.0 Messaging API
Aaron Skonnard

Code download available at: XMLFiles0309.exe (263KB)
Contents
Senders and Receivers
Receiving Messages
Sending Messages
Correlating Messages
Higher-level Messaging
Using TCP and HTTP
WseChat
Conclusion

The ASP.NET WebMethod framework makes it possible to build XML-based Web Services that communicate over HTTP. It accomplishes this by mapping a pair of SOAP messages—one for the request and another for the response—to traditional remote procedure call (RPC) invocations. This is the only mechanism provided by the Microsoft® .NET Framework for building Web Services without resorting to writing most of the SOAP infrastructure yourself (which would require writing a custom HTTP handler, for instance).

The popularity of the WebMethod framework and others like it has nearly convinced mainstream developers that Web Services are only useful for RPC over HTTP. The irony is that some of the most interesting and powerful Web Services of the future probably won't use either. The Web Services concept is designed around XML messaging over arbitrary transports. XML messaging applications can benefit from a variety of different communications protocols as long as they aren't detrimental to the fundamental interoperability goals of the system.

For example, when performance and reliability are key goals, some Web Services might benefit from using TCP directly instead of a higher-level protocol like HTTP. Other Web Services might benefit from using asynchronous communication patterns such as those offered by TCP, SMTP, or Microsoft Message Queue (MSMQ). In-process integration between different components of the same system could also be beneficial to certain Web Services. There are a variety of messaging possibilities and the Web Services platform is designed to accommodate all of them.

The Web Services Enhancements (WSE) 2.0 toolkit is the first of its kind for the .NET Framework that lets you start experimenting with other XML messaging possibilities beyond HTTP. WSE 2.0 is an add-on to the .NET Framework that provides additional Web Services support, mostly for the various Web Services Architecture (WSA) specifications that are continuing to evolve (note that WSA was previously known as GXA). WSE 2.0 specifically provides support for WS-Security, WS-Trust, WS-SecureConversation, WS-Policy, WS-SecurityPolicy, WS-Addressing, and several other specifications that were introduced with version 1.0 including WS-Routing, WS-Referral, and Direct Internet Message Encapsulation (DIME). In addition to the new WSA functionality, WSE 2.0 also provides an improved messaging API that supports other transports and custom message exchange patterns (MEPs).

The WSE 2.0 messaging API really spurs the imagination. As you begin working with it, you'll realize the potential of the Web Services platform and the common misconceptions surrounding its relationship to RPC and HTTP. This column introduces the new messaging features provided by WSE 2.0 and shows you how to begin using them to build XML-based messaging systems that leverage asynchronous messaging patterns.

You can download WSE 2.0 from the MSDN® Web Services Developer Center. You'll need to install the WSE 2.0 bits before you begin experimenting with the code that is shown throughout this column.

Senders and Receivers

Once you move away from the traditional RPC model you have to stop thinking in terms of clients and servers. These terms often lead to confusion when you start using nontraditional MEPs. Instead, it's more appropriate to think of the different entities in your Web Services architecture as SOAP senders and receivers. Some nodes are designed to send SOAP messages while others are designed to receive and process them. And these roles are completely interchangeable, meaning that a given node could sometimes act as a sender and at other times as a receiver.

WSE 2.0 emphasizes this thinking by providing two fundamental classes for sending and receiving SOAP messages: SoapSender and SoapReceiver. As their names suggest, SoapSender encapsulates the process of sending a SOAP message to a URI while SoapReceiver encapsulates the process of receiving a SOAP message at a particular URI.

WSE 2.0 supports sending and receiving messages using three communications protocols: in-process, TCP, and HTTP. In the case of in-process, the SOAP messages are delivered directly to the receiver without hitting the network layer. You indicate which mechanism you want to use (from either a SoapSender or a SoapReceiver) through a URI, which must conform to the following syntax in WSE 2.0:

protocol_scheme://host[:port_number]/path_info/...
The protocol scheme indicates which protocol to use, while the host and port number identify where to send the messages. The path info identifies which resource to access on that particular host. Figure 1 shows the WSE 2.0 protocol schemes for each one, along with an example. The new protocol scheme is soap.tcp, for using TCP communications.

Figure 1 WSE 2.0 URI Schemes (Close Figure 1)

Protocol | URI Scheme | Example
TCP | soap.tcp | soap.tcp://mymachine:333/math
HTTP | http | http://mymachine/math/myservice.wse

The .NET Framework provides the System.Uri class for representing URIs in your code. You can instantiate Uri objects by supplying the appropriate URI string to the constructor, like this:

Uri myuri = new Uri("soap.tcp://mymachine/math");
SoapSender and SoapReceiver require you to supply Uri objects to indicate the protocol and endpoint details to be used.

In addition to the support for new transports, another big difference between the WSE 2.0 messaging API and the existing WebMethod framework is that SoapSender and SoapReceiver require the developer to work with SOAP messages directly. Although this raises the bar a bit on the level of XML and SOAP development experience required, it also increases flexibility and places the emphasis on XML messaging.

WSE 2.0 provides the Microsoft.Web.Services.SoapEnvelope class for building and processing SOAP messages in your code. The SoapEnvelope class derives from System.Xml.XmlDocument and provides additional members that only apply to SOAP messages like the Envelope, Header, and Body properties. Since the SoapEnvelope class is really just a special implementation of the Document Object Model (DOM), the standard DOM APIs are available along with the other DOM helpers like SelectNodes and SelectSingleNode that provide XPath support.

For example, consider the following SOAP message that represents the canonical example of adding two numbers:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
   <s:Body>
      <v:add xmlns:v='urn:math'>
         <x>33</x>
         <y>66</y>
      </v:add>
   </s:Body>
</s:Envelope>

You can build this SOAP message programmatically through the SoapEnvelope class, as shown here:

SoapEnvelope CreateAddMessage()
{
    SoapEnvelope env = new SoapEnvelope();
// must create body before setting it
    env.CreateBody();
    env.Body.InnerXml = "<v:add xmlns:v='urn:math'><x>33</x>
                        <y>66</y></v:add>";
    return env;
}
Once you've created the SoapEnvelope object you can send it to an endpoint using SoapSender (more on this shortly). If you're building a SoapReceiver, on the other hand, you'd write code to process the incoming SoapEnvelope object, as shown here:
double ProcessAddMessage(SoapEnvelope e)
{
    // extract x and y values from incoming SOAP message
    double x = XmlConvert.ToDouble(
         e.SelectSingleNode("//x").InnerText);
    double y = XmlConvert.ToDouble(
         e.SelectSingleNode("//y").InnerText);
    return x+y;
}

Then, if in the future you need to send a response back to the original sender, you'd create another SoapEnvelope with the response information. The way you send SOAP messages around ultimately defines your MEP as well as the types of SoapSender and SoapReceiver objects you'll need to write.

Receiving Messages

A SoapReceiver is implemented by deriving a new class from Microsoft.Web.Services.SoapReceiver and overriding its Receive method, which takes a SoapEnvelope argument:

public class MathReceiver : SoapReceiver
{
protected override void Receive(SoapEnvelope e)
    {
        // implement Receive here
     }
}

Within Receive, it's your job to process the incoming SOAP message and perform the requested operation. For example, in the case of the add message shown before this last example, you would simply extract the x and y values and calculate the sum:

public class MathReceiver : SoapReceiver
{
    protected override void Receive(SoapEnvelope e)
    {
   // extract x and y values from incoming SOAP
   // message
   double x = XmlConvert.ToDouble(
       e.SelectSingleNode("//x").InnerText);
   double y = XmlConvert.ToDouble(
       e.SelectSingleNode("//y").InnerText);
   double result = x+y;

       // do something with result
     }
}

If you want your receiver to support multiple operations, you need to key off of either the message's action (available through SoapEnvelope.Context.Action) or the request element's name. For example, you could update the implementation of the Receive method to support several math operations (see Figure 2).

Figure 2 Updated Receive Method (Close Figure 2)
public class MathReceiver : SoapReceiver
{
    protected override void Receive(SoapEnvelope e)
    {
       // extract x and y values from incoming SOAP message
       double x = XmlConvert.ToDouble(
           e.SelectSingleNode("//x").InnerText);
       double y = XmlConvert.ToDouble(
           e.SelectSingleNode("//y").InnerText);
       double result;

       // determine which operation to perform from 'action'
       switch (e.Context.Action)
       {
        case "urn:math:add":
            result = x + y;
            break;
        case "urn:math:sub":
            result = x - y;
            break;
        case "urn:math:mul":
            result = x * y;
            break;
        case "urn:math:div":
            result = x / y;
            break;
        default:
            throw new Exception("unsupported operation");
       }

       // do something with result
    }
}

The Receive method is called whenever a message arrives for a particular URI. You use the Microsoft.Web.Services.SoapReceivers class to register a SoapReceiver object with a particular URI. SoapReceivers provides a static Add method that accepts a Uri object along with the receiver to call when messages arrive for that endpoint. The following example configures an instance of MathReceiver to listen on the soap.tcp://localhost/math URI:

Uri receiverUri = new Uri("soap.tcp://localhost/math");
MathReceiver math = new MathReceiver();
SoapReceivers.Add(receiverUri, math);

Since you should only register the receiver once, make sure to place this code in the application initialization—SoapReceivers throws an exception if you try to register two receivers for the same URI. Once you've configured the receiver, it will be called whenever future messages arrive at the specified endpoint. Now let's take a look at how to send messages to the receiver using SoapSender.

Sending Messages

To send a SOAP message using SoapSender, first you instantiate a SoapSender object and bind it to a destination URI. You should then specify the message's action through the SoapEnvelope.Context.Action property. Finally, you call the Send method, passing in the SoapEnvelope object you want to send:

SoapEnvelope env = CreateAddMessage();
env.Context.Action = "urn:math:add";
SoapSender ssender =
   new SoapSender(new Uri("soap.tcp://localhost/math"));
ssender.Send(env);

You can change the destination URI at any point through SoapSender's Destination property:

ssender.Destination =
   new Uri("soap.tcp://localhost/mathv2");
ssender.Send(env);

The Send method delivers a one-way message and therefore doesn't return anything to the caller. However, if the sender wants a response from the receiver, it would have to implement a SoapReceiver of its own and indicate where the original receiver should send the response messages.

Figure 3 MessageOneWay
Figure 3 MessageOneWay

I've provided a sample Windows® Forms application called MessagingOneWay that you can download from the link at the top of this article. The sample illustrates the concepts I covered up to this point. I dropped the MathReceiver class into the project and configured it to listen on soap.tcp://localhost/math. When the user presses any of the buttons, a SoapEnvelope is created representing the appropriate operation and is sent to the same URI. Since the receiver is in the same process as the sender, it can get access to the textbox and update it with the result value from within the Receive method, as shown in Figure 3.

Correlating Messages

In the canonical math example, MathReceiver needs to send response messages back to the sender. This means the sender and receiver switch roles, but everything else works the same. The original sender must implement a SoapReceiver (to receive the response messages) and the original SoapReceiver must use SoapSender (to send the response messages). When the receiver processes the message and then needs to send a response, it constructs a SoapEnvelope and returns it to the original sender.

The math operations shown in the MessagingOneWay sample could generate a SOAP response message that looks like the following code snippet:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
   <s:Body>
      <v:addResponse xmlns:v='urn:math'>
         <result>99</result>
      </v:addResponse>
   </s:Body>
</s:Envelope>

The receiver can generate this SOAP message using SoapEnvelope just like the original sender did. The original sender also needs to implement a SoapReceiver class to process the response messages as they come in. Here's an example of one that writes the result element to the result textbox on the form:

public class MathResponseReceiver : SoapReceiver
{
    public Form1 SenderForm;

    protected override void Receive(SoapEnvelope e)
    {
       // write result to sender's form
       SenderForm.ResultBox.Text =
         e.SelectSingleNode("//result").InnerText
    }
}
Then, you must register the response receiver using SoapReceivers.Add and specifying a unique URI (see Figure 4).

Figure 4 Registering the Response Receiver (Close Figure 4)
// target URIs
Uri receiverUri = new Uri("soap.tcp://localhost/receiver");
Uri responseUri = new Uri("soap.tcp://localhost/response");

// initialize and register SOAP receiver objects
MathReceiver math = new MathReceiver();
SoapReceivers.Add(receiverUri, math);
MathResponseReceiver mathResponse = new MathResponseReceiver();
mathResponse.SenderForm = this;
SoapReceivers.Add(responseUri, mathResponse);

So basically I've set up the reverse scenario for communicating in the opposite direction. The only thing left is to specify in the original message where the receiver should send the response messages. This can be accomplished using the ReplyTo property made available through WS-Addresssing:

private SoapEnvelope CreateMessage(string op)
{
    // create SOAP message
    SoapEnvelope env = new SoapEnvelope();
    env.Context.ReplyTo = new ReplyTo(responseUri);

   // continue creating envelope like before
}
Now when the receiver processes the message, it can use the ReplyTo address as the destination for response messages (see Figure 5).

Figure 5 Setting Response Message Destination (Close Figure 5)
public class MathReceiver : SoapReceiver
{
    protected override void Receive(SoapEnvelope e)
    {
        ... // process incoming envelope (like before)

        // send response message to original sender
        SoapEnvelope env = new SoapEnvelope();
        env.Context.Action = e.Context.Action;
        env.CreateBody();
        env.Body.InnerXml = String.Format("<v:{0}Response
            xmlns:v='urn:math'><result>{1}</result>
            </v:{0}Response>",
            e.Body.SelectSingleNode("*").LocalName, result);

        // create SoapSender for the reply address and send message
        SoapSender responseSender = new SoapSender(
               e.Context.ReplyTo);
        responseSender.Send(env);
}

With this code in place, I've extended the previous one-way example to support the request/response MEP. I've provided an updated sample called MessagingRequestResponse in the code download that illustrates how to evolve the one-way sample to support request/response messaging.

Higher-level Messaging

Up to this point, I've been manually performing two common but somewhat tedious tasks. The first task was to determine which operation should be processed based on the incoming SOAP message. I did this in the previous examples by inspecting the message's action within the Receive method. The second task was correlating response messages with request messages. I accomplished this by inspecting the ReplyTo value in the received message and sending another message back to the original sender. When using SoapSender and SoapReceiver directly, you're always required to handle these tasks manually.

Since both of these tasks are quite common in many messaging scenarios, WSE 2.0 offers a higher-level API that builds on the SoapSender and SoapReceiver classes. This API is provided through the SoapClient and SoapService classes, which automate the two tasks that I have just described.

SoapService derives from SoapReceiver and provides the basic functionality that most receivers need. When using SoapService, you no longer need to manually implement a SoapReceiver. To implement a SoapService, you simply derive a new class from SoapService and define the operations you want to support as methods on the new class, like so:

public class MathService : SoapService
{
... // define operations here
}

Each of the operations should be defined as a method taking a SoapEnvelope as input and returning a SoapEnvelope as output, unless it doesn't need to send response messages. In that case, the return type should be void. Then, you annotate each method with the Microsoft.Web.Services.SoapMethod attribute, specifying the action that should be associated with the given method, like this:

public class MathService : SoapService
{
    [SoapMethod("urn:math:add")]
    public SoapEnvelope Add(SoapEnvelope e)
    {
       ... // implement Add here
    }
    [SoapMethod("urn:math:sub")]
    public SoapEnvelope Sub(SoapEnvelope e)
    {
       ... // implement Sub here
    }
•••
}

The SoapService implementation of Receive determines which method to call based on the incoming message's action and some .NET reflection. This makes it possible for you to focus on implementing a single operation at a time while avoiding the long switch statement I had to use earlier.

The other benefit of this approach is that the operation can simply return a SoapEnvelope object containing the response message without having to instantiate a SoapSender object and harvesting the ReplyTo address. All of this work is handled by the SoapService base class. The example in Figure 6 illustrates how you could implement the math operations using this technique.

Figure 6 Implementing MathService (Close Figure 6)
public class MathService : SoapService
{
    [SoapMethod("urn:math:add")]
    public SoapEnvelope Add(SoapEnvelope e)
    {
        double x = XmlConvert.ToDouble(
            e.SelectSingleNode("//x").InnerText);
        double y = XmlConvert.ToDouble(
            e.SelectSingleNode("//y").InnerText);
        return CreateResponseMessage(e, x+y);
    }
    [SoapMethod("urn:math:sub")]
    public SoapEnvelope Sub(SoapEnvelope e)
    {
        double x = XmlConvert.ToDouble(
            e.SelectSingleNode("//x").InnerText);
        double y = XmlConvert.ToDouble(
            e.SelectSingleNode("//y").InnerText);
        return CreateResponseMessage(e, x-y);
    }
•••
}

You still need to register your SoapService class using SoapReceivers.Add just like you would with any SoapReceiver:

MathService mathService = new MathService();
SoapReceivers.Add(
   new Uri("soap.tcp://localhost/soapservice",
   mathService);
To take advantage of this same approach on the client, you'll need to implement another class that derives from SoapClient. This class will encapsulate the process of using SoapSender to send messages to a particular endpoint. Hence, using SoapClient, you no longer need to use SoapSender directly. You basically follow the same procedure as with SoapService; the only difference is how you implement the operations.

As with SoapService, you define methods for the operations you intend to invoke and then annotate them with SoapMethodAttribute, specifying the operation's action, as shown in Figure 7.

Figure 7 Defining Methods with MathClient (Close Figure 7)
public class MathClient : SoapClient
{
    public MathClient() {}
    public MathClient(Uri uri) : base(uri) {}

    [SoapMethod("urn:math:add")]
    public SoapEnvelope Add(SoapEnvelope e)
    {
        return base.SendRequestResponse("Add", e);
    }
    [SoapMethod("urn:math:sub")]
    public SoapEnvelope Sub(SoapEnvelope e)
    {
        return base.SendRequestResponse("Sub", e);
    }
    •••
}

The SoapClient base class provides various methods for invoking the operation, including Send (inherited from SoapSender), SendOneWay, and SendRequestResponse. SendRequestResponse encapsulates the process of sending a message with SoapSender and setting up a receiver to handle the response message. When you call SendRequestResponse, you must specify the name of the method you want to invoke along with the SOAP envelope.

My custom MathClient class makes it much easier for clients to interact with MathService since they no longer have to worry about specifying the action or correlating request and response messages manually. The code in Figure 8 illustrates how to use MathClient to send a message and process the response.

Figure 8 Sending a Message with MathClient (Close Figure 8)
private void CreateMessageAndSend(string op)
{
    // create & send message
    SoapEnvelope reqEnv = CreateMessage(op);
    MathClient mathClient = new MathClient(receiverUri);
    SoapEnvelope respEnv;
    switch(op)
    {
      case "add":
        respEnv = mathClient.Add(reqEnv);
        break;
      case "sub":
        respEnv = mathClient.Sub(reqEnv);
        break;
      case "mul":
        respEnv = mathClient.Mul(reqEnv);
        break;
      case "div":
        respEnv = mathClient.Div(reqEnv);
        break;
      default:
        throw new Exception("unsupported operation");
    }
    this.textBox3.Text = respEnv.SelectSingleNode("//result").InnerText;
}

These classes simplify the tasks that are involved in setting up different messaging scenarios and they have optimized for request/response messaging. I've provided another sample in the code download called MessagingHigherLevel that provides the same functionality as the previous sample, but implements it through SoapService and SoapClient classes.

Using TCP and HTTP

Up to this point, both the sender code and the receiver code have been running in the same process. As I mentioned, the WSE 2.0 messaging API also lets you use TCP or HTTP as the communications mechanism. Using one of these protocols makes it possible to isolate the sender code from the receiver code in separate processes or even separate machines. To illustrate this, I created a new project called MessagingTcpService and only included the MathService code shown earlier. This project contains a simple form displaying the URI it's listening on (see Figure 9).

Figure 9 MessagingTcpService
Figure 9 MessagingTcpService

I updated the initialization code to register a soap.tcp URI instead of soap.tcp, as illustrated here:

receiverUri = new Uri(String.Format(
        "soap.tcp://{0}:123/receiver",
        System.Net.Dns.GetHostName()));

// initialize and register SOAP receiver objects
  MathService mathService = new MathService();
  SoapReceivers.Add(receiverUri, mathService);
In addition, I updated the previous sample and called it MathTcpClient. I removed the MathService code, since that's now in the MathTcpService project, and updated the URI to be the same one that's used by the service. These are the only changes I had to make to enable SOAP over TCP. Now MathTcpClient can communicate with MathTcpService over TCP even if the processes are running across the Internet.

Figure 10 New Unformatted Trace
Figure 10 New Unformatted Trace

It's an interesting exercise to trace the TCP messages between the two applications using the Trace Utility that ships with Microsoft SOAP Toolkit 3.0. To do this, you need to update your MathService code to listen on a different port (say 456). Then create a new unformatted trace and specify 123 as the port to listen on and 456 as the destination port, as shown in Figure 10. Now when you run the client program and invoke one of the operations, you should see the captured TCP messages containing the SOAP request and response messages (see Figure 11).

Figure 11 Tracing SOAP over TCP
Figure 11 Tracing SOAP over TCP

In addition to the in-process and TCP communications mechanisms, it's also possible to integrate SoapReceivers with HTTP pipeline in ASP.NET. If you look at the definition for SoapReceiver, you'll notice that it implements IHttpHandler:

public abstract class SoapReceiver : SoapPort, IHttpHandler
{
   •••
}

Thanks to this, any SoapReceiver or SendService class can now be configured in ASP.NET as an HTTP handler. You can configure HTTP handlers by adding a new mapping in the httpHandlers section of your web.config file. The web.config entry should map a verb/path combination to your SoapReceiver type:

<configuration>
    <system.web>
        <httpHandlers>
            <add verb="*" path="receiver.ashx"
                  type="MathService, Messaging"/>
        </httpHandlers>
    </system.web>
</configuration>
With this in place, MathService will be called for every message that enters this virtual directory and targets receiver.ashx. Now you don't have to worry about configuring your SoapReceiver/SoapService by calling SoapReceivers.Add since ASP.NET is essentially doing that on your behalf.

If you update the client to send messages to the HTTP endpoint (http://localhost/messaginghttpservice/receiver.ashx), it will work just like it did when using TCP, only now it's communicating over HTTP. I've provided sample projects in the download that illustrate the TCP and HTTP examples described here.

WseChat

As you can see, the SoapSender and SoapReceiver classes in WSE 2.0 make it possible to build SOAP messaging applications that can run over different transport protocols. SoapClient and SoapService build on this foundation to simplify the common tasks. To illustrate the flexibility that comes from these core classes, I've provided a more interesting sample called WseChat.

Figure 12 Bob and Mary Chat
Figure 12 Bob and Mary Chat

WseChat is a SOAP-based chat program built completely in terms of SoapSender and SoapReceiver running over TCP. The chat program allows the user to specify a user name and port number. When the user clicks the Listen button, it constructs a URI based on those values and the program begins listening over TCP. The user can type in a target URI indicating where to send the chat messages. Then, when the user types a message and presses Send (at the bottom of the form), the application sends the message over TCP to the target URI. You can run the program on separate machines and chat between them. Figure 12 shows the program in action.

Conclusion

WSE 2.0 provides a flexible API for sending and receiving SOAP messages via TCP or HTTP. SoapSender and SoapReceiver provide the basic functionality for sending and receiving SOAP messages. WSE 2.0 also provides a higher-level API through the SoapClient and SoapService classes which help you deal with multiple operations at a given endpoint and correlate the request and response messages. Overall, the use of WSE 2.0 helps developers take a large step towards realizing the full potential of XML-based Web Services.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard is an instructor/researcher at DevelopMentor, where he develops the XML and Web Service-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.


Advanced Basics: Creating Text Images On the Fly with GDI+

Advanced Basics
Creating Text Images On the Fly with GDI+
Ken Spencer

Code download available at: AdvancedBasics0309.exe (193KB)

Q I am building a Windows® Forms application and need to be able to create graphical text on the fly. Basically, I want to create an image, add text to it, then use the new image later in the app.

A The first step in solving this problem is pretty easy. I created a simple form that allows you to enter a string of text, then enter the starting point for the text in the resulting output. After you enter the string, the code will create an image from it that is then placed in one or both of the textboxes on the form (see Figure 1).

Figure 1 Creating an Image in a Textbox
Figure 1 Creating an Image in a Textbox

The application creates a bitmap image, then draws the string on the bitmap. The bitmap can then be displayed in a PictureBox. Let's take a look at the code, then explore GDI+a bit more.

The first step in creating this form was to define a set of variables. The following lines are from frmGraphics1.vb. The drawBrush and fillBrush variables are used to draw the string and to fill the rectangle that sits behind the string. The drawFont variable is a reference to the font I am using, MySize is used with the MeasureString method, and the last variable, ImageFortext, is the image used for the text:

Dim drawBrush As New SolidBrush(Color.Black)
Dim fillBrush As New SolidBrush(Color.White)
Dim drawFont As New Font("Arial", 16)
Dim iX, iY As Integer
Dim MySize As SizeF
Private ImageFortext As Bitmap

Now let's walk through the code in cmdCreateText_Click. This code does not display the image; it simply creates it from text. The first line creates a reference to the Graphics class:

Dim MyGraphics As Graphics
Next, a bitmap image is created:
ImageFortext = New Bitmap(773, 1060, PixelFormat.Format32bppArgb)
This image is created at an arbitrary size of 773 pixels wide and 1060 pixels high. Ideally you will want to calculate the image size based on the actual text, the font being used, and some margin space. You can vary the size by changing the first two variables to the Bitmap constructor. The last parameter to the constructor specifies the pixel format used by the image, in this case 32 bits per pixel.

Next, MyGraphics is set to a new graphics object that is created from the image that I just made:

MyGraphics = MyGraphics.FromImage(ImageFortext)
Then the MeasureString method is called to determine the physical size of the text the user entered:
MySize = MyGraphics.MeasureString(txtGraphicText.Text, drawFont)
Now I can fill a rectangle with an image that is the same size as the text, as shown here:
MyGraphics.FillRectangle(fillBrush, _
    New Rectangle(iX, iY, MySize.Width, MySize.Height))

The rectangle is created using the white fillBrush. You can do all sorts of things with the FillRectangle method. Since you can pass this method a brush of your choosing, you can create solid or hatched fills and more, drawing almost any type of background for your text.

Finally, I can place the string on the image by calling DrawString:

MyGraphics.DrawString(txtGraphicText.Text,drawFont,drawBrush,iX,iY)
In this case, I'm drawing a simple horizontal string on top of the background. Of course, I could draw the string vertically as well.

Now that the image is all set, I can call DrawImage to generate it:

MyGraphics.DrawImage(ImageFortext, New Point(0, 0))
Next, I dispose of the Graphics object to free up resources, then enable the buttons to show the text:
MyGraphics.Dispose()
cmdShowText.Enabled = True
cmdShowText2.Enabled = True

Now, let's look at how to display the new text image. My first experiments led me to try creating a new Graphics object and set it to the image. Then I stumbled on the simple but elegant solution shown here. All you need to do is set the Image property of the PictureBox to the new image (ImageFortext) and the new text shows up in the image. The surrounding code simply handles the suspension and resumption of the layout and sets the size of the control equal to the part of the image you want to show:

With picMyText
.SuspendLayout()
.Image = ImageFortext
.Size = New Size(MySize.Width + iX, MySize.Height + iY)
.ResumeLayout()
End With

Now you can reuse the image at any place in this application by using the ImageForText image. The Image class also contains a Save method, which allows you to save your image to a file or almost anywhere else (such as the Response object in ASP.NET or a stream).

Now let's look at a more practical way of using the functionality I've shown you so far. In this case, I want to pull a list from a database and display the retrieved text as an image with a checkbox beside it. The code I have already created is remarkably close to solving this problem.

First, I created a new user control named ctrlCheckImages. The control has the following private variables:

Public pvtChecked As String
Public pvtName As String
Dim drawBrush As New SolidBrush(Color.Black)
Dim fillBrush As New SolidBrush(Color.White)
Dim drawFont As New Font("Arial", 16)
Dim MySize As SizeF
Dim pvtRectangleWidth As Integer
In addition, it has three public variables: TextWidth, Checked, and ItemName. The first two properties are standard implementations that set or retrieve values from private variables. The ItemName property is slightly different as it also calls the CreateImage procedure when the value is set:
Public Property ItemName() As String
        Get
            Return pvtName
        End Get
        Set(ByVal Value As String)
            pvtName = Value
            CreateImage(pvtName)
        End Set
    End Property

I also added a checkbox to the user control, placing it in the far left position. I set the size to 16, 24, and the location to 0, 0. The Text was set to an empty string. This checkbox has one event (chkIndicator_CheckedChanged), which sets the pvtChecked variable to the state of the control:

pvtChecked = chkIndicator.Checked

Next, I implemented a procedure to create the text (CreateImage). This procedure takes one argument, of type PaintEventArgs, because it must be called in the Paint event. This code is similar to the CreateImage code, with a few variations. First, the Graphics object is now the control's surface instead of an image. Why? In this case, I am simply outputting an image to the control, so there is no need to incur the overhead of creating an image in memory first when I can just manipulate the control directly.

The MyGraphics variable is set to reference the control:

Dim MyGraphics As Graphics
Dim TextString As String
MyGraphics = e.Graphics
This code also picks up the text string from the private variable instead of pulling it from the textbox:
TextString = pvtName
Then the code in Figure 2 paints the control. Now, I can draw the string on the control:
MyGraphics.DrawString(TextString, drawFont, drawBrush, 26, 0)
Next, I reset the control to approximately the same size as the text string, as shown in the following line:
Me.Size = New Size(pvtRectangleWidth + 26, MySize.Height)
Finally, I dispose of the Graphics object to free up resources:
MyGraphics.Dispose()
Now let's take a look at the code in frmGraphicsStringFromDB.vb that uses this control. The ShowData procedure creates several variables, including the GraphicsChecker variable which is a reference to the new control:
Dim yloc, yincr As Integer
Dim rw As DataRow
Dim GraphicsChecker As ctrlCheckImages
Dim i As Integer

Figure 2 Painting the Control (Close Figure 2)
MySize = MyGraphics.MeasureString(TextString, drawFont)

If pvtRectangleWidth > 0 Then
    MyGraphics.FillRectangle(fillBrush, _
        New Rectangle(0, 0, _
        pvtRectangleWidth, MySize.Height))
Else
    pvtRectangleWidth = MySize.Width
        MyGraphics.FillRectangle(fillBrush, _
        New Rectangle(0, 0, _
        MySize.Width, MySize.Height))
End If

Next, I set several variables:

yloc = 50
yincr = 26.2
i = 0

As the code iterates through the datatable, it creates and initializes a new GraphicsChecker for each record. The new control is then added to the Controls collection of the form. It doesn't have to work with the image directly in this example because the image handling is done by the control. The rest of the code is maintenance or cleanup code, as shown in Figure 3.

Figure 3 Maintenance (Close Figure 3)
For Each rw In dsCustomer.Tables(0).Rows
    GraphicsChecker = New ctrlCheckImages
    With GraphicsChecker
        .TextWidth = 380
        .Checked = False
        .ItemName = rw("CompanyName").ToString
        .Location = New Point(20, yloc)
    End With

    Me.Controls.Add(GraphicsChecker)

    yloc += yincr

    GraphicsChecker = Nothing
    If i = NoOfItemsPerForm Then
        Exit For
    End If
Next

GDI+ is a great addition to your toolkit for building applications. You can add graphics to your application with a few lines of code and present a rich interface to your users. But before you decide to use images everywhere, just remember that they come at a price. Every time you use GDI+ to output a different graphic there is overhead involved. Consider the example in which I used the control with GDI+ to output the checkbox and string. In the code download, there is another file that implements the sample control (frmGraphicsStringFromDB2). This form has a control that does not use DrawString, instead employing a label control for the text. It still uses FillRectangle to define the text area but because it isn't calling an image, the difference in performance is dramatic, as you will see when you run it.

No matter how cool a UI element idea may seem, never forget to consider performance. I remember a project in which the application would write out a simple line of text as it processed items. Performance improved dramatically when I stopped writing to the console. All output is expensive, so keep that in mind.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencerworks for 32X Tech (http://www.32X.com), where he provides training, software development, and consulting services on Microsoft technologies.


The ASP Column: The Internet Explorer Toolbar Control

The ASP Column
The Internet Explorer Toolbar Control
George Shepherd

Code download available at: ASPColumn0309.exe (139KB)
Contents
Toolbar Example
Toolbars and the Designer
Toolbar Events
Examining Toolbar State
Using Toolbars Programmatically
Customizing the Toolbar
The Rendered Code
Conclusion

Over several issues of MSDN® Magazine I've been taking a look at the Microsoft® Internet Explorer Web controls. In the June 2003 issue I looked at the TreeView control, which supports hierarchical representations of data. This time I'll take a look at the last of the Web controls—the Toolbar. Toolbars have been around for some time, providing shortcuts for accessing program features. The Internet Explorer Toolbar control lets you put a toolbar on your ASP.NET Web page, and performs the jobs of rendering the correct tags and managing the toolbar state.

Just as with other server-side controls, toolbars are represented by tags within an ASP.NET page and derive from the class named Microsoft.Web.UI.WebControls.Toolbar. The Internet Explorer Web Toolbar control enables you to provide many of the Windows® toolbar features within a Web page. (If you're looking for these Web controls, full source code is available at http://www.asp.net/Default.aspx?tabindex=7&tabid=41.)

Toolbar Example

To illustrate how the Internet Explorer Toolbar works, this month's example displays a string in a variety of fonts and settings. For example, text may appear in one of several colors, styles (bold, italic, or underlined), and sizes (large, medium, or small). Each of these features is selectable from the toolbar using various button, dropdown list, and text controls within the toolbar itself. Figure 1 shows the sample program running.

Figure 1 Changing Font Styles Through the Internet Explorer Toolbar (Close Figure 1)

The Toolbar in Figure 1 shows all the toolbar items an Internet Explorer Toolbar can provide. On the left of the toolbar is an edit box for entering text. The three buttons immediately to the right of the textbox work like radio buttons, allowing you to select one of several exclusive options (that is, the large, medium, or small font). To the right of these buttons are a group of three checkbox-style buttons for selecting up to three inclusive-or operations (that is, the bold, italic, and underline font styles). To the right of the checkbox-style buttons is a dropdown listbox. Finally, there is a regular button for advancing to the next page.

The Internet Explorer server-side toolbar is comprised of a number of toolbar items, each one performing a specific function. The toolbar items are similar to their standalone counterparts (for example, the Button class, the Label class, the Textbox class, and the DropDownList class). Figure 2 lists the toolbar items and the properties they expose.

Figure 2 Toolbar Items and Their Properties (Close Figure 2)

Toolbar Item | Description | Properties
ToolbarButton | Regular button on a Toolbar | ID, AccessKey, DefaultStyle, Enabled, HoverStyle, ImageURL, SelectedStyle, TabIndex, Text, ToolTip
ToolbarCheckButton | Standard toggle (inclusive OR) | ID, AccessKey, AutoPostBack, DefaultStyle, Enabled, HoverStyle, ImageURL, Selected, SelectedStyle, TabIndex, Text, ToolTip
ToolbarCheckGroup | Radio button style (exclusive OR) | ID, AccessKey, AutoPostBack, DefaultStyle, Enabled, ForceSelection, HoverStyle, Items, SelectedStyle, TabIndex, ToolTip
ToolbarLabel | Regular label | ID, AccessKey, DefaultStyle, Enabled, ImageURL, TabIndex, Text, ToolTip
ToolbarSeparator | Separates items | ID, AccessKey, DefaultStyle, Enabled, TabIndex, Text, ToolTip
ToolbarTextBox | Textbox on a Toolbar | ID, AccessKey, AutoPostBack, BackColor, BorderColor, BorderStyle, BorderWidth, Columns, CssClass, DefaultStyle, Enabled, Font, ForeColor, Height, HoverStyle, MaxLength, ReadOnly, TabIndex, Text, ToolTip, Width
ToolbarDropDownList | Combobox on a Toolbar | ID, AccessKey, AutoPostBack, BackColor, CssClass, DataMember, DataSource, DataTextField, DataTextFormatString, DataValueField, DefaultStyle, Enabled, Font, ForeColor, Items, TabIndex, Width

Toolbars and the Designer

The easiest and most straightforward way to use the Toolbar control is through the ASP.NET forms designer in Visual Studio® .NET. The Internet Explorer Toolbar includes explicit design support. To use a Toolbar on a Web Form, drag it onto the form as you would any other control. It is initially rendered as a gray box (with some instructions about adding items using the property pane). Figure 3 shows a fresh Toolbar control on a Web Form.

Figure 3 Toolbar Control Placed on a Web Form
Figure 3 Toolbar Control Placed on a Web Form

The Toolbar object itself includes a number of properties, such as background color, border styles, and fonts. Modifying the appearance of the toolbar is as simple as modifying each of these properties. As most Web controls do, the Toolbar control exposes an AutoPostback property. When AutoPostback is true, every item causes a postback to the server. When it's set to false, each item may be tailored to post back (or not).

Once the Toolbar is in place on the form, you may use the Items property to add toolbar items to it. The dialog is pretty straightforward, and there's nothing unusual you need to watch out for.

Toolbar Events

The Toolbar itself exposes two main events that may be wired up through the designer. The Internet Explorer Toolbar is composed of buttons and other controls that normally have events wired up to them. The constituent controls cannot be connected through the designer, but they can be wired up manually. I'll describe both techniques.

The two events exposed by the Toolbar control (above and beyond the normal complement of Disposed, Load, Init, Prerender, Unload, and DataBinding) are ButtonClick and CheckChange. ButtonClick occurs whenever one of the buttons on the toolbar has been pressed. CheckChange occurs whenever one of the checkbox-style buttons is changed.

You can have the designer write the event-handling code for you by highlighting the toolbar on the form and choosing the event button on the Properties dialog (the event button is represented by the little lightning bolt) and double-clicking in the ButtonClick event box. The designer will stub out a button click event handler in your codebehind page. You may then interrogate the object (the first argument) about the event that just occurred, as shown in the following code:

private void OptionsToolbar_ButtonClick(object sender,
    System.EventArgs e)
{
   // one way to respond to a button press:
   ToolbarButton tbbtn;
   tbbtn = ((ToolbarButton)sender);
   if(tbbtn.ID == "GoToMainPage")
   {
      Server.Transfer("mainpage.aspx");
   }
}

Alternatively, you may respond to button click events by writing an event handler and manually attaching it to the button when the page is initialized, as shown in Figure 4. The code in Figure 4 shows the font page's OnInit method, which hooks up the MainPageClick event handler to the 11th toolbar item (which happens to the main page button). Notice the signature of the handler—it's defined by the Microsoft.Web.UI.WebControls.ToolbarItemEventHandler delegate from the Internet Explorer Web Controls assembly. It returns a Boolean (rather than the void type that most event handlers return). You should return true if you want the event to bubble and false if you want the event handling to stop here. Responding to the button clicks directly results in tighter code, but then you have to write the handlers yourself.

Figure 4 Responding to a Button Directly (Close Figure 4)
// OnInit called when page is created
override protected void OnInit(EventArgs e)
{
   //
   // CODEGEN: This call is required by the ASP.NET Web Form Designer.
   //
   InitializeComponent();
   ToolbarButton MainPageButton;
   MainPageButton =
    (ToolbarButton)this.OptionsToolbar.Items[11];

   MainPageButton.ButtonClick += new
    ToolbarItemEventHandler(OptionsToolbar_MainPageClick);
   base.OnInit(e);
}

private bool OptionsToolbar_MainPageClick(object sender,
   System.EventArgs e)
{
   Server.Transfer("mainpage.aspx");
   return true;
}

Examining Toolbar State

The Toolbar control in this example has its AutoPostback property set to true. Whenever a button is pressed or a control is changed, the Toolbar causes a postback. Once inside the postback, you can easily examine the state of the checkbox and radio buttons. The code download for this column contains the font page's rendering code, which checks each of the font style buttons to render the correct tags. In it, each control is extracted from the Items collection using the numerical index of the control. This is straightforward enough. However, it means that you need to update this code if you decide to insert a control into the toolbar and change the position of the existing subcontrols.

Before I discuss how to create toolbars programmatically, let me say a couple of things about the Render method. Many regular ASP.NET pages are displayed by walking the list of server-side controls on the page and asking each one to render itself (by calling Control.Render). Sometimes pages render by calling Response.Write directly as well. Response.Write places text directly into the output stream of the page, and the font page in the example could easily have been rendered using this method, although that is not recommended for controls.

However, because System.Web.UI.Page derives from System.Web.UI.Control, you can use the HtmlTextWriter class to manage the HTML tags by overriding Render, which you'll see when you look at the code download. The HtmlTextWriter parameter that is passed to the Render method will manage the HTML tags correctly, regardless of the browser that made the request. While it doesn't make any difference in this application (which only manages font styles), using the HtmlTextWriter is very useful when rendering tables and other HTML elements whose syntax varies between HTML versions.

Using Toolbars Programmatically

In addition to developing toolbars via the designer, you may also develop them programmatically. Just imagine a situation in which a toolbar might need to be rendered differently depending on the identity or authorization of the user. When these conditions are determined at run time, it's definitely appropriate to manipulate the toolbar programmatically.

To illustrate runtime toolbar manipulation, my sample signs in users with form-based authentication and then assigns them to an administrator and/or a user role. (Look at the files Web.Config, Login.ASPX, and Global.ASAX in the code download to see how the credentials and roles work.) Once a user is signed in, the example shows the main page of the application with the appropriate toolbars. Figure 5 shows the main page of the application with both toolbars showing (presumably, a user with both a user and administrator role is logged on).

Figure 5 Main Page of App with Two Toolbars
Figure 5 Main Page of App with Two Toolbars

The main page starts with toolbars (the AdministratorsToolbar and the UsersToolbar) that were added via the designer. The visibility for each of these toolbars is set to false, which means that they won't render. The toolbars are assembled programmatically (based upon the user and his or her role). Figure 6 shows the code for setting up the UsersToolbar, as well as for responding to the button clicks on the toolbar.

Figure 6 Assembling the Toolbars Programmatically (Close Figure 6)
protected void SetupUsers()
{
  string cssString =
   @"Z-INDEX: 101; LEFT: 170px; POSITION:
    absolute; TOP: 200px";
  UsersToolbar.CssClass = cssString;

  this.UsersToolbar.AutoPostBack = true;
  // Users can get news, do email, messaging, fonts
  ToolbarLabel tblbl = new ToolbarLabel();
  tblbl.Text = "Stuff Users can do:";
  CssCollection cssc =
    new CssCollection(@"font-family:Arial;
      font-weight:bold;background:white;border:
      solid 1px;color:black");
  tblbl.DefaultStyle = cssc;
  UsersToolbar.Items.Add(tblbl);

  ToolbarButton tbbtnNews = new ToolbarButton();
  tbbtnNews.Text = "News";
  tbbtnNews.ID = "News";
  UsersToolbar.Items.Add(tbbtnNews);

  ToolbarButton tbbtnEmail = new ToolbarButton();
  tbbtnEmail.Text = "Email";
  tbbtnEmail.ID = "Email";
  UsersToolbar.Items.Add(tbbtnEmail);

  ToolbarButton tbbtnMsg = new ToolbarButton();
  tbbtnMsg.Text = "Messaging";
  tbbtnMsg.ID = "Messaging";
  UsersToolbar.Items.Add(tbbtnMsg);

  ToolbarButton tbbtnFonts = new ToolbarButton();
  tbbtnFonts.ID = "Fonts";
  tbbtnFonts.Text = "Fonts";
  UsersToolbar.Items.Add(tbbtnFonts);

  UsersToolbar.Visible = true;
}
// Similar function for the Administrators Toolbar

private void Page_Load(object sender, System.EventArgs e)
{
   // Set up all the toolbar items during the first HTTP get, but
   // not during subsequent posts.
   if(!this.IsPostBack)
   {
      if(User.IsInRole("Administrators"))
      {
         SetupAdministrators();
      }
      if(User.IsInRole("Users"))
      {
         SetupUsers();
      }
   }
}

private void UsersToolbar_ButtonClick(object sender,
    System.EventArgs e)
{
  if(sender is ToolbarButton)
  {
    if(((ToolbarButton)sender).ID == "Fonts")
    {
      Server.Transfer("fontplay.aspx");
    } else if(((ToolbarButton)sender).ID == "Email")
    {
      this.LabelWhatsHappening.Text = "Doing email...";
    } else if(((ToolbarButton)sender).ID == "Messaging")
    {
      this.LabelWhatsHappening.Text = "Doing messaging...";
    } else if(((ToolbarButton)sender).ID == "News")
    {
      this.LabelWhatsHappening.Text = "Doing news...";
    }
  }
}

This code also determines the role of the user and sets up each Toolbar appropriately. It creates and adds each of the buttons to the UsersToolbar. Also, notice that the visible flag is flipped to true for the appropriate Toolbars. The code responds to the Toolbar's ButtonClick event by interrogating the sender of the event in order to find out the cause of the postback.

Customizing the Toolbar

The Toolbar class and each of the items includes a set of Cascading Style Sheet collections for customizing them. If you take a look at the code in Figure 6, you'll notice that it applies the CssCollection class to customize the look of the Toolbar in the SetUpUsers method. The Toolbar and its constituent elements have a default style, a hover style, and a selected style.

The Rendered Code

One of the most interesting aspects of the Toolbar is the HTML it renders. In a downlevel browser, the toolbar is represented using table tags and toolbar buttons are implemented using HREF tags. The dropdown listbox is represented using selection tags within the table. When the client is an uplevel browser, the toolbar is implemented within a file named "/webctrl_client/1_0/toolbar.htc". The file toolbar.htc includes script to enable features such as tooltips and hovering.

Conclusion

This month wraps up my coverage of the Internet Explorer Web controls. The Toolbar control is useful for consolidating related commands (as with the font Toolbar) and it includes good designer support (except for individual control events). And remember, you can also manipulate the Toolbar programmatically when you need that flexibility.

Send your questions and comments for George to asp-net@microsoft.com.

George Shepherd delivers seminars with DevelopMentor and is an architect of Syncfusion's .NET Windows Forms tools. He is the author of a number of programming books, including Programming with Microsoft Visual C++ .NET (Microsoft Press, 2002), and Applied .NET (Addison-Wesley, 2001). George may be reached at georges@syncfusion.com or georges@develop.com.


.NET Column: Introducing Generics in the CLR

[ Editor's Update - 1/20/2006: This article refers to a beta version of Visual Studio 2005. An updated version of the article, reflecting features found in the final release of Visual Studio 2005, can be found at .NET: Introducing Generics in the CLR.]
.NET
Introducing Generics in the CLR
Jason Clark

Contents
A First Look at Generics
A Simple Example
C# Generic Syntax
Indirection
Conclusion

In the last installment of this column, I covered Interop with unmanaged code via P/Invoke. In some ways this topic revisited the past by showing how your managed code can access legacy Win32® code. In contrast, this month I'm going to peek into the future by looking at a cool new feature, generics, which will be coming soon to the common language runtime (CLR). I'll introduce generics and discuss the benefits that it brings to your code, and in a future column I'll dig into more details of how the compiler and runtime implement generics.

Generics are an extension to the CLR's type system that allow developers to define types for which certain details are left unspecified. Instead, these details are specified when the code is referenced by consumer code. The code that references the generic type fills in the missing details, tailoring the type to its particular needs. The name generics reflects the goal of the feature: to enable the writing of code while not specifying details that might limit the scope of its usefulness. The code itself is generic. I'll get more specific in just a moment.

Just how far in the future are generics? Microsoft plans to ship generics in the release of the CLR code-named "Whidbey," and there should be a beta release of the Whidbey CLR sometime after this column goes to press. Meanwhile, in the same release of the CLR, there is a planned update to the language and compiler that takes full advantage of generics. Finally, the research group at Microsoft has modified a version of the shared source common language implementation (CLI), code-named "Rotor," to include generics support. This modified runtime, code-named "Gyro," is available at http://research.microsoft.com/projects/clrgen.

A First Look at Generics

As with any new technology, it is helpful to ask just why it's useful. Those of you who are familiar with templates in C++ will find that generics serve a similar purpose in managed code. However, I hesitate to draw too much of a comparison between CLR generics and C++ templates because generics have some additional benefits, including the absence of two common problems: code bloat and developer confusion.

Some of the strengths of CLR generics are compile-time type safety, binary code reuse, performance, and clarity. I'll briefly describe these benefits, and as you read the rest of this column, you'll understand them in more detail. As an example let's take two hypothetical collection classes: SortedList, a collection of Object references, and GenericSortedList<T>, a collection of any type.

Type safety When a user adds a String to a collection of type SortedList, there is an implicit cast to Object. Similarly, if a String object is retrieved from the list, it must be cast at run time from an Object reference to a String reference. This lack of type safety at compile time is both tedious for the developer and prone to error. In contrast, a use of GenericSortedList<String> where T is typed to String causes all add and lookup methods to work with String references. This allows for specifying and checking the type of the elements at compile time rather than at run time.

Binary code reuse For maintenance purposes, a developer may elect to achieve compile-time type safety using SortedList by deriving a SortedListOfStrings from it. The problem with this approach is that new code has to be written for every type for which you want a type-safe list, which can quickly become laborious. With GenericSortedList<T>, all you need to do is instantiate the type with the desired element type as T. As an added value, generics code is generated at run time, so two expansions over unrelated element types such as GenericSortedList<String> and GenericSortedList<FileStream> are able to reuse most of the same just-in-time (JIT)-compiled code. The CLR simply works out the details—no more code bloat!

Performance The bottom line is this: if type checking is done at compile time rather than at run time, performance improves. In managed code, casts between references and values incur both boxings and unboxings, and avoiding such casts can have an equally negative impact on performance. Current benchmarks of a quick-sort of an array of one million integers shows the generic method is three times faster than the non-generic equivalent. This is because boxing of the values is avoided completely. The same sort over an array of string references resulted in a 20 percent improvement in performance with the generic method due to the absence of a need to perform type checking at run time.

Clarity Clarity with generics comes in a number of forms. Constraints are a feature in generics that have the effect of making incompatible expansions of generic code impossible; with generics, you're never faced with the cryptic compiler errors that plague C++ template users. In the GenericSortedList<T> example, the collection class would have a constraint that makes it only work with types for T that can be compared and therefore sorted. Also, generic methods can often be called without employing any special syntax by using a feature called type inference. And, of course, type safety at compile time increases clarity in application code. I will look at constraints, type inference, and type safety in detail throughout this column.

A Simple Example

The Whidbey CLR release will provide these benefits out-of-the-box with a suite of generic collection classes in the class library. But applications can further benefit from generics by defining their own generic code. To examine how this is done, I'll start by modifying a simple linked-list node class to make it a generic class type.

The Node class in Figure 1 includes little more than the basics. It has a field, m_data, which refers to the data for the node and a second field, m_next, which refers to the next item in the linked list. Both of the fields are set by the constructor method. There are really only two additional bells and whistles, the first of which is that the m_data and m_next fields are accessed using read-only properties named Data and Next. The second is the override of System.Object's ToString virtual method.

Figure 1 Simple Linked List (Close Figure 1)
using System;

// Definition of a node type for creating a linked list
class Node {
   Object  m_data;
   Node    m_next;

   public Node(Object data, Node next) {
      m_data = data;
      m_next = next;
   }

   // Access the data for the node
   public Object Data {
      get { return m_data; }
   }

   // Access the next node
   public Node Next {
      get { return m_next; }
   }

   // Get a string representation of the node
   public override String ToString() {
      return m_data.ToString();
   }
}

// Code that uses the node type
class App {
   public static void Main() {

      // Create a linked list of integers
      Node head = new Node(5, null);
      head = new Node(10, head);
      head = new Node(15, head);

      // Sum-up integers by traversing linked list
      Int32 sum = 0;
      for (Node current = head; current != null;
         current = current.Next) {
         sum += (Int32) current.Data;
      }

      // Output sum
      Console.WriteLine("Sum of nodes = {0}", sum);
   }
}

Figure 1 also shows code that uses the Node class. This referencing code suffers from some limitations. The problem is that in order to be usable in many contexts, its data must be of the most basic type, System.Object. This means that when you use Node, you lose any form of compile-time type safety. Using Object to mean "any type" in an algorithm or data structure forces the consuming code to cast between Object references and the actual data type. Any type-mismatch bugs in the application are not caught until run time, when they take the form of an InvalidCastException at the point where the cast is attempted.

Additionally, any assignment of a primitive value such as an Int32 to an Object reference requires a boxing of the instance. Boxing involves a memory allocation and a memory copy, as well as an eventual garbage collection of the boxed value. Finally, casts from Object references to value types such as Int32, as you can see in Figure 1, incur unboxing which also includes a type check. Since boxing and unboxing hurt the overall performance of the algorithm, you can see why using Object to mean "any type" has its disadvantages.

A rewrite of Node using generics is an elegant way to address these problems. Take a look at the code in Figure 2 and you will see the Node type rewritten as the Node<T> type. A type with generic behaviors such as Node<T> is parameterized and can be called Parameterized Node, Node of T, or Generic Node. I will address the new C# syntax in a moment; first let's delve into how Node<T> differs from Node.

Figure 2 Defining Code: Generic Linked-list Node (Close Figure 2)
class Node<T> {
   T        m_data;
   Node<T>  m_next;

   public Node(T data, Node<T> next) {
      m_data = data;
      m_next = next;
   }

   // Access the data for the node
   public T Data {
      get { return m_data; }
      set { m_data = value; }
   }

   // Access the next node
   public Node<T> Next {
      get { return m_next; }
      set { m_next = value; }
   }

   // Get a string representation of the node
   public override String ToString() {
      return m_data.ToString();
   }
}

The Node<T> type is functionally and structurally similar to the Node type. Both support the building of linked lists of any given type of data. However, where Node uses System.Object to represent "any type," Node<T> leaves the type unspecified. Instead, Node<T> uses a type parameter named T that is a placeholder for a type. The type parameter is eventually specified by an argument to Node<T> when consuming code makes use of Node<T>.

The code in Figure 3 makes use of Node<T> with 32-bit signed integers by constructing a type name like so: Node<Int32>. In this example Int32 is the type argument to the type parameter T. (Incidentally, C# would also accept Node<int> to indicate T as Int32.) If the code required a linked list of some other type, such as String references, it could do this by specifying it as the type argument to T, like this: Node<String>.

Figure 3 Referencing Code: Generic Linked-list Node (Close Figure 3)
class App {
   public static void Main() {

      // Create a linked list of integers
      Node<Int32> head = new Node<Int32>(5, null);
      head = new Node<Int32>(10, head);
      head = new Node<Int32>(15, head);

      // Sum up integers by traversing linked list
      Int32 sum = 0;
      for (Node<Int32> current = head; current != null;
         current = current.Next) {
         sum += current.Data;
      }

      // Output sum
      Console.WriteLine("Sum of nodes = {0}", sum.ToString());
   }
}

The power behind Node<T> is that its algorithmic behaviors are well defined while the type of data that it operates on remains unspecified. In this way, the Node<T> type is specific in terms of how it works but generic in terms of what it works with. Details such as what type of data a linked list should hold is really best left specified by the code that uses Node<T> anyway.

When discussing generics it can be helpful to be explicit about two roles: defining code and referencing code. Defining code includes code that both declares the existence of the generic code and defines the type's members such as methods and fields. The code shown in Figure 2 is the defining code for the type Node<T>. Referencing code is consumer code that makes use of predefined generic code that may also be built into another assembly. Figure 3 is a referencing code example for Node<T>.

The reason that it is helpful to consider these roles is that both defining code and referencing code play a part in the actual construction of useable generic code. The referencing code in Figure 3 uses Node<T> to construct a new type named Node<Int32>. Node<Int32> is a distinct type that was built with two key ingredients: Node<T> created by defining code, and the type argument Int32 to parameter T specified by referencing code. Only with these two ingredients does generic code become complete.

Note that generic types, such as Node<T>, and types constructed from the generic type, such as Node<Int32> or Node<String>, are not related types in terms of object-oriented derivation. The types Node<Int32>, Node<String>, and Node<T> are all siblings derived directly from System.Object.

C# Generic Syntax

The CLR supports multiple programming languages, so there will be a variety of syntaxes for CLR generics. However, regardless of syntax, generic code written in one CLR-targeted language will be useable by programs written in others. I am going to cover C# syntax here because at the time of this writing, the C# syntax for generics is fairly stable among the big three managed languages. However, it is worth noting that both Visual Basic® .NET and Managed C++ should also support generics in their Whidbey releases.

Figure 4 shows basic C# syntax for generic defining code and generic referencing code. The syntax differences between the two reflect the different responsibilities of the two parties that are involved with generic code.

Figure 4 C# Essential Generic Syntax (Close Figure 4)

Defining Code Referencing Code
                    class Node<T> {
   T        m_data;
   Node<T>  m_next;
}

                    class Node8Bit : Node<Byte> {
 •••
}

                    struct Pair<T,U> {
   T  m_element1;
   U  m_element2;
}

Pair<Byte,String> pair;
pair.m_element1 = 255;
pair.m_element2 = "Hi";

                    interface IComparable<T> {
   Int32 CompareTo(T other);
}

                    class MyType : IComparable<MyType> {
   public Int32 CompareTo(MyType other)
   { ... }
}

void Swap<T>(ref T item1, ref T item2) {
   T temp = item1;
   item1 = item2; item2 = temp;
}

Decimal d1 = 0, d2 = 2;
Swap<Decimal>(ref d1, ref d2);

                    delegate void EnumerateItem<T>(T item);

•••
   EnumerateItem<Int32> callback =
     new EnumerateItem<Int32>(CallMe);
}
void CallMe(Int32 num) { ... }

The current plan is that the CLR, and therefore C#, will support generic classes, structures, methods, interfaces, and delegates. The left side of Figure 4 shows an example of C# syntax for each of these cases of defining code. Note that angle brackets denote a type parameter list. Angle brackets appear immediately after the name of the generic type or member. Also there are one or more type parameters in the type parameter list. Parameters also appear throughout the definition of the generic code in place of a specific CLR type or as an argument to a type constructor. The right side of Figure 4 shows examples of C# syntax for the matching referencing code cases. Note here that angle brackets enclose type arguments; a generic identifier plus brackets constructs a distinct new identifier. Also note that type arguments specify which types to use for constructing a type or method from a generic.

Let's spend a moment on defining code syntax. The compiler recognizes that you are defining a generic type or method when it encounters a type-parameter list demarcated by angle brackets. The angle brackets for a generic definition immediately follow the name of the type or method being defined.

A type-parameter list indicates one or more types that you want to leave unspecified in the definition of the generic code. Type parameter names can be any valid identifier in C# and are separated by commas. Here are some things to note regarding type parameters from the Defining Code section of Figure 4:

  • In each code example you can see that the type parameters T or U are used throughout the definitions in places where a type name would normally appear.
  • In the IComparable<T> interface example, you can see the use of both a type parameter T and a regular type Int32. In the definitions of generic code you can mix both unspecified types through type parameters and specified types using a CLR type name.
  • You can see in the Node<T> example that type parameter T can be used alone as in the definition of m_data, and can also be used as part of another type construction as in the case of m_next. Use of a type parameter as an argument to another generic type definition, such as Node<T>, is called an open generic type. The use of a concrete type as a type argument, such as Node<System.Byte>, is called a closed generic type.
  • Like any generic method, the example generic method Swap<T> as seen in Figure 4 can be part of a generic or non-generic type and can be an instance, virtual, or static method.

In this column I am using single character names such as T and U for type parameters, primarily to keep things simple. However, you'll see that it is possible to use descriptive names as well. For example, the Node<T> type could be equivalently defined as Node<ItemType> or Node<dataType> in production code.

At the time of this writing, Microsoft has standardized on single-character type parameter names in library code to help distinguish the names from those used for ordinary types. Personally, for production code I like camelCasing type parameters because it distinguishes them from simple type names in code while remaining descriptive.

Generic referencing code is where the unspecified becomes specified. This is necessary if referencing code will actually use generic code. If you look at the examples in the Referencing Code section of Figure 4, you will see that in all cases, a new type or method is constructed from a generic by specifying CLR types as type arguments to the generic. In generic syntax, code like Node<Byte> and Pair<Byte,String> denote type names for new types constructed from a generic type definition.

I have one more syntax detail to cover before I dig in deeper on the technology itself. When code calls a generic method such as the Swap<T> method in Figure 4, the fully qualified call syntax includes any type arguments. However, it is sometimes possible to optionally exclude the type arguments from the call syntax, as shown in these two lines of code:

Decimal d1 = 0, d2 = 2;
Swap(ref d1, ref d2);

This simplified call syntax relies on a feature of the C# compiler called type inference, where the compiler uses types of the parameters passed to the method to infer the type arguments. In this case the compiler infers from the data types of d1 and d2 that the type argument to the type parameter T should be System.Decimal. If ambiguities exist, type inference won't work for the caller and the C# compiler will produce an error suggesting that you use the full call syntax, complete with angle brackets and type arguments.

Indirection

A good friend of mine is fond of pointing out that most elegant programming solutions revolve around adding an additional level of indirection. Pointers and references allow a single function to affect more than one instance of a data structure. Virtual functions allow a single call-site to route calls to a set of similar methods—some of which may be defined at a later time. These two examples of indirection are so common that the indirection itself often goes unnoticed by the programmer.

A primary purpose of indirection is to increase the flexibility of code. Generics are a form of indirection in which the defininition does not result in code that is directly usable. Instead, in defining generic code a "code factory" is created. The factory code can then be used by referencing code in order to construct code that can be used directly.

Let's look at this idea first with a generic method. The code in Figure 5 defines and references a generic method named CompareHashCodes<T>. The defining code creates a generic method named CompareHashCodes<T>, but none of the code shown in Figure 5 ever calls CompareHashCodes<T> directly. Instead, CompareHashCodes<T> is used by the referencing code in Main to construct two different methods: CompareHashCodes<Int32> and CompareHashCodes<String>. These constructed methods are instances of CompareHashCodes<T> and they are called by referencing code.

Figure 5 Indirection as a Generic Method (Close Figure 5)
using System;

class App {
   public static void Main() {
      Boolean result;

      // Constructs an instance of CompareHashCodes<T> for Int32
      result = CompareHashCodes<Int32>(5, 10);

      // Constructs an instance of CompareHashCodes<T> for String
      result = CompareHashCodes<String>("Hi there", "Hi there");
   }

   // Generic method, a "method factory" of sorts
   static Boolean CompareHashCodes<T>(T item1, T item2) {
      return item1.GetHashCode() == item2.GetHashCode();
   }
}

Normally, the definition of a method directly defines what the method does. A generic method definition, on the other hand, defines what its constructed method instances will do. The generic method itself does nothing but act as a model for how specific instances are constructed. CompareHashCodes<T> is a generic method from which method instances that compare hash codes are constructed. A constructed instance, such as CompareHashCodes<Int32>, does the actual work; it compares hash codes of integers. In contrast, CompareHashCodes<T> is one level of indirection removed from being callable.

Generic types are similarly one level of indirection removed from their simple counterparts. A simple type definition such as a class or structure is used by the system to create objects in memory. For example, the System.Collection.Stack type in the class library is used to create stack objects in memory. In a way, you can think of the new keyword in C# or the newobj instruction in intermediate language code as an object factory that creates object instances using a managed type as the blueprint for each object.

Generic types, on the other hand, are used to instantiate closed types rather than object instances. A type, constructed from a generic type, can then be used to create objects. Let's revisit the Node<T> type defined in Figure 2 along with its referencing code shown in Figure 3.

A managed application could never create an object of type Node<T>, even though it's a managed type. This is because Node<T> lacks sufficient definition to be instantiated as an object in memory. However, Node<T> can be used to instantiate another type during the execution of an application.

Node<T> is an open generic type and is only used for creating other constructed types. If a constructed type created using Node<T> is closed, such as Node<Int32>, then it can be used to create objects. The referencing code in Figure 3 uses Node<Int32> much like a simple type. It creates objects of type Node<Int32>, calls methods on the objects, and so on.

The extra layer of indirection that generic types provide is a very powerful feature. Referencing code that makes use of generic types results in tailor-made managed types. Framing generic code in your mind as a level of indirection removed from its simple counterpart will help you to intuit many of the behaviors, rules, and uses of generics in the CLR.

Conclusion

This time around you saw the benefits of generic types—how their use can improve type safety, code reuse, and performance. You also got a taste of the syntax in C# and saw how generics lead to another level of indirection, resulting in greater flexibility. Stay tuned next time when I'll dig deeper into generics internals.

Send your questions and comments for Jason to dot-net@microsoft.com.

Jason Clark provides training and consulting for Microsoft and Wintellect (http://www.wintellect.com) and is a former developer on the Windows NT and Windows 2000 Server team. He is the coauthor of Programming Server-side Applications for Microsoft Windows 2000 (Microsoft Press, 2000). You can get in touch with Jason at JClark@Wintellect.com.


C++ Q&amp;A: Retrieving Hidden Path Names, Mouse Events in C#

C++ Q&A
Retrieving Hidden Path Names, Mouse Events in C#
Paul DiLascia

Code download available at: CQA0309.exe (164KB)

Q In your January 2002 column, you answered a question about customizing the Open File dialog. You showed how to get selected file names directly from the list control by calling GetItemText, but you pointed out that there's no way to get the true file name from the list control when the user has checked "Hide extensions for known file types" in the Windows® Explorer folder options dialog. At the end of the column you said, "I'm sure there's an answer to this question, but to find out what it is, you'll have to read next month's column. Same bat time... same bat channel..." Well, I tuned to the bat channel but I never saw an answer. What gives?

Several Readers

A Gosh, do you guys really pay attention to everything I write? In fact, the solution is quite simple. The reason I never published it is that it involves undocumented features I was never able to confirm with the Redmondtonians. Then, well, I kind of forgot about it. So let me now set the record straight lest you think the bats have flown the coop. (Hey, bats don't live in coops!)

In the January 2002 column, I wrote a class called CFileDlgHelper that provides handy functions for dealing with file dialogs. There's CFileDlgHelper::GetListCtrl to get the list control that holds the file names and GetItemName to get the name of an item from the list control. As I pointed out back then, the problem with GetItemName (which calls CListCtrl::GetItemText) is that if the user has decided to hide extensions for known file types in Windows Explorer, the return string does not include the extension. For example, there's no way to distinguish bugs.txt from bugs.bmp since GetItemName returns bugs for both files. What to do?

The (undocumented) trick is knowing that in the Open File dialog, the list item data—that is, the DWORD returned by CListCtrl::GetItemData—is actually the item's PIDL. For those of you unfamiliar with shell programming, a PIDL uniquely identifies a shell object such as a file, folder, link, disk drive, or pseudo-object like the My Documents folder. For ordinary files, the PIDL is just the file's relative path name in wide characters.

If you know the item's data is really its PIDL, you can get its path name. All it requires is a bit of yucky shell programming. First, send the dialog a CDM_GETFOLDERIDLIST message to get the PIDL of the current folder. This is a two-step process: send CDM_GETFOLDERIDLIST once with a NULL buffer to get the length, and again after allocating a real buffer. Next you have to combine the folder's PIDL (path name) and the item's PIDL (relative path name) to get the full PIDL (full path name). You have to get the folder's IShellFolder and call IShellFolder::GetDisplayNameOf with the magic flag SHGDN_FORPARSING, which gets the full path name, including the extension.

To protect you from all those programming grungies, I added a new function, CFileDlgHelper::GetItemPathName, that does what it claims. To use it all you have to do is write the following:

// get true path name of ith list item
CString path = m_dlgHelper.GetItemPathName(i);

Figure 1 Modified Test Program
Figure 1 Modified Test Program

What could be easier? CFileDlgHelper::GetItemPathName is just a wrapper that calls CFileDlgHelper::GetDisplayNameOf, which comes in several overloaded flavors, one of which does all the work. Figure 1 shows the modified test program displaying the true paths of selected items with the same base name. Figure 2 shows the new functions in all their glory. As always, you can grab the full source from the link at the top of this article. You know what they say—better late than never!

Figure 2 FileDlgHelper.cpp (Close Figure 2)
////////////////////////////////////////////////////////////////
// MSDN Magazine -- January 2002
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual C++ 6.0 for Windows XP and probably Windows 2000
// too.
// Set tabsize = 3 in your editor.
//
#include "StdAfx.h"
#include "FileDlgHelper.h"
•••
//////////////////
// Get display name of item in file open dialog. Flags tell how.
// SHGDN_FORPARSING gets the full path name even when user has
// checked "Hide extensions for known file types" in Explorer.
//
CString CFileDlgHelper::GetDisplayNameOf(int i, DWORD flags)
{
   CListCtrl* plc = GetListCtrl();
   ASSERT(plc);
   return GetDisplayNameOf((LPCITEMIDLIST)plc->GetItemData(i), flags);
}

// Overload to get from PIDL.
CString CFileDlgHelper::GetDisplayNameOf(LPCITEMIDLIST pidl, DWORD flags)
{
   CString path;

   // First, get PIDL of current folder by sending CDM_GETFOLDERIDLIST
   // Get length first, then allocate.
   int len = m_pDlg->GetParent()->SendMessage(CDM_GETFOLDERIDLIST, 0,NULL);
   if (len>0) {
      CComQIPtr<IMalloc> malloc;
      SHGetMalloc(&malloc);
      LPCITEMIDLIST pidlFolder = (LPCITEMIDLIST)malloc->Alloc(len);
      ASSERT(pidlFolder);
      m_pDlg->GetParent()->SendMessage(CDM_GETFOLDERIDLIST, len,
         (LPARAM)(void*)pidlFolder);

      // Now get IShellFolder for pidlFolder
      CComQIPtr<IShellFolder> ishDesk;
      SHGetDesktopFolder(&ishDesk);
      CComQIPtr<IShellFolder> ishFolder;
      ishDesk->BindToObject(pidlFolder, NULL,
         IID_IShellFolder, (void**)&ishFolder);

      // finally, get the path name from pidlFolder
      path = GetDisplayNameOf(ishFolder, pidl, flags);

      malloc->Free((void*)pidlFolder);
   }
   return path;
}

// Overload to get from IShellFolder/pidl
CString CFileDlgHelper::GetDisplayNameOf(IShellFolder* ish,
   LPCITEMIDLIST pidl, DWORD flags)
{
   CString name;
   STRRET str;
   str.uType = STRRET_WSTR;
   if (SUCCEEDED(ish->GetDisplayNameOf(pidl, flags, &str))) {
      name = str.pOleStr;
   }
   return name;
}

//////////////////
// Get underlying list control
//
CListCtrl* CFileDlgHelper::GetListCtrl()
{
   ASSERT(m_pDlg);
   CWnd* pWnd = m_pDlg->GetParent()->GetDlgItem(lst2);
   ASSERT(pWnd);
   CListCtrl* plc = (CListCtrl*)pWnd->GetDlgItem(1);
   ASSERT(plc);
#ifdef _DEBUG
   char classname[32];
   ::GetClassName(plc->m_hWnd, classname, sizeof(classname));
   ASSERT(strcmp(classname,"SysListView32")==0);
#endif
   return plc;
}

Q As a C++/MFC developer, I'm used to being able to provide right mouse click functionality to my user. I have just moved over to C# and the .NET Framework. Where has the right mouse click gone on most of the controls? Can I make it available on such things as textboxes, panels, labels, and the main window background?

David Rogers

A Sometimes the .NET Framework is so easy that C++ programmers can't figure it out. That's because they're used to doing everything the hard way. If all you want to do is display a context menu, there's no need to mess with events. Just set your control's ContextMenu property, like so:

ContextMenu myContextMenu = ...
myControl.ContextMenu = myContextMenu;
That's all. Now your menu is automatically displayed when the user right-clicks.

If you want to do something fancier like implement right-drag, you need to use events. Coming from MFC, it's natural to think of implementing OnRButtonDown or OnRButtonUp, but that's not how Windows Forms works. While many window classes do provide virtual functions that you can override to handle common messages (for example, Form.OnActivated and Form.OnClosed), in general, when programming for the .NET Framework you must learn to think in terms of events, and look at the events a class exposes.

Figure 3 lists some of the more common events exposed by the Control class. I've highlighted the mouse events: MouseDown, MouseEnter, MouseHover, MouseLeave, MouseMove, and MouseUp, which do as their names imply. So if you want to handle the right-button down event, you can install a MouseDown event handler and check the MouseEventArgs for MouseEventArgs.Button == MouseButtons.Right. The following code snippet illustrates this:

public class MyControl : Control
{
   // ctor
   public MyControl() {
      // install button-down event handler
      this.myctl1.MouseDown +=
         new MouseEventHandler(this.OnMouseDown);
   }

   // hander
   private void OnMouseDown(object sender, MouseEventArgs e) {
      if (e==MouseButtons.Right)
         // do something
   }
}

Figure 3 Public Events for the Control Class (Close Figure 3)

Click | HelpRequested | MouseLeave
ContextMenuChanged | Invalidated | MouseMove
DoubleClick | KeyDown | MouseUp
DragDrop | KeyPress | MouseWheel
DragEnter | KeyUp | Move
DragLeave | Layout | Paint
DragOver | Leave | ParentChanged
EnabledChanged | LostFocus | Resize
Enter | MouseDown | SizeChanged
FontChanged | MouseEnter | StyleChanged
GotFocus | MouseHover | TextChanged

Figure 3 reveals three useful mouse events not directly available through MFC: MouseEnter, MouseHover, and MouseLeave. In Win32®, these are available through the special TrackMouseEvent function. The .NET Framework raises MouseEnter and MouseLeave when the mouse enters or leaves your control. These events are useful when you want to highlight your control as the mouse is moved over it, for instance highlighting a link in a browser. The .NET Framework raises MouseHover when the mouse lingers over your control for a determined amount of time (as specified by SystemParametersInfo(SPI_GETMOUSEHOVERTIME)). It's useful for tooltips and various other timed popup-doohickeys.

How do all these mouse events relate to the familiar Click event? Think of Click as a high-level, logical event, as opposed to the more low-level mouse events. The meaning of Click depends on the type of control. For example, Forms raise a Click event if the user presses either the left or right button, whereas TextBoxes only raises Click for the left button. Figure 4 shows what Click means for various kinds of controls.

Figure 4 Events Raised by Click (Close Figure 4)

Control | Left Mouse Click | Left Mouse Double-click | Right Mouse Click | Right Mouse Double-click | Middle Mouse Click | Middle Mouse Double-click
MonthCalendar, DateTimePicker, RichTextBox, HScrollBar, VScrollBar | None | None | None | None | None | None
Button, CheckBox, RadioButton | Click | Click, Click | None | None | None | None
ListBox, CheckedListBox, ComboBox | Click | Click, DoubleClick | None | None | None | None
TextBox, DomainUpDown, NumericUpDown | Click | Click, DoubleClick | None | None | None | None
*TreeView, *ListView | Click | Click, DoubleClick | Click | Click, DoubleClick | None | None
ProgressBar TrackBar | Click | Click, Click | Click | Click, Click | Click | Click, Click
Form, DataGrid, Label, LinkLabel, Panel,GroupBox, PictureBox, Splitter, DoubleClick, StatusBar,ToolBar, TabPage, **TabControl | Click | Click, DoubleClick | Click | Click, DoubleClick | Click | Click, DoubleClick
* The mouse pointer must be over a child object (TreeNode or ListViewItem) ** The TabControl must have at least one TabPage in its TabPages collection

Finally, when desperate, you can always override Control.WndProc to handle any WM_XXX message you want—for example, WM_NCLBUTTONDOWN:

protected override void WndProc(ref Message m)
{
    // value comes from winuser.h
    const int WM_NCLBUTTONDOWN = 0x00A1;
    if (m.Msg==WM_NCLBUTTONDOWN) {
        //do something
    }
    base.WndProc(ref m); // don't forget!
}

Figure 5 MouseTrap
Figure 5 MouseTrap

I wrote a short program called MouseTrap that shows how. Figure 5 shows it running. If you go the WndProc route, you're more or less programming in C because you have to deal with IntPtrs to get the message params—but since you're already familiar with C++, you'll feel right at home!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


Resource File: Microsoft Virtual Machine Remediation

Resource File
Microsoft Virtual Machine Remediation

Contents

On this page we provide information on the most current Microsoft developer resources. This month we'll discuss methods for transitioning away from the use of the Microsoft virtual machine.

Developers who use the Microsoft virtual machine (VM) will need to transition away from its use by January 2004. To comply with a court settlement with Sun Microsystems, Microsoft will not be able to update the Microsoft VM, even to repair critical security vulnerabilities. No new Microsoft products will utilize the Microsoft VM. Microsoft will be providing a tool to ensure that customers will be able to protect their systems from any vulnerabilities that are discovered after January 2004.

It is important to understand what types of applications are affected. If you have built Web sites that expect the Microsoft VM in conjunction with the <APPLET> or <OBJECT> tag, or applications or components built with the Microsoft SDK for Java or Visual J++ (any version), you will be affected by this change. JavaScript, JScript, J2EE, and ECMAScript usage will not be affected unless they use one of the technologies just mentioned.

Microsoft recommends that if you use Visual J++ and the Microsoft SDK for Java and you've built applications or Web sites that use the Microsoft VM, start to replace it now. If you use the _language_ extensions in the Microsoft VM, you should plan to convert your applications using either Microsoft Visual J# .NET or the Microsoft Java Language Conversion Assistant (JLCA). These products will assist you in migrating your existing source code to the .NET Framework and help you understand the specific extensions available.

The first step in any plan to migrate away from the Microsoft VM is to understand its dependencies. In order to assist in this, Microsoft has made available a tool that will scan your systems and produce a report on the affected resources. Once you understand your dependencies, it is recommended that you migrate from the Microsoft VM by doing one or more of the following:

  • Use Visual J# .NET to recompile your Microsoft VM code to the Microsoft .NET Framework.
  • Use the JLCA to convert your Microsoft VM code to C# on the .NET Framework.
  • Replace your Java applets with either ASP.NET or DHTML/ECMAScript.
  • Move to a third-party display technology (such as Flash or Shockwave).
  • Replace any use of the deprecated <APPLET> tag with <OBJECT>.

Visual J# .NET is a tool for developers who use the Java language to build applications and services on the .NET Framework. It provides you with a simple way of taking your existing Visual J++ or Microsoft SDK for Java applications and moving them to the .NET Framework without leaving the Java language. Microsoft has also released the beta version of an add-on to Visual J# .NET, the J# Browser Controls, that will round out support for migrating applet code to .NET using the Java language.

The JLCA, built on ArtinSoft migration technology, automatically converts most existing Java-language source code into C#, converting both language syntax and library calls.

If you used applets to overcome the limitations of early versions of HTML, you can take this opportunity to adopt a server-side solution. By converting to either ASP.NET (and utilizing ASP.NET server controls for advanced functionality) or replacing their applets with DHTML and ECMAScript, you'll be able to provide a solution to your users that can be viewed from any browser. With ASP.NET you'll also be able to provide device support to PDAs and cell phones from one Web site with minimal effort.

Developers can also move completely to another rendering technology. Many companies provide rich rendering technology that can be viewed in a number of browsers and that generally has far more functionality and flexibility than that provided by applets.

However, if you want to continue using applets, at a minimum you should follow the recommendation of the W3C and switch from the deprecated <APPLET> tag to the supported <OBJECT> tag. This provides you with additional functionality and the ability to select from a number of different Java Runtime Environments (JREs). If you want to move your existing standalone applications from the Microsoft VM, many third-party JRE vendors provide guidance on how to achieve this.

For further information see:
  • Transitioning from the Microsoft Virtual Machine
  • Java Technologies Support Center
  • Visual J# .NET Developer Center

For the Microsoft JLCA see:
  • Microsoft Java Language Conversion Assistant 2.0