 MSDN Magazine July 2007

MSDN Magazine July 2007

Mobility: Make Your WPF Apps Power-Aware

Here Andre Michaud shows you how to use power notifications to make your applications power aware.

Andre Michaud Share Code: Write Code Once For Both Mobile And Desktop Apps

If you're building .NET client apps already, target them to Windows Mobile using the same skills and toolsets.

Daniel Moth NTFS: Enhance Your Apps With File System Transactions

Presented here is an overview of Transactional NTFS and how it revolutionizes transactions.

Jason Olson Security: Applying Cryptography Using The CNG API In Windows Vista

Cryptography Next Generation (CNG) is meant to be a long-term replacement for the CryptoAPI, providing replacements for all of the cryptographic primitives it offered.

Kenny Kerr Speak Up: Support Dictation With Text Services Framework

Learn how to make your apps speech-aware by supporting Windows Speech Recognition and the Text Services Framework.

Eric Brown Editor's Note: What Is Mobility?

Does mobility come from a motorized scooter, a cell phone, or a Pocket PC? It depends on who you are.

Joshua Trupin Toolbox: Find Bottlenecks, Code Similarities, and More

This month, find performance and memory bottlenecks, essential security reading, and more.

Scott Mitchell CLR Inside Out: Digging into IDisposable

This month: cleaning up resources that are not garbage collected by the CLR garbage collector.

Shawn Farkas Data Points: ADO.NET Entity Framework Overview

The new Entity Framework in ADO.NET will let you manipulate data using an object model. John Papa explains.

John Papa Cutting Edge: Context-Sensitive Feedback with AJAX

Beyond progress bars: talking to server-side apps with ASP.NET AJAX.

Dino Esposito Service Station: WCF Bindings In Depth

The WCF programming model makes it easy to configure services with a variety of wire formats and message protocols, thanks to binding.

Aaron Skonnard Bugslayer: Wait Chain Traversal

Windows Vista has a new API called Wait Chain Traversal (WCT), which allows you to determine when and why a process is deadlocked. Read on.

John Robbins Security Briefs: Active Directory Cache Dependencies

If you're not taking advantage of Active Directory, you should be. Learn the benefits from Keith Brown.

Keith Brown Foundations: Extending the WPF Animation Classes

Learn what the WPF Animation Classes can do, what they can't, and how to extend them.

Charles Petzold { End Bracket }: Weaving Your Photos with Photosynth

See what the Interactive Visual Media Group at Microsoft Research is up to these days.

Richard Szeliski


Mobility: Make Your WPF Apps Power-Aware

Mobility

Make Your WPF Apps Power-Aware

Andre Michaud
This article discusses:
  • Creating power-aware applications
  • Power support on Windows XP
  • Power support on Windows Vista
  • Custom WPF dependency properties and routed events
This article uses the following technologies:
.NET Framework 3.0, Windows Vista, Windows XP
Code download available at: PowerAware2007_07.exe (360KB)
Contents
Existing Work
Overview of the Code
Notification of System Power Changes
PowerModeChanged Event and Handler
Trapping the WM_POWERBROADCAST Window Message
Registering for Notifications on Windows Vista
Detecting the Operating System
Custom Power Dependency Properties
Custom Power Events
Windows XP Power Structs
Initializing the Dependency Properties on Windows XP
Windows Vista Power Structs
How the Message Is Handled
Using the Dependency Properties and Events
Controlling an Animation
Conclusion

A
s mobile computers become increasingly prevalent, battery life has become more important. Hardware manufacturers are aware of this and design mobile devices with battery consumption in mind. But can the same be said for your software? If not, maybe you should begin to think about coding software that is power-aware. Both Windows® XP and Windows Vista™ provide system information to enable your applications to do just that. In this article, I provide a starting point from which developers can understand power awareness in their Windows Presentation Foundation applications.

First, you must understand that Windows Presentation Foundation itself does not inherently make power notifications available to a would-be power-aware application. You’ll see that for this functionality, my sample relies on the operating system—either Windows XP or Windows Vista (the operating systems on which Windows Presentation Foundation is designed to run).

Before I show you how to get this information into a Windows Presentation Foundation application, I should note that the power notification system on Windows Vista is more evolved than that on Windows XP. Windows Vista contains all the functionality of the Windows XP power notification system and then some, which can present quite a challenge. What are the notification systems, how do they differ, and how do you manage these differences to design a successful power-aware application? These are especially important questions in light of the fact that Windows Presentation Foundation applications are meant to run on both Windows XP and Windows Vista.

Existing Work

There are two excellent hands-on labs designed for those who are looking for sample code and hands-on experience with the Windows XP and Windows Vista power notification systems. These labs can be found at (for Windows XP) and (for Windows Vista). The labs cover the power notification systems as they would be used in managed code; however, they don’t represent how the power notification systems would be used specifically in Windows Presentation Foundation. Still, these labs provide an excellent starting point and cover the notification systems as a whole in greater detail than I can do here. I recommend you use them in conjunction with this article.

Another important point is that every application will have its own unique concerns in regard to power consumption. Not every application will perform 3D animation or write data to disk when the system is about to enter a suspended state. So, a single sample or hands-on lab cannot realistically address all the power needs of all applications. I would thus encourage developers to review the details of each notification system, as well as the measures taken in this article, within the context of his or her application’s needs.

Overview of the Code

In this article, I attempt to reconcile the differences between the Windows XP and Windows Vista power notification systems for use in a Windows Presentation Foundation application. The bulk of the work is performed inside a custom class: PowerAwareWindow. This class derives from the Window class (the element in which many applications live) and exposes much of the power-related system information and notifications in a fashion typical of the Windows Presentation Foundation—using custom dependency properties and events. The definition of this custom class can be found in the PowerAwareWindow.cs file (all of the code for this article is available for download from the MSDN® Magazine Web site).

The functions, variables, and associated structures that make power notifications from the operating system possible exist in native code. In order to make the work that is done in the PowerAwareWindow class more readable, these native elements are placed in a separate code file, NativeMethods.cs.

The sample application, which uses PowerAwareWindow, lives within the Window1.xaml and Window1.xaml.cs files. Rather than being hosted within a standard Window, the application is hosted within a PowerAwareWindow; it hooks up to the custom events and makes use of the custom dependency properties declared within the PowerAwareWindow to receive and respond to changes in the system’s power status.

Note that the sample code in the download differs slightly from the code presented here. Many of the native elements of the sample reside within the NativeMethods.cs file. Therefore, in the sample code, when native elements are referenced they are prefixed with NativeMethods. For example, the downloadable code refers to the operating system version as:

NativeMethods.VISTA_OS_VERSION

In this article, however, for clarity I simply refer to it as:

VISTA_OS_VERSION

In addition, the formatting differs slightly for better readability, the comments differ, and the code in the download includes some Debug.Assert calls. These calls are included as an extra precaution—some sections of the code assume that the application is running on Windows Vista and others assume it to be running on Windows XP.

Notification of System Power Changes

The first step is to get notified of power-related system changes. As I mentioned earlier, this article does not provide a complete overview of the Windows XP and Windows Vista notification systems. I present a subset of this functionality specific to its use in the Windows Presentation Foundation. (For a complete overview of the notification systems, you should check out the Mobile PC Development Guides listed in the "Additional Resources" sidebar.)

I discuss two ways in which you can acquire notifications of system power changes: with the PowerModeChanged event on the SystemEvents class in the Microsoft.Win32 namespace in System.dll, and by trapping the WM_POWERBROADCAST window message.

The PowerModeChanged event is actually built on top of the WM_POWERBROADCAST message, but was done so before new power-related features were added to Windows Vista. As such, on Windows Vista the latter approach is more powerful. For example, when using only the PowerModeChanged event, an application would know nothing about the Windows Vista user’s defined power plan (to be discussed shortly).

PowerModeChanged Event and Handler

The PowerModeChanged event passes a PowerModes enumeration to registered event handlers. This enum has three values: Resume, StatusChange, and Suspend. With this event, an application is notified when:

  • The OS is about to resume from a suspended state (Resume)
  • The power source has transitioned between AC and battery or some other change has occurred in the status of the system power supply (StatusChange)
  • The operating system is about to be suspended (Suspend)

In the PowerAwareWindow, this event would normally be hooked up in the Window.Loaded event handler and unhooked in the Window.Closed event handler (see Figure 1). This, of course, is after the Window.Loaded and Window.Closed events are hooked up. This event covers much of the power-related information that an application might be interested in, and it may be the best option for your power-aware application. If an application is interested only in this information, then this method of receiving power events is much more straightforward than the window message trapping method that I’ll discuss momentarily.

Figure 1 Hooking Up PowerModeChanged Event (Close Figure 1)
protected override void OnInitialized(EventArgs e)
{
    // Hook up the Window.Loaded and Window.Closed events
    this.Loaded += new RoutedEventHandler(PowerAwareWindow_Loaded);
    this.Closed += new EventHandler(PowerAwareWindow_Closed);
    base.OnInitialized(e);
}

void PowerAwareWindow_Loaded(object sender, RoutedEventArgs e)
{
    Microsoft.Win32.SystemEvents.PowerModeChanged +=
        new PowerModeChangedEventHandler(
            XP_SystemEvents_PowerModeChanged);
}

void PowerAwareWindow_Closed (object sender, RoutedEventArgs e)
{
    Microsoft.Win32.SystemEvents.PowerModeChanged -=
        new PowerModeChangedEventHandler(
            XP_SystemEvents_PowerModeChanged);
}

The XP_SystemEvents_PowerModeChanged function is implemented in the code download and can be viewed there. It isn’t utilized in the sample as the code for hooking up the event is commented out and is included only for demonstration purposes.

Trapping the WM_POWERBROADCAST Window Message

The operating system sends out WM_POWERBROADCAST Window messages to alert listening applications of changes to power-related status or settings. However, in order to receive some of these notifications, an application must first register for them. The sample accompanying this article uses the window messages for power notifications (rather than the PowerModeChanged event) in the interest of thoroughness.

With WM_POWERBROADCAST, an application can receive the full gamut of system power change notifications. In order to receive window messages in a Windows Presentation Foundation application, an event handler must first be declared to receive window messages, like so:

void PowerAwareWindow_Loaded(object sender, RoutedEventArgs e)
{
    WindowInteropHelper helper = new WindowInteropHelper(this);
    HwndSource source = HwndSource.FromHwnd(helper.Handle);
    source.AddHook(new HwndSourceHook(this.MessageProc));
    ...
}

This code declares the MessageProc function as the window message event handler. That function will check for the WM_POWERBROADCAST message (see Figure 2) and act accordingly.

Figure 2 WM_POWERBROADCAST Message (Close Figure 2)
const int WM_POWERBROADCAST = 0x0218;

IntPtr MessageProc(
    IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)
{
    switch (msg)
    {
        case WM_POWERBROADCAST:
            // Handle the message here...
            break;

       // Handle other window messages (if the application needs to) here
    }
}

The information about the event is contained in the wParam of the window message. The wParam can take many forms to indicate different power-related system events. For example, the following value indicates that either the power source changed or that the battery percentage remaining has changed:

const int PBT_APMPOWERSTATUSCHANGE = 0x000A;

These indicate that the system is about to enter or is in the process of leaving a suspended state:

const int PBT_APMSUSPEND = 0x0004;
const int PBT_APMRESUMESUSPEND = 0x0007;

Some values are not available on all platforms. For example, the value shown below is available starting with Windows Vista and indicates that one of the following has changed: the power source, the power plan, the battery percentage remaining, or the monitor state.

const int PBT_POWERSETTINGCHANGE = 0x8013;

A complete list of such events can be found in the Power Management documentation. For comparison, the PBT_APMPOWERSTATUSCHANGE, PBT_APMSUSPEND, and PBT_APMRESUMESUSPEND wParams are analogous to the StatusChange, Suspend, and Resume PowerModes in the PowerModeChange event, respectively.

It is important to note the difference between wParam values for PBT_APMPOWERSTATUSCHANGE and the PBT_ POWERSETTINGCHANGE. The latter can be understood as a Windows Vista-only superset of PBT_APMPOWERSTATUSCHANGE. It indicates all the power-related changes that PBT_APMPOWERSTATUSCHANGE indicates as well as other information specific only to Windows Vista, such as power plans and monitor state. Both of these wParams exist and are passed along with the WM_POWERBROADCAST window message on Windows Vista, but processing both would result in redundancy in this sample when running on Windows Vista.

The PBT_POWERSETTINGCHANGE wParam, however, will not be sent to a Windows Vista application until the application registers for power setting notifications (PBT_APMPOWERSTATUSCHANGE is received without registration). This is a very important step for receiving power notifications specific to Windows Vista.

Registering for Notifications on Windows Vista

To register for power setting notifications on Windows Vista, the first step is to make the RegisterPowerSettingNotification function available for use in the application. You can do that with the following P/Invoke function definition:

[DllImport(@”User32.dll”, SetLastError=true,
    CallingConvention=CallingConvention.StdCall)]
static extern IntPtr RegisterPowerSettingNotification(
    IntPtr hRecipient, ref Guid PowerSettingGuid, Int32 Flags);

This function returns a handle that will be used to unregister for the events when the application is shutting down.

The Flags passed into this function indicate to the operating system how the application would like to receive notifications. For this sample, I declare this as:

const int DEVICE_NOTIFY_WINDOW_HANDLE = 0x00000000;

This indicates to the registration function that notifications are to be sent to the application using the WM_POWERBROADCAST window message with a wParam of PBT_POWERSETTINGCHANGE. The other option for Flags is relevant for OS services and is explained in the RegisterPowerSettingNotification documentation mentioned earlier.

The PowerSettingGuid value passed to the registration function indicates to the operating system the notifications in which the application is interested. Figure 3 shows the four PowerSettingGuid values you’re probably most interested in.

Figure 3 Four Types of PowerSettingGuid (Close Figure 3)
// notify application when percentage of battery remaining changes
static Guid GUID_BATTERY_PERCENTAGE_REMAINING =
    new Guid(“A7AD8041-B45A-4CAE-87A3-EECBB468A9E1”);

// notify application when the state of the monitor changes
static Guid GUID_MONITOR_POWER_ON =
    new Guid(“02731015-4510-4526-99E6-E5A17EBD1AEA”);

// notify application when the system power source changes
static Guid GUID_ACDC_POWER_SOURCE =
    new Guid(“5D3E9A59-E9D5-4B00-A6BD-FF34FF516548”);

// notify application when the user-defined power plan changes
static Guid GUID_POWERSCHEME_PERSONALITY =
    new Guid(“245D8541-3943-4422-B025-13A784F679B7”);

Other notification GUIDs are available and can be used to get notified for events such as when the system is entering or exiting an away mode or when the system will be moving into an idle state in the near future and the current time is a good time to perform background or idle tasks.

In the sample, I have grouped all the Windows Vista power registration into one function (see Figure 4). Similarly, I have also grouped all the unregistration into one function. Registration occurs when the PowerAwareWindow loads (the Window.Loaded event is handled by PowerAwareWindow_Loaded function) and unregistration occurs when it closes (Window.Closed handled by PowerAwareWindow_Closed).

Figure 4 Windows Vista Power Registration (Close Figure 4)
IntPtr _hBattCapacity;
IntPtr _hMonitorOn;
IntPtr _hPowerScheme;
IntPtr _hPowerSrc;

void RegisterForVistaPowerNotifications(IntPtr hWnd)
{
    _hPowerSrc = RegisterPowerSettingNotification(
        hWnd, ref GUID_ACDC_POWER_SOURCE, DEVICE_NOTIFY_WINDOW_HANDLE);
    _hBattCapacity = RegisterPowerSettingNotification(
        hWnd, ref GUID_BATTERY_PERCENTAGE_REMAINING,
        DEVICE_NOTIFY_WINDOW_HANDLE);
    _hMonitorOn = RegisterPowerSettingNotification(
        hWnd, ref GUID_MONITOR_POWER_ON, DEVICE_NOTIFY_WINDOW_HANDLE);
    _hPowerScheme = RegisterPowerSettingNotification(
        hWnd, ref GUID_POWERSCHEME_PERSONALITY,
        DEVICE_NOTIFY_WINDOW_HANDLE);
}

void UnregisterForVistaPowerNotifications()
{
    UnregisterPowerSettingNotification(_hBattCapacity);
    UnregisterPowerSettingNotification(_hMonitorOn);
    UnregisterPowerSettingNotification(_hPowerScheme);
    UnregisterPowerSettingNotification(_hPowerSrc);
}

Registration introduces one of the main issues that I address in this sample—to create a consistent approach to power-awareness in one Windows Presentation Foundation application that accounts for the differences regarding power notifications on Windows XP and Windows Vista. The RegisterPowerSettingNotification and UnregisterPowerSettingNotification functions exported from User32.dll are not available on operating systems prior to Windows Vista; as a result, any code that tries to use these functions should only do so after first ensuring the functions are available.

Detecting the Operating System

It is important for your application to detect the operating system and act accordingly. To this end, my sample has a private variable marked readonly for storing this information:

readonly int _osVersion = Environment.OSVersion.Version.Major;

The operating system major versions for Windows Vista and Windows XP are:

const int VISTA_OS_VERSION = 6;
const int XP_OS_VERSION = 5;

At this point, the PowerAwareWindow_Loaded and PowerAwareWindow_Closed functions should look something like that shown in Figure 5. At this point, the application is receiving power notifications via window messages in the MessageProc function. Once the power notifications are received, the application needs to respond appropriately.

Figure 5 PowerAwareWindow_Loaded and_Closed Functions (Close Figure 5)
void PowerAwareWindow_Loaded(object sender, RoutedEventArgs e)
{
    // Optional: If the PowerModeChanged event contains the desired power-
    // notification functionality for an application, no OS detection is
    // necessary as PowerModeChanged will work with both
    SystemEvents.PowerModeChanged += new PowerModeChangedEventHandler(
        XP_SystemEvents_PowerModeChanged);

    // Use the helper to hook up our Windows Presentation Foundation window
    // to receive window messages.
    WindowInteropHelper helper = new WindowInteropHelper(this);
    HwndSource source = HwndSource.FromHwnd(helper.Handle);
    source.AddHook(new HwndSourceHook(this.MessageProc));

    if (_osVersion >= VISTA_OS_VERSION)
    {
        RegisterForVistaPowerNotifications(source.Handle);

        // There’s no need to initialize the dependency properties on
        // Windows Vista because this application will get notified of the
        // power status when it registers for the events.
    }
    else
    {
        // Application power initialization should be done here (otherwise
        // the application is left waiting for a WM_POWERBROADCAST window
        // message that might not come right away).
    }
}

void PowerAwareWindow_Closed(object sender, EventArgs e)
{
    // Optional: Unregister for PowerModeChanged if registration for it
    // was done in PowerAwareWindow_Loaded
    SystemEvents.PowerModeChanged -= new PowerModeChangedEventHandler(
        XP_SystemEvents_PowerModeChanged);

    if (_osVersion >= VISTA_OS_VERSION)
    {
        UnregisterForVistaPowerNotifications();
    }
    else
    {
        // Do Windows XP-specific application cleanup here
    }
}

Custom Power Dependency Properties

In many cases it is necessary for a power-aware application to declare and make use of custom dependency properties other than the ones declared in this sample. Note that custom dependency properties is an advanced topic. If you need more background before reading this section, check out the Custom Dependency Properties documentation.

While these custom dependency properties may seem complicated, they provide a lot of power and flexibility for an application that uses the PowerAwareWindow. The custom dependency properties declared in the sample are PowerPlan, MonitorOn, RemainingBattery, and RunningOnBattery.

PowerPlan exposes the current user-specified power plan. This piece of information is specific to Windows Vista. On Windows XP this will always be a default value (Automatic). The associated PowerPlanChanged event does not fire on Windows XP as there is no notion of power plans on that OS.

The PowerPlan enumeration is defined as:

public enum PowerPlan
{
    Automatic,
    HighPerformance,
    PowerSaver,
};

In the Windows Vista UI, these are called "Balanced," "High Performance," and "Power Saver," respectively. The dependency property registration looks like the code shown in Figure 6.

Figure 6 PowerPlan Dependency Property Registration (Close Figure 6)
private static readonly DependencyPropertyKey PowerPlanPropertyKey =
    DependencyProperty.RegisterReadOnly(
        “PowerPlan”, typeof(PowerPlan), typeof(UIElement),
        new FrameworkPropertyMetadata(PowerPlan.Automatic,
            FrameworkPropertyMetadataOptions.Inherits |
            FrameworkPropertyMetadataOptions.Journal |
            FrameworkPropertyMetadataOptions.
                OverridesInheritanceBehavior,
            new PropertyChangedCallback(OnPowerPlanChanged)));

public static readonly DependencyProperty PowerPlanProperty =
    PowerPlanPropertyKey.DependencyProperty;

private static void OnPowerPlanChanged(
    DependencyObject d, DependencyPropertyChangedEventArgs e)
{
    PowerAwareWindow myPowerAwareWindow = d as PowerAwareWindow;
    if (myPowerAwareWindow != null &&
        myPowerAwareWindow.PowerPlanChanged != null)
    {
        myPowerAwareWindow.PowerPlanChanged(myPowerAwareWindow, e);
    }
}

public event DependencyPropertyChangedEventHandler PowerPlanChanged;

public PowerPlan PowerPlan
{
    get { return (PowerPlan)GetValue(PowerPlanProperty); }
    private set
    {
        SetValue(PowerAwareWindow.PowerPlanPropertyKey, value);
    }
}

MonitorOn exposes the current state of the monitor (display)—either on or off. This piece of information is specific to Windows Vista. On Windows XP, it will always have a default value (True). The associated MonitorOnChanged event does not fire on Windows XP as there is no notion of Monitor state in that OS.

The dependency property registration looks very similar to the one described for PowerPlan, except that the property is of type Boolean, the property is named MonitorOn, and the corresponding event is MonitorOnChanged. The full implementation for this and the rest of the dependency properties can be found in the code download.

RemainingBattery exposes the percentage of the battery currently remaining—ranging from 0 to 100 percent. For this case, the dependency property registration is very similar to that of the previous two dependency properties.

RunningOnBattery is true if the system is running on a battery and false if it is running on AC power.

Custom Power Events

As with the custom dependency properties, it is also often necessary for a power-aware application to declare and make use of custom events. (If you need more background on custom events, check out the Routed Event Overview). The custom events in my sample are PreviewSystemResuming, SystemResuming, PreviewSystemSuspending, and SystemSuspending.

PreviewSystemResuming occurs when the system is resuming from a suspended state. The Preview part of the event indicates that the event’s routing strategy is Tunnel as opposed to Direct or Bubble. The event registration looks like the code shown in Figure 7.

Figure 7 PreviewSystemResuming Event Registration (Close Figure 7)
static readonly RoutedEvent PreviewSystemResumingEvent =
    EventManager.RegisterRoutedEvent(“PreviewSystemResumingEvent”,
        RoutingStrategy.Tunnel, typeof(RoutedEventHandler),
        typeof(PowerAwareWindow));

public event RoutedEventHandler PreviewSystemResuming
{
    add { AddHandler(PreviewSystemResumingEvent, value); }
    remove { RemoveHandler(PreviewSystemResumingEvent, value); }
}

//  This is called when the PowerAware window wants to raise the event
void RaisePreviewSystemResumingEvent()
{
    RaiseEvent(new RoutedEventArgs(
        PowerAwareWindow.PreviewSystemResumingEvent, this));
}

SystemResuming occurs when the system is resuming from a suspended state. This uses the Bubble routing strategy version of the PreviewSystemResuming event. The event registration looks like the code shown in Figure 8.

Figure 8 SystemResuming Event Registration (Close Figure 8)
static readonly RoutedEvent SystemResumingEvent =
    EventManager.RegisterRoutedEvent(“SystemResumingEvent”,
        RoutingStrategy.Bubble, typeof(RoutedEventHandler),
        typeof(PowerAwareWindow));

public event RoutedEventHandler SystemResuming
{
    add { AddHandler(SystemResumingEvent, value); }
    remove { RemoveHandler(SystemResumingEvent, value); }
}

//  This is called when the PowerAware window wants to raise the event
void RaiseSystemResumingEvent()
{
    RaiseEvent(new RoutedEventArgs(
        PowerAwareWindow.SystemResumingEvent, this);
}

PreviewSystemSuspending occurs when the system is about to enter a suspended state. The Preview part of the event indicates that the event’s routing strategy is Tunnel, as it is with PreviewSystemResuming.

SystemSuspending occurs when the system is about to enter a suspended state. As with SystemResuming, it uses the Bubble routing strategy.

Windows XP Power Structs

The custom dependency properties aren’t much use unless they contain the latest system information. Similarly, the custom power events need to be raised appropriately. Let’s look at this now, as well as some of the trickier things involved in getting Windows Presentation Foundation power-awareness to function well for a single application that runs on both Windows XP and Windows Vista.

Let’s start with a look at important Windows XP power structures. First there’s the SYSTEM_POWER_STATUS (see Figure 9). As you can see, this sample uses the ACLineStatus and BatteryFlag enumerations.

Figure 9 SYSTEM_POWER_STATUS (Close Figure 9)
class SYSTEM_POWER_STATUS
{
    public ACLineStatus ACLineStatus;
    public BatteryFlag BatteryFlag;

    // can be a value in the range 0 to 100, or 255 if status is unknown
    public byte BatteryLifePercent;
    public byte Reserved1;            // reserved; must be zero

    // number of seconds of battery life remaining,
    // or –1 if remaining seconds are unknown
    public Int32 BatteryLifeTime;

    // number of seconds of battery life when at full charge,
    // or –1 if full battery lifetime is unknown
    public Int32 BatteryFullLifeTime;
}

The SYSTEM_POWER_STATUS is returned from the GetSystemPowerStatus function. It is made available to the application with the following import:

[DllImport(“Kernel32.DLL”, CharSet=CharSet.Auto, SetLastError=true)]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool GetSystemPowerStatus(
    [In, Out] SYSTEM_POWER_STATUS SystemPowerStatus);

This function is called whenever the WM_POWERBROADCAST message is received with the PBT_APMPOWERSTATUSCHANGE wParam. This wParam indicates that something in the system power has changed and the GetSystemPowerStatus function gives the application access to that information. That information is returned inside the SYSTEM_POWER_STATUS that is passed in.

Initializing the Dependency Properties on Windows XP

As discussed in the Detecting the Operating System section, the dependency properties need to be initialized when the application is loading in order for them to be up-to-date (see Figure 5). This is not necessary in Windows Vista, as the operating system makes the application aware of the current system power status through the window message when the application registers for power notifications. This, however, does not happen on Windows XP and therefore initialization is necessary. This initialization is done in the PowerAwareWindow_Loaded function when the operating system is detected and appears as follows:

void InitializeDependencyPropertiesForXP()
{
    if (GetSystemPowerStatus(this._xpPowerStatus))
    {
        this.RunningOnBattery =
            (ACLineStatus.Battery == _xpPowerStatus.ACLineStatus);
        this.RemainingBattery = _xpPowerStatus.BatteryLifePercent;
    }

    this.MonitorOn = true;
    this.PowerPlan = PowerPlan.Automatic;
}

At this point it is important to point out that there isn’t a Windows XP monitor or power plan notification analogous to the Windows Vista notification. Therefore, when a Windows XP application attempts to get these values, default values will be returned. And even if the application hooks up the MonitorOnChanged event or the PowerPlanChanged event, after initialization nothing will trigger the firing of the events.

Windows Vista Power Structs

Next up are the Windows Vista power structures. POWERBROADCAST_SETTING is the main Windows Vista power struct:

struct POWERBROADCAST_SETTING
{
    public Guid PowerSetting;
    public UInt32 DataLength;
}

In the official definition of the struct, there is a third member present: Data. As indicated in the struct, this can vary in size, being either a GUID or a DWORD. Thus, declared within the application is a custom variation of this struct that can hold either of these data types (note that this struct doesn’t mimic the layout of the original native structure and thus can’t be used for marshaling; it’s simply used so that the values can continue to be passed around within the application in a strongly typed fashion):

struct my_POWERBROADCAST_SETTING
{
    public IntPtr wParam;
    public Guid PowerSetting;
    public int PowerChangeData;
    public Guid PowerPlanGuid;
}

This struct also includes the wParam so that with this custom variation, all relevant data can be passed from the window message handler to the application’s Windows Vista message handler (defined later) in one struct.

So when receiving the WM_POWERBROADCAST window message on Windows Vista along with a wParam of PBT_POWERSETTINGCHANGE, the lParam associated with the message is a pointer to a POWERBROADCAST_SETTING struct that contains information about the change in the system’s power status.

One last important bit of information concerns how Windows Vista defines Power Plans. Maximum Power Savings involves very aggressive power-savings measures to help stretch battery life:

static Guid GUID_MAX_POWER_SAVINGS =
    new Guid(“a1841308-3541-4fab-bc81-f71556f20b4a”);

No Power Savings uses almost no power savings measures:

static Guid GUID_MIN_POWER_SAVINGS =
    new Guid(“8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c”);

Finally, the typical Power Savings power plan uses relatively aggressive power savings measures:

static Guid GUID_TYPICAL_POWER_SAVINGS =
    new Guid(“381b4222-f694-41f0-9685-ff5bb260df2e”);

These correspond to PowerPlan.PowerSaver, PowerPlan.HighPerformance, and PowerPlan.Automatic, respectively. As mentioned earlier, they appear as Power Saver, High Performance, and Balanced in the Windows Vista UI, respectively.

How the Message Is Handled

The code in Figure 10 provides a high-level look at the MessageProc function. There are two important points to explain here. The first point is how the work is sent to each of the handlers. Second is how (and when) data is grabbed in the event of a WM_POWERBROADCAST message on Windows Vista.

Figure 10 MessageProc Function (Close Figure 10)
IntPtr MessageProc(
    IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)
{
    switch (msg)
    {
        case WM_POWERBROADCAST:
            if (_osVersion >= VISTA_OS_VERSION)
            {
                // Grab the relevant power data from the message here.

                // Then send the data and work to the Windows Vista
                // handler here.
            }
            else
            {
                //  Send work to the Windows XP handler here.
            }
            break;

        // Other (non-power) window messages in which the
        // application is interested would be processed here
        // (if the application needs to do so).
    }

    return IntPtr.Zero;
}

Normally, sending the work to each of the handlers would be as simple as calling functions defined in the sample code. These functions would handle the message appropriately when the application is running on either Windows XP or Windows Vista. In this case, though, a bit more precaution has been taken to prevent system hangs. If, for example, the function that handles the message is complicated and processor-intensive, then control wouldn’t be returned to the window message handle immediately and this could result in noticeable system hangs. To avoid hangs like this, the work is sent to the Windows Vista and Windows XP handlers via a post to the Dispatcher thread (see Figure 11).

Figure 11 Asynchronous Invocation (Close Figure 11)
delegate void VoidXPDelegate(/*power data arguments*/ );
delegate void VoidVistaDelegate(/*power data arguments*/ );

// A call to the Windows XP handler would look like this:
this.Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle,
    new VoidXPDelegate(/*XP handler function here*/),
    /*power data here*/);

// A call to the Windows Vista handler would look like this:
this.Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle,
    new VoidVistaDelegate(/*Vista handler function here*/),
    /* power data here*/);

The message will be handled asynchronously at the specified priority on the thread with which the Dispatcher is associated; control is returned immediately to the MessageProc function after the work is posted to the Dispatcher. More on the Dispatcher and Windows Presentation Foundation threading can be found in the WPF Fundamentals Threading Model documentation.

Now for the topic of grabbing data for Windows Vista power events. The wParam portion of the WM_POWERBROADCAST window message is crucial to understanding the nature of the message, as explained previously. On Windows XP, this is all that is necessary for successfully handling the power message. On Windows Vista, however, this wParam can indicate that the lParam in the message is a pointer to data necessary for determining the nature of the message. Figure 12 shows how I copy that data out of the lParam pointer. There you can see the MessageProc function with these changes.

Figure 12 Copying Data out of the lParam Pointer (Close Figure 12)
// First grab the wParam, that’s important information to the
// Windows Vista handler no matter what
my_POWERBROADCAST_SETTING myPBS = new my_POWERBROADCAST_SETTING();
myPBS.wParam = wParam;

if (wParam == (IntPtr)PBT_POWERSETTINGCHANGE)
{
    // Grab the power change information from the pointer passed
    // in the window message
    POWERBROADCAST_SETTING ps = (POWERBROADCAST_SETTING)
        Marshal.PtrToStructure(lParam, typeof(POWERBROADCAST_SETTING));

    // Copy the PowerSetting in the POWERBROADCAST_SETTING to
    // our own structure
    myPBS.PowerSetting = ps.PowerSetting;

    // lParam points to the POWERBROADCAST_SETTING, but we need to
    // calculate a pointer to where the data about the setting lives.
    IntPtr pData = (IntPtr)((int)lParam + Marshal.SizeOf(ps));

    // Use the pointer calculated to grab the relevant data. Note that
    // the relevant data can be either a DWORD or a GUID, depending on
    // the datalength indicated in the POWERBROADCAST_SETTING struct.
    if (ps.DataLength == Marshal.SizeOf(typeof(Int32)))
    {
        myPBS.PowerChangeData = (Int32)Marshal.PtrToStructure(
            pData, typeof(Int32));
    }
    else if (ps.DataLength == Marshal.SizeOf(typeof(Guid)))
    {
        myPBS.PowerPlanGuid = (Guid)Marshal.PtrToStructure(
            pData, typeof(Guid));
    }
}

// At this point ‘myPBS’ contains all the data the Windows Vista handler
// needs for this application to process the system power change

Once you have the relevant data for the message, you need to update the relevant properties and fire the relevant events. Considering all the work that’s been done thus far to get notifications and set up custom dependency properties, this step is actually very straightforward. All it involves is either assigning the data provided by the operating system to the appropriate application variables or raising the appropriate custom events.

Using the Dependency Properties and Events

Now let’s take a look at an application that is hosted within a PowerAwareWindow. At this level, power-awareness fits very well into the Windows Presentation Foundation model and creating a power-aware application that runs on both Windows XP and Windows Vista is quite straightforward.

The first thing to examine is the user interface, as defined in Window1.xaml. Note that the application lives within a Power- AwareWindow:

<local:PowerAwareWindow x:Class=”WPFPower_Aware.Window1”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:local=”clr-namespace:WPFPower_Aware”
    x:Name=”MainWindow”
    Title=”WPF Power Aware Application” Height=”480” Width=”550”
    Loaded=”Window1_Loaded”
>
    ...
</local:PowerAwareWindow>

As a result, the XAML file can make use of the custom dependency properties for data binding. My sample, for example, contains a status bar at the bottom of the window. Within the status bar, there are two pieces of power-related data—a text block indicating the current system power source and a progress bar indicating the current percentage of remaining battery power. The values assigned to these are data-bound to their corresponding dependency properties.

You’ll find that data binding is quite simple. Consider the progress bar, for example:

<ProgressBar IsIndeterminate=”False” x:Name=”myBatteryProgressBar”
    Width=”200” Height=”20”
    Value=”{Binding ElementName=MainWindow,
            Path=RemainingBattery, Mode=OneWay}” />

With this little bit of markup, the progress bar will always be up-to-date and no codebehind intervention is necessary. You can even data-bind to your own custom control, such as a battery meter, to display this information!

The power-source text block is somewhat more complicated as it involves converting from a Boolean value (RunningOnBattery) to a string for the text block. If you’d like to view the details of this, take a look in the sample code.

Controlling an Animation

TIf you run the sample application, you’ll see two animation elements. The one on the left is controlled in the codebehind, the other is controlled in XAML. The animation that is controlled by XAML starts and stops when the RunningOnBattery property is false and true, respectively.

The snippet in Figure 13 is for an animation that demonstrates the power of custom dependency properties via their use in XAML. With this piece of XAML, no code intervention is necessary to stop and start the animation. And this is just the beginning—there’s much more you can do with animations and triggers in XAML. For more information, see the Windows Presentation Foundation Animation Overview and the Trigger Class documentation.

Figure 13 Custom Dependency Properties in XAML (Close Figure 13)
<Rectangle Name=”rect” Width=”140” Height=”90” Fill=”Blue” Stroke=”Red”>
  <Rectangle.RenderTransform>
    <ScaleTransform/>
  </Rectangle.RenderTransform>
  <Rectangle.Style>
    <Style>
      <Style.Triggers>
        <Trigger Property=”local:PowerAwareWindow.RunningOnBattery”
  Value=”False”>
          <Trigger.EnterActions>
            <BeginStoryboard Name=”myStoryboard”>
              <Storyboard>
                <DoubleAnimation Storyboard.TargetProperty=
                  “(Rectangle.RenderTransform).(ScaleTransform.ScaleY)”
                  From=”1” To=”1.5” RepeatBehavior=”Forever”
                  AutoReverse=”True”/>
              </Storyboard>
            </BeginStoryboard>
          </Trigger.EnterActions>
          <Trigger.ExitActions>
            <StopStoryboard BeginStoryboardName=”myStoryboard”/>
          </Trigger.ExitActions>
        </Trigger>
      </Style.Triggers>
    </Style>
  </Rectangle.Style>
</Rectangle>

As I mentioned, the left element in the sample app has an animation that is controlled in the codebehind. When a relevant system power event occurs, the current power state of the system is used to determine if the animation should be paused to conserve power. First, the animation gets set up and then the custom events are hooked up. The necessary code for this is found in Window1 .xaml.cs (see Figure 14).

Figure 14 Code in Window1.xaml.cs (Close Figure 14)
DoubleAnimation myDoubleAnimation;
Storyboard myStoryboard;

void Window1_Loaded(object sender, RoutedEventArgs e)
{
    // Initialize the animation
    myDoubleAnimation = new DoubleAnimation();
    myDoubleAnimation.From = 0.0;
    myDoubleAnimation.To = 360.0;
    myDoubleAnimation.Duration = new Duration(TimeSpan.FromSeconds(10));
    myDoubleAnimation.RepeatBehavior = RepeatBehavior.Forever;

    // Note that yRotate is an AxisAngleRotation3D declared
    // in the Window1.xaml file
    Storyboard.SetTargetName(myDoubleAnimation, “yRotate”);
    Storyboard.SetTargetProperty(myDoubleAnimation,
        new PropertyPath(AxisAngleRotation3D.AngleProperty));
    myStoryboard = new Storyboard();
    myStoryboard.Children.Add(myDoubleAnimation);

    // Begin the animation if appropriate
    if (!ShouldStopAnimation()) myStoryboard.Begin(this, true);

    // Hook up the events from the PowerAwareWindow
    this.RemainingBatteryChanged +=
        new DependencyPropertyChangedEventHandler(
            Window1_RemainingBatteryChanged);
    this.RunningOnBatteryChanged +=
        new DependencyPropertyChangedEventHandler(
            Window1_RunningOnBatteryChanged);
    this.SystemSuspending += new RoutedEventHandler(
        Window1_SystemSuspending);
    this.SystemResuming += new RoutedEventHandler(
        Window1_SystemResuming);
    this.MonitorOnChanged +=
        new DependencyPropertyChangedEventHandler(
            Window1_MonitorOnChanged);
    this.PowerPlanChanged +=
        new DependencyPropertyChangedEventHandler(
            Window1_PowerPlanChanged);
}

Deciding how and when the application should take measures to conserve power depends largely on the behavior and needs of your application. In this sample, the decision is made as follows, and the power event handlers make use of this function in the appropriate event handlers:

bool ShouldStopAnimation()
{
    // If the monitor is off, or if we’re running on a battery
    // and the user hasn’t explicited requested high performance,
    // stop the animation
    return
        !MainWindow.MonitorOn ||
        (MainWindow.PowerPlan != PowerPlan.HighPerformance &&
         MainWindow.RunningOnBattery);
 }

One thing to note in this sample is that power events are reported in a list box in the UI. Here, for example, is the RunningOnBatteryChanged event handler:

void Window1_RunningOnBatteryChanged(
    object sender, DependencyPropertyChangedEventArgs e)
{
    myEventsListBox.Items.Add(
        “RunningOnBattery changed - now it’s {0}”, (bool)e.NewValue);

    if (ShouldStopAnimation()) myStoryboard.Pause(this);
    else myStoryboard.Resume(this);
}

You can learn more about other measures an application can take to conserve power by taking a look in the Mobile PC Power and Device Awareness documentation.

Conclusion

This article and the accompanying code provide a starting point from which you can learn to make your Windows Presentation Foundation applications power-aware. Note that there is a lot more to power awareness and the Windows Presentation Foundation than I’ve discussed here. Therefore, I highly recommend that you review the resources mentioned throughout this article before you design your power-aware application. You’ll discover many more capabilities and techniques that will help you tailor the best solution for your program.

Additional Resources

For general information regarding power and device management, and ideas on how you can reduce power consumption, check out these MSDN links for helpful documentation:

  • Mobile PC User Experience Guidelines for Developers: Power and Device Awareness
  • Mobile PC Development Guide: Power Management on Windows Vista
  • Mobile PC Development Guide: Power Management on Windows XP
  • Win32 and COM Development: Power Management

For an overview of the Windows Presentation Foundation features used in this article, check out these links:

  • Routed Event Overview
  • Custom Dependency Properties documentation
  • WPF Fundamentals: Threading Model documentation
  • Windows Presentation Foundation Data Binding Overview
  • Windows Presentation Foundation: Animation Overview

And for specific definitions of certain functions, classes, and enumerations used in this article, see these links:

  • SystemEvents.PowerModeChanged Event documentation
  • WM_POWERBROADCAST documentation
  • RegisterPowerSettingNotification documentation
  • SYSTEM_POWER_STATUS documentation
  • POWERBROADCAST_SETTING documentation
  • Trigger Class documentation

Andre Michaud is an SDET on the Tablet PC Platform Team at Microsoft where he works with handwriting recognition. His previous experience includes the Tablet PC components of the Windows Presentation Foundation.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Share Code: Write Code Once For Both Mobile And Desktop Apps

Share Code

Write Code Once For Both Mobile And Desktop Apps

Daniel Moth
This article discusses:
  • Windows Mobile and Windows desktop development
  • Porting applications between platforms
  • Retargetable assemblies
  • Sharing code between platforms
This article uses the following technologies:
.NET Framework
Code download available at: NetCfSharing2007_07.exe (246KB)
Contents
Platform Differences
How Not to Proceed
Retargetable
Share the Code, Not the Binary
Project and Code Files in the Same Directory
Stub It
Platform Invoke
Different Framework Implementations
Partial Types
System.Diagnostics.Conditional
Deploy to My Computer
Use Reflection
Extension Methods
Wrapping It Up

F
or the last several years, while developers were building Microsoft® .NET Framework client applications for Windows®, many had no idea that they could also have been creating applications for Windows Mobile® using the same skills and toolsets. But Windows Mobile wasn’t as widespread in the enterprise then, so the need to write custom applications targeting mobile devices was not as great. Today, there’s a huge demand and many desktop developers are getting their feet wet with mobile development. Unfortunately, many miss the opportunity to share their .NET code cross-platform even though it is relatively easy to do so.

There are many reasons you might want to share business logic across both platforms, regardless of whether the original application was written for Windows or Windows Mobile. You can start from either, and port to the other. If your application is currently running on laptops in the field, it is probably a good candidate to run on a mobile device, and vice versa. And if your product is similar to your competitor’s, extending the reach to a new platform gives you an edge. As a bonus, by writing your code to execute on the desktop you can take advantage of developer tools that don’t work with smart device projects.

The techniques and principles discussed in this article can be utilized when writing cross-device code—applications that target Windows devices with differing form factors (different screen sizes, orientations, touch screens and so on).

Platform Differences

When I think of the .NET Framework, I think of the tools, the languages, the libraries, and the runtime engine. Differences in these four elements affect the goal of writing cross-platform code. Fortunately, the tool we use for both desktop and mobile apps is the same: Visual Studio®. With each version of Visual Studio, the Visual Studio for Devices team continues to add new features for device development and to ensure that important new desktop development features work as expected for mobile device development. (You need to use Visual Studio 2005 Standard or higher for device development.)

Both Visual Basic® and C# are fully supported on both platforms, from a syntactical perspective. But if you plan to use Visual Basic, be advised that the Microsoft.VisualBasic.Compatibility assembly and implicit late binding are not supported. The former is an assembly used on desktop projects that are migrated from Visual Basic 6.0, and the latter is not an issue if you are following best practices by enabling Option Strict On in Visual Basic projects. Also realize that managed C++ is not supported (although the same IDE can be used for native device projects). Third-party languages can add support for the .NET Compact Framework by referencing the correct assemblies and by ensuring that their compilers don’t generate IL opcodes that are not supported by the .NET Compact Framework common language runtime (CLR). The offending IL opcodes are: jmp, calli, refanyval, refanytype, mkrefany, arglist, localloc, unaligned, and tail.

It’s worth noting that while the implementations of the garbage collector and the just-in-time (JIT) compiler are different in the .NET Compact Framework than they are in the .NET Framework, they still serve the same purpose and offer the same services to managed code.

Like any modern programming framework, the .NET Framework comes with a rich set of class libraries. The .NET Compact Framework achieves 80 percent of the relevant functionality found in the .NET Framework with only 20 percent of the footprint. Major areas of the .NET Framework that are absent from the compact version include: ASP.NET, CLR hosting, code access security (CAS), binary serialization, Reflection.Emit, and CodeDOM.

Windows Presentation Foundation and Windows Workflow Foundation are not going to be supported any time soon, but there will be a compact version of Windows Communication Foundation in version 3.5 of the .NET Compact Framework. Also in version 3.5, there will be support for LINQ to Objects, LINQ to XML, and LINQ to DataSet, but no support for any other LINQ flavors.

Most of the time, however, it isn’t these large missing areas that cause issues. Typically, problems arise from the fact that some of the supported namespaces don’t have all the classes you expect and, more importantly, some classes do not have all the members you’d expect. I’ll show you some workarounds later on.

The reverse is also true. That is, the .NET Compact Framework is not a strict subset of the .NET Framework; it actually adds some members of its own. There are four device-specific assemblies: System.Data.SqlServerCe, System.Net.IrDA, Microsoft.WindowsMobile.DirectX and Microsoft.WindowsCE.Forms.

SqlServerCe contains classes for working with the in-memory database on the device, IrDA contains classes for programming against infrared, DirectX® is used for rich graphics primarily for game programming, and Forms has a number of GUI classes that are specific to devices. In addition to these device-specific assemblies, there are also some device-specific Windows Mobile APIs for Windows Mobile 5.0 and higher that ship as part of the Mobile platform. Later I’ll show you how to cope with types that exist on one framework and not the other.

Of course, input methods on desktop and mobile devices are different, so you don’t have to worry about keyboard shortcuts, tooltips, and so forth on a mobile device. Moreover, mobile forms are not resizable and are always full screen, which also affects the way you navigate between forms of an application. Finally, the most obvious difference is that the screen size of devices is much smaller. The takeaway is that it is best not to share the UI layer of an application between these two platforms, but rather create platform-specific user interfaces and instead just share the business logic.

How Not to Proceed

The first thing desktop developers usually do when trying to port an application to a mobile device is to take an assembly built against the desktop framework and reference it from a device project. This will not work and is definitely not supported. Even if in your desktop project you’ve been careful to use only types and members that are also available on the .NET Compact Framework, you will still get TypeLoadException and MissingMethodException exceptions at run time. So, if you created a Visual Studio project by choosing a non-smart-device project template, the output of that project will simply not execute on the Windows CE platform (which is what Windows Mobile builds on, of course).

Unfortunately, it is not hard to find yourself in the situation just described. When referencing assemblies from a smart-device project, there are no checks to see if it’s a desktop assembly, so you could be using class libraries built from a desktop project template. This situation arises often when using third-party DLLs. In such cases, you’ll only discover the errors at run time.

If you suspect you are facing this problem, there are a few clues to look for. When you deploy the project to the target through Visual Studio (such as when you run your project with debugging), notice in the Visual Studio output window the diagnostic messages that tell you which assemblies are being deployed to the device (see Figure 1). If the deployment process is taking too long, it may be because Visual Studio is copying to the target the full set of .NET Framework assemblies that it detected as dependencies to the desktop assembly you accidentally referenced.

Figure 1 Output Window of Visual Studio
Figure 1 Output Window of Visual Studio(Click the image for a smaller view)
Figure 1 Output Window of Visual Studio
Figure 1 Output Window of Visual Studio (Click the image for a larger view)

If you are lucky, the deployment process will fail with an error informing you that there is not enough disk space. If the desktop dependencies are limited, then the deployment process will succeed. In that case, using a remote file viewer that shows all file types, look into the folder on the target where your application was deployed. You’ll find all the .NET Framework assemblies that were deployed. Note that the public key token for the .NET Compact Framework assemblies starts with 9 whereas the desktop equivalents start with B. You can use that information to quickly determine whether what you are deploying or referencing is a desktop or a device framework assembly. That information also appears in the .NET Compact Framework loader log, which will contain errors if your application fails at run time due to these issues.

Retargetable

Though desktop assemblies cannot execute on the device, the good news is that the reverse is possible. If you create a smart device project and build it, the output (the EXE or DLL) can be executed on the desktop and can be referenced by desktop projects. This works because the .NET Compact Framework assemblies are retargetable. If you open one of them with ILDASM, you can observe that they have the System.Reflection.AssemblyNameFlags.Retargetable attribute (see Figure 2). This means that at run time all references to the .NET Compact Framework assemblies are retargeted to the desktop equivalents. So, to be clear, your assembly uses the desktop implementation, not the device implementation of the .NET Framework.

Figure 2 Retargetable Flag in ILDASM
Figure 2 Retargetable Flag in ILDASM(Click the image for a smaller view)
Figure 2 Retargetable Flag in ILDASM
Figure 2 Retargetable Flag in ILDASM (Click the image for a larger view)

Let’s look at an example you can try at home. Create a new smart device application targeting your favorite Windows Mobile platform. Add some controls to the main form, for example a TabControl, a ProgressBar, and four MenuItems to the MainMenu control (see Figure 3). Generate the Click event handler methods for each of the MenuItems and add the code in Figure 4 in these event handler methods.

Figure 4 Click Event Handler Methods (Close Figure 4)
private void mnuFormShow_Click(object sender, EventArgs e)
{
  // TODO later
}

private void mnuMbox_Click(object sender, EventArgs e)
{
  MessageBox.Show(“some text”, “random caption”);
}

private void mnuDateTime_Click(object sender, EventArgs e)
{
  this.Text = DateTime.Now.TimeOfDay.ToString();
}

private void mnuCalc_Click(object sender, EventArgs e)
{
  string pth = @”\Windows\calc.exe”;
  Process.Start(pth, null);
}
Figure 3 MainMenu Design
Figure 3 MainMenu Design

Run the project targeting the emulator or your own device and observe that the behavior is what you expect when you click on each menu item. Back on your desktop, navigate to the build folder (bin/Debug) and notice how you can double-click on the executable and it runs on your desktop machine (see Figure 5).

Figure 5 Device Project
Figure 5 Device Project

The controls render themselves in the desktop style—the TabControl has the tabs at the top, the menus appear at the top, the ProgressBar has a slightly different appearance, and so forth. Now click on the MenuItem that shows a MessageBox and it works as expected. Click on the MenuItem that changes the caption of the form and notice how we get higher precision here with the addition of milliseconds. This is a further clue that you are now using the desktop implementation that is slightly different. The lesson here is not to completely rely on retargetability and instead test your application thoroughly to be positive that any minor differences in implementation are not important to you. Finally, click on the MenuItem that launches the calculator and observe the application crashing with a "System.ComponentModel.Win32Exception: The system cannot find the file specified" exception. Clearly, the culprit here is that the path to the calculator on the desktop is different from the path of the calculator on the device. To deal with such platform-specific issues and to offer a single implementation that works cross-platform, you need to detect at run time what platform you are on and branch your code accordingly. See the following modifications to the event handler as an example:

private void mnuCalc_Click(
  object sender, EventArgs e)
{
  string pth;
  if (Environment.OSVersion.Platform == PlatformID.WinCE)
  {
    pth = @”\Windows\calc.exe”;
  }
  else
  {
    pth = @”c:\Windows\system32\calc.exe”;
  }
  Process.Start(pth, null);
}

You can now run the project on either Windows Mobile or Windows desktop and the calculator application corresponding to the platform will be launched correctly.

Remember that there are device-specific assemblies that do not exist on the desktop. Let’s see what happens if you use such an assembly. Add a new form (Form2) to the smart device project, drop an InputPanel control on it, then add two Button controls and a TextBox. Generate the event handlers for the controls and add code as shown in Figure 6.

Figure 6 textBox Event Handlers (Close Figure 6)
private void textBox1_GotFocus(object sender, EventArgs e)
{
  inputPanel1.Enabled = true;
}

private void textBox1_LostFocus(object sender, EventArgs e)
{
  inputPanel1.Enabled = false;
}

private void button1_Click(object sender, EventArgs e)
{
  Cursor.Current = Cursors.WaitCursor;
}

private void button2_Click(object sender, EventArgs e)
{
  Cursor.Current = Cursors.Default;
}

Now revisit the empty MenuItem Click event handler method of the first form and add code that shows Form2, as follows:

private void mnuFormShow_Click(object sender, EventArgs e)
{
  Form2 f = new Form2();
  f.ShowDialog();
}

If you run the project on the device, you’ll notice that when you open Form2 from Form1’s menu item, you can click on Form2’s buttons to show or hide the busy cursor on the device. Giving the focus to the Textbox brings up the Soft Input Panel (SIP) while taking the focus away hides it. Back on the desktop, navigate to your build directory like before and run the executable on the desktop. Clicking on the menu item to show Form2 results in an exception like this:

System.IO.FileNotFoundException: Could not load file or assembly
 ‘Microsoft.WindowsCE.Forms, Version=2.0.0.0, Culture=neutral,

PublicKeyToken=969db8053d3322ac’ or one of its dependencies.
The system cannot find the file specified. File name: ‘Microsoft.WindowsCE.Forms,
 Version=2.0.0.0, Culture=neutral, PublicKeyToken=969db8053d3322ac’

This happens because when you added an InputPanel control to the project, a reference to Microsoft.WindowsCE.Forms was added automatically. That’s a device-only assembly so there’s nothing on the desktop to retarget to, thus the exception is thrown when the form gets constructed and attempts to create a Microsoft.WindowsCE.Forms.InputPanel from that assembly. Let’s make some modifications to better observe this. First delete the InputPanel control from the form’s designer view and then modify the textBox event handlers as you see in Figure 7.

Figure 7 Checking Platform (Close Figure 7)
private void textBox1_GotFocus(object sender, EventArgs e)
{
  if (Environment.OSVersion.Platform == PlatformID.WinCE) //naive
  {
    Microsoft.WindowsCE.Forms.InputPanel inputPanel1 =
                new Microsoft.WindowsCE.Forms.InputPanel();
    inputPanel1.Enabled = true;
  }
}

private void textBox1_LostFocus(object sender, EventArgs e)
{
  // in the interest of space, let’s ignore this method
  //inputPanel1.Enabled = true;
}

What this does is isolate the use of the InputPanel to one method, and it also attempts a naïve fix for the exception by checking the platform at run time. If you run this project on the device, it still works as expected, but if you run it on the desktop it fails with the same exception as before. It fails only when you give the focus to textBox1 rather than as soon as the form is shown, but it still fails nonetheless.

A runtime check is not enough in this case because of JIT compiling. At run time when the method call is made, the CLR checks to see if any native code exists for the method being called (textBox1_GotFocus). If not, then the JIT compiler will generate native opcodes from the IL code that the method entry points to. At that point, the JITer does not "care" what paths get executed at run time, it simply needs to find all the types used in the method. So it tries to load the Microsoft.WindowsCE.Forms.InputPanel type and corresponding assembly, but it can’t because the assembly doesn’t exist on the desktop. Then the exception gets thrown. You could extract the use of the InputPanel into a separate method and ensure with a run-time check from the calling method that the extracted method never gets called, but that’s really messier than most developers would like. The correct way to resolve this issue is by exploring an alternative approach to sharing code cross-platform. (If you are interested in pursuing the previously mentioned approach, see the "Platform Detection" section of msdn2.microsoft.com/ aa480686, which has an example of a factoring approach similar to the one noted here.)

Share the Code, Not the Binary

The previous section explored how you could create smart device projects and use the output binaries on the desktop via retargeting. However, there are two limitations with the retargetable approach: First, you can’t use any desktop-only types or methods in your code. They simply will not compile. Second, you must be careful not to load any device-specific assemblies on the desktop as they will fail with run-time exceptions.

Both of these problems can be solved by using conditional branching of the code at compile time rather than at run time, and by creating two projects, with two separate outputs to achieve platform-specific assemblies that both use a single set of source code files. The latter is an important point since ultimately the goal is to have all modifications to the business logic propagate to both platforms simultaneously without any copy/paste error-prone overhead. Let’s try another approach.

First, create a new desktop Windows project. Delete the code files added by default (the Form1 and Program files). From the project menu select "Add Existing Item..." and browse to the folder of the smart device project created earlier, where the code files reside. Finally, select all the code files only ( *.cs) and add them to the project as links. This is important because if you select the default Add option of the dialog, you will be making copies rather than linking to the files (see Figure 8).

Figure 8 Add Existing Files
Figure 8 Add Existing Files

Build the project and notice that you now find at compile time the error that previously was only found at run time (see Figure 9). Of course, in this case there’s no point adding a reference to the device-specific assembly, so this is where I will introduce conditional compilation.

Figure 9 Finding Problems at Compile Time
Figure 9 Finding Problems at Compile Time(Click the image for a smaller view)
Figure 9 Finding Problems at Compile Time
Figure 9 Finding Problems at Compile Time (Click the image for a larger view)

Go to the desktop project properties and under the Build tab add a new "Conditional compilation symbol" named FULL_FRAME (short for "full framework") as in Figure 10.

Figure 10 Conditional Compilation Constant
Figure 10 Conditional Compilation Constant(Click the image for a smaller view)
Figure 10 Conditional Compilation Constant
Figure 10 Conditional Compilation Constant (Click the image for a larger view)

Next, fix the compilation error in Form2.cs by using the newly introduced compilation constant. Modify the offending method as follows and then rebuild the desktop project:

private void textBox1_GotFocus(object sender, EventArgs e)
{
#if !FULL_FRAME
    Microsoft.WindowsCE.Forms.InputPanel inputPanel1 =
          new Microsoft.WindowsCE.Forms.InputPanel();
    inputPanel1.Enabled = true;
#endif
}

Now that this (desktop) project builds, you can run it and observe how everything works as expected. If you switch to the device project, note how it warns you that a source file has been changed outside of that instance of Visual Studio and asks if you want to reload it; click on "Yes."

Since this is a separate project using a separate set of conditional compilation constants, the project will compile and run fine on the device—you will not be running the output of this project on the desktop anymore! This means that you can change the runtime conditional introduced earlier when launching the calculator, to a compile-time conditional thus reducing the size of the method.

Now that you’ve seen the mechanics of using conditional compilation, let’s reflect on the benefits. Having two separate projects means you can have a different set of project references in each. You can also include code files in one project and not in the other. On the desktop side, you can take advantage of members of the full framework in order to create a good experience for the desktop user; simply ensure that usage of those desktop members is enclosed in an #if preprocessor directive or is separated out into source files that exist only in the desktop project. There are many examples in which you can use full framework members to enhance the experience on the desktop side, so let’s take a look at one.

In the Program.cs file, modify the Main function as follows:

static void Main()
{
#if FULL_FRAME
  Application.EnableVisualStyles();
#endif
  Application.Run(new Form1());
}

Now run the project and compare the appearance of the original control with the one resulting from the retargetable approach (see Figure 11).

Finally, the astute among you will have observed that the code that shows and hides the cursor will not work on the desktop (even though it produces no compilation or runtime error). Also note that device projects, starting with Visual Studio 2005, have certain conditional compilation constants already defined: PocketPC, Smartphone, Windows CE as appropriate. You can choose to use only those built-in constants or your own, such as the FULL_FRAME that was introduced, or a combination of both—it doesn’t really matter.

Figure 11 Original and Retargeted Controls
Figure 11 Original and Retargeted Controls(Click the image for a smaller view)
Figure 11 Original and Retargeted Controls
Figure 11 Original and Retargeted Controls (Click the image for a larger view)

Now that you understand both approaches to sharing code assets cross-platform, it is time to examine some tips and tricks. Some of these apply to the retargetable approach, some to the conditional compilation, and some to both.

Project and Code Files in the Same Directory

When sharing code, I created two separate projects and added as links the source code files of the first to the second. I prefer having all source code files reside in the same directory as the project files, thus making it very easy in Visual Studio Solution Explorer to include or exclude files as appropriate. To achieve this, after creating say the device project, then create a Windows desktop project, and delete the code files as before. Rather than adding any more files, close Visual Studio and navigate to the directory where the desktop project file was created. Cut the project file and paste it in the same directory where the device project file resides. For example, if your device project was at "C:\Temp\", then cut and paste the desktop project file (csproj) into that folder (ensure that desktop and device projects have different names).

Now open the desktop project and "Show All Files" in Solution Explorer. You can see all the device source code files and include in the project whichever you think are applicable—and vice versa if you open the device project (see Figure 12).

Figure 12 Choosing Files
Figure 12 Choosing Files

With this approach, when adding a new file to one of the projects, it becomes very easy to spot when you open the other project type and it is easier to include the file if you wish.

Note that the project and output assembly names are different, but the namespace and type names should remain the same. If you prefer having the same assembly name for both platforms, then in project properties you need to rename the build folders, for example, binCF and binFF. I also recommend having only one project open at any time as this avoids confusion and it works very well with source control.

Stub It

Many times it is best to avoid littering the code with conditional compilation and instead to add dummy empty stubs of the code that is not applicable on the platform. For example, if you make heavy use of a class throughout the project and that class does not apply on the other platform, you can add the class with the same (but empty) interface. Here’s an example using the InputPanel class:

namespace Microsoft.WindowsCE.Forms
{
    public class InputPanel
    {
        public bool Enabled;
    }
}

Now, wherever the InputPanel is used, the code can stay the same with no conditional branching; it will simply have no effect on the desktop platform.

Platform Invoke

Many .NET Compact Framework applications achieve additional functionality by using P/Invoke services to access functionality residing in native DLLs. Typically the names of those DLLs are different on Windows CE and desktop versions of Windows. There are a couple of ways to cope with that, depending on which of the earlier approaches you chose. If you are using conditional compilation, the code in Figure 13 illustrates how to write cross-platform P/Invoke declarations. The technique in Figure 14 is useful if you have chosen the retargetable approach.

Figure 14 Run-Time P/Invoke Decision (Close Figure 14)
class PInvokesRuntime
{
    [DllImport(“coredll.dll”, EntryPoint = “QueryPerformanceCounter”)]
    internal static extern int QueryPerformanceCounterCE(
        out Int64 perfCounter);

    [DllImport(“kernel32.dll”, EntryPoint = “QueryPerformanceCounter”)]
    internal static extern int QueryPerformanceCounterFull(
        out Int64 perfCounter);

    private void UseIt()
    {
        long p;
        int i;
        if (Environment.OSVersion.Platform == PlatformID.WinCE)
        {
            i = QueryPerformanceCounterCE(out p);
        }
        else
        {
            i = QueryPerformanceCounterFull(out p);
        }
    }
}
Figure 13 Compile-Time P/Invoke Decision (Close Figure 13)
class PInvokesCompileTime
{
#if FULL_FRAME
    private const string DllName = “kernel32.dll”;
#else
    private const string DllName = “coredll.dll”;
#endif

    [DllImport(DllName)]
    internal static extern int QueryPerformanceCounter(
        out Int64 perfCounter);

    private void UseIt()
    {
        long p;
        int i;
        i = QueryPerformanceCounter(out p);
    }
}

Different Framework Implementations

Earlier you saw how System.DateTime.TimeOfDay returns a value of different precision depending on which framework you use. So after testing you can decide how to deal with that in your code if it matters to you. There are other examples where methods behave differently and in particular there are a few cases where methods on one framework throw different exceptions than methods on the other framework, for example:

private void button1_Click(object sender, EventArgs e)
{
    Type t = this.GetType();
    MethodInfo m = t.GetMethod(“DoIt”);
    m.Invoke(this, null); //throws
}

public void DoIt()
{
    throw new InvalidOperationException(“my message”);
}

This code results in an exception on both platforms on the line that invokes the MethodInfo. On the device, the exception thrown is the InvalidOperationException. On the desktop, however, the exception thrown is a TargetInvocationException (that has the original InvalidOperationException set as the InnerException property). Therefore, during testing you should test the exceptional cases as well and ensure that you have a catch block for both Exception types.

Another example of different framework implementations is what you saw earlier with displaying the busy cursor. What works for the device has no effect on the desktop and what is required on the desktop is not supported on the device. If you want consistent behavior across both platforms, you could extract a single method and isolate there the conditional behavior that encapsulates the display of the cursor as follows:

public static class TheCursor
{
    public static void CursorCurrent(Control c, Cursor defaultOrWait)
    {
#if FULL_FRAME
        if (c != null) c.Cursor = defaultOrWait;
#else
        Cursor.Current = defaultOrWait;
#endif
    }
}

Then the calling code can remain the same across both the device and the desktop platforms:

private void button1_Click(object sender, EventArgs e)
{
    TheCursor.CursorCurrent(this, Cursors.WaitCursor);
}

private void button2_Click(object sender, EventArgs e)
{
    TheCursor.CursorCurrent(this, Cursors.Default);
}

Partial Types

Earlier I mentioned that you can include source code files in one project and not in the other if they contain platform-specific functionality. A similar scenario occurs when a class is applicable to both platforms but some members of it are not. Instead of wrapping large sections of the class with a #if directive you can use the partial classes feature introduced with Visual Studio 2005. In other words, split the class across two files using the partial keyword, thus separating out the functionality that is applicable to just one platform into a different file. Then simply do not include that file in the inapplicable project.

System.Diagnostics.Conditional

A tip worth noting is to use the System.Diagnostics.ConditionalAttribute for methods you don’t want to be called at run time. The result of applying the attribute to a method is that any calls to the method will be left out of the compiled binary. The method that has the attribute applied must still be able to compile, of course:

[Conditional(“FULL_FRAME”)]
public void SomeDesktopOnlyMethod()
{
    //TODO code that compiles in both projects
}

If you enclosed the entire method with a #if, you would have to find all places where the method was called and wrap the call with a #if as well. Instead here, you simply apply the attribute to the undesired method and ignore all the call sites; they will not be compiled in the binary.

Deploy to My Computer

If you choose the retargetable approach, you know that you can only find desktop platform errors at run time. The question of how to debug the application on the desktop arises, since a smart device project only deploys to the emulator or physical device. There are a couple of approaches you can take.

The easiest, albeit most tedious approach, is to run the application from the file system and then attach the debugger to it via the Visual Studio Tools | Attach to Process menu (see Figure 15). You can also semiautomate this process by creating a desktop project to act as the launcher. Add the desktop project to the solution with the smart device project, set it as the startup project, and reference the smart device project. Ensure the desktop project has just a single Program file with a single static Main function and from there pass to the Application.Run method a new instance of the startup form of the smart device project.

Figure 15 Attach to Process Dialog
Figure 15 Attach to Process Dialog(Click the image for a smaller view)
Figure 15 Attach to Process Dialog
Figure 15 Attach to Process Dialog (Click the image for a larger view)

It would be ideal if in the deployment options of the smart device project there was a "Deploy to My Computer"choice. In fact, during an early Community Technology Preview of Visual Studio 2005, that option existed (but it was dropped from the final product, and the option is not coming back in the Visual Studio "Orcas" release).

You can re-enable the "Deploy to My Computer" functionality by following the steps described on my blog, but please note that is not recommended or supported by Microsoft.

Use Reflection

If you have chosen the retargetable approach, then you can’t use desktop-only methods since they will not compile in the device project. However, if you really want to, you can invoke such methods using reflection. Revisiting the earlier example where I wanted to enable themes for the application, I could have achieved this as follows:

static void Main()
{
    //Application.EnableVisualStyles();
    if (Environment.OSVersion.Platform != PlatformID.WinCE)
    {
        Type t = typeof(Application);
        MethodInfo m = t.GetMethod(“EnableVisualStyles”);
        m.Invoke(null, null);
    }

    // As before
    Application.Run(new Form1());
}

Extension Methods

While using .NET Compact Framework classes, the occasion often arises that a particular class has members missing compared to the full Framework. Typically, developers have to write the missing functionality themselves and then decide whether to use inheritance if possible to add the missing members or add them to a utility helper class. For example, assume you want to perform a Cut operation on a TextBox in a device project that you are sharing with the desktop. The Cut method is missing from the TextBox class in the .NET Compact Framework. Based on what you’ve learned in this article, you might add a code file to the device project that looks like Figure 16.

Figure 16 TextBoxExtensions (Close Figure 16)
using System;
using System.Windows.Forms;
using Microsoft.WindowsCE.Forms;

static class TextBoxExtensions
{
    const int WM_CUT = 0x0300;
    const int WM_COPY = 0x0301;
    const int WM_PASTE = 0x0302;

    public static void Cut(TextBox t)
    {
        Message msg = Message.Create(
            t.Handle, WM_CUT, IntPtr.Zero, IntPtr.Zero);
        MessageWindow.SendMessage(ref msg);
    }
}

The file in Figure 16 needs no conditionals as it is only present in the device project. You could then write the calling code (that will exist in both projects) as follows:

private void SomeMethod()
{
#if FULL_FRAME
    textBox1.Cut();
#else
    TextBoxExtensions.Cut(textBox1);
#endif
}

There is a new feature called Extension Methods in C# 3.0 and in the upcoming version of Visual Basic. I don’t have space here to fully describe the feature, however, so if you are not familiar with it please visit my blog or see Anson Horton’s article in the June 2007 issue of MSDN® Magazine. The feature is also supported in .NET Compact Framework 3.5 that ships with Visual Studio code-named "Orcas" and it helps address the situation described above.

In the device project add a reference to System.Core.dll and then modify the device-only file by adding the this keyword before the first parameter of the helper method, thus making it an extension method, like so:

public static void Cut(this TextBox t)

With that change in place, the calling code can be the same cross-platform with no conditionals:

private void SomeMethod()
{
  textBox1.Cut();
}

On the desktop, the framework method will be called, and on the device the extension method will be called. If in the future the .NET Compact Framework team adds the method, then the code will seamlessly call the real .NET Framework method (as instance methods take precedence over extension methods), and at some point you can remove the superfluous code file.

Wrapping It Up

In this article I looked at the differences between developing for the desktop and developing for Windows Mobile, concluding that most of the differences are in the Framework libraries rather than anywhere else. I offered examples of two approaches for sharing code assets cross-platform, indicated my preference for the conditional compilation approach, and suggested some handy tips. While I personally recommend that you use this knowledge to share business logic contained in class libraries, you may wish to share your GUI as well, especially if the application is targeting devices such as the Ultra Mobile PC (UMPC) or other touch-screen devices running Windows desktop rather than Windows Mobile.

Daniel Moth works for Microsoft in the Developer and Platform group in the UK. He is a coauthor of Microsoft Mobile Development Handbook (Microsoft Press, 2007) and can be reached via his blog at www.danielmoth.com/Blog.


NTFS: Enhance Your Apps With File System Transactions

NTFS

Enhance Your Apps With File System Transactions

Jason Olson
This article discusses:
  • The Kernel Transaction Manager
  • The Distributed Transaction Coordinator
  • Using Transactional NTFS from native code
  • Using Transactional NTFS from managed code
This article uses the following technologies:
Windows Vista, .NET Framework
Code download available at: TxF2007_07.exe (159KB)
Contents
The Benefits of Transactional NTFS
When to Avoid Transactional NTFS
Getting Started
Secondary Resource Managers
Management
What Happened to the Transacted Command Line?
In Closing

I
t’s a revolutionary step for Windows®-based applications. And it can help you develop applications that are more robust than they could ever have been before. It’s Transactional NTFS, also known as TxF. In this article, I explain Transactional NTFS, why it matters to developers, and how you can use this new technology in your applications.

What is TxF? Conceptually, it is quite simple and can be expressed in a logical equation: Transactional NTFS = Transactions + NTFS. A new feature in Windows Vista™ and the next version of Windows Server®, code-named "Longhorn," TxF introduces the concept of transacting file operations.

When I say transacted file operations, I am talking about transactions with fully ACID semantics that can be exposed directly to the developer. This means the ability to perform actions that are fully atomic, consistent, isolated, and durable is now enabled directly for file operations. These ACID capabilities are something that database developers have benefited from for years, and having this power for file operations is a very welcomed addition.

You’re probably wondering if all these transactional capabilities have been built into NTFS, what is the performance overhead of Transactional NTFS? Well, TxF has a strictly pay-to-play model. If you aren’t using transacted file operations, there is no overhead. And even if you are using transacted file operations, the overhead is pretty minimal (in the 1 to 2 percent range). For an application that is not I/O-bound, this performance impact probably would not even be noticeable.

Here’s another aspect of performance to be aware of: Transactional NTFS is optimized for commit. When you write new pages to disk within a transaction, they are written directly in place while the previous pages are saved for retrieval by other readers if necessary. Because of this optimization, when the transaction is committed, the altered pages are already in place on disk. As a result, the overhead of committing the transaction is minimal.

The Benefits of Transactional NTFS

Transactional NTFS sounds nice, in theory. But why should you, as a developer, use TxF? There are a number of distinct reasons. Specifically, TxF achieves three goals that help you develop powerful applications:

  • Improved application stability
  • Improved platform stability
  • Increased innovation

Let’s take a closer look at how TxF achieves these goals.

Transactional NTFS enables better application stability by reducing or eliminating the amount of error-handling code that needs to be written and maintained for a given application. This ultimately reduces application complexity and makes the application easier to test. Say, for instance, you are developing a document management system where a SQL data source needs to be kept consistent with a file store on disk. Ensuring this consistency can be tricky and non-trivial in a non-transactional system. Without transactional file operations, it would be nearly impossible to account for every possible failure scenario, up to and including the operating system crashing at any imaginable point during the process.

One of the ways this was handled in the past was by storing the new version of the file with a temporary file name, writing the new data to the SQL database, and then renaming the temporary file to the real file name when committing the SQL transaction. But consider what happens if the application crashes or there is a power outage right after committing to the SQL database but before the file is renamed. Not only would you have an inconsistent data set, but you would also have an artifact on the file system that you would have to clean up at some point. As usual, the extremely difficult part lies in the details of how many different ways the process can fail.

Some solutions will implement their own entire two-phase commit protocol, which commits changes to the database and the file system at the same time. While this gets you a step closer to achieving the goal of application stability, implementing a custom two-phase commit system in your own application can be very difficult. And this requires code that you have to write, test, and maintain yourself.

Wouldn’t it be better if transaction management was built into the platform rather than having to be implemented by each developer who needs this type of behavior? With the introduction of TxF in Windows Vista and Windows Server "Longhorn," these capabilities are now built directly into the platform. And because Transactional NTFS is embedded in the system itself, it is capable of providing a level of integration that would not otherwise be possible for your applications.

One of the coolest parts about Transactional NTFS is that it can work with a large number of other transactional technologies. Because TxF uses the new Kernel Transaction Manager (KTM) features, and because the new KTM can work directly with the Microsoft® Distributed Transaction Coordinator (DTC), any technology that can work with DTC as a transaction coordinator can use transacted file operations within a single transaction. This means that you can now enlist transacted file operations within the same transaction as SQL operations, Web service calls via WS-AtomicTransaction, Windows Communication Foundation services via the OleTransactionProtocol, or even transacted MSMQ operations. You can even enlist transacted file operations directly with other technologies that are using XA-Transactions since DTC can also work with XA-Transactions. Simply put, if DTC can work with something, you can incorporate transacted file operations into it.

Using the document management example, how can you ensure consistency within a document management application more easily with TxF? This is where the DTC comes in handy. To absolutely ensure consistency between your SQL database and your file store, you can start a transaction, perform your SQL statements and file operations within that same transaction, and then commit or roll back the complete transaction based on the outcome. If your SQL call fails, your file will never be written. If your file system call fails, your SQL is rolled back. Everything remains consistent, and all of this is handled automatically by the platform since the operations are enlisted within a transaction.

When using this approach in my own application, I no longer have to worry about handling and accounting for every possible exception that could occur. I simply incorporate both operations within the same transaction, guaranteeing that if any exception is thrown, the platform will roll back the operations in question. Less code, more robust, very cool!

That leaves two more goals. As for platform stability, this is achieved by some of the ways that Microsoft is using TxF in its own technologies. There are three core features in Windows Vista and Windows Server "Longhorn" that now make use of Transactional NTFS: Windows Update, System Restore, and Task Scheduler. All of these use TxF to write files to the file system within the scope of a transaction in order to handle rollback/commit in case of any exceptions, such as a system reboot due to a loss of power. If you’ve ever experienced a loss of power right in the middle of Windows Update, you know this can leave your system dead and in need of a fresh installation. But now that Windows Update uses TxF, if this same situation occurs on your Windows Vista system, the computer can recover by rolling back the transaction when the system reboots. By adopting TxF internally, Microsoft has helped make its own operating system more stable.

Finally, TxF drives innovation by providing a framework for using transactions outside of SQL calls. Ultimately, Transactional NTFS can fundamentally change the way developers write applications, allowing them to build more robust code. By incorporating transactions into your design, you can write code without having to account for every single possible failure that can occur. The operating system will take care of those mundane details!

When to Avoid Transactional NTFS

Despite all the benefits offered by TxF, there are some scenarios in which you should think about not using it. These are the same scenarios that are discussed when talking about avoiding traditional database transactional systems.

You shouldn’t use TxF if you have a system that will have long-running transactions. The definition of "long-running" here is relative, however. A long-running transaction is any transaction that has been alive longer than many other transactions within the same log. This could mean a few seconds if there are thousands of transactions happening and lasting for very brief periods of time. On the other hand, a day or two might not be a long time if there are very few transactions happening in the system.

You should also be wary of using TxF if you are going to have multiple writers against the same file. Transacted file operations only allow for one writer at a time (even shared writers), meaning that a second writer won’t be allowed to open the file while it is being held open by another writer, unless that second writer is within the same transaction. Because of this, TxF can introduce contention into a system that has multiple writers against a single file.

Finally, you need to be careful about auto-started services and mixing transaction resources. If the service is using Transactional NTFS and SQL, the files it needs access to could remain locked until recovery completes (since your service has a direct dependency on DTC and SQL, those services need to be started first, so you might not be able to fully recover your files until DTC and SQL finish recovering first).

Getting Started

Now let’s take a look at how you can use TxF. If you are at all familiar with the existing file I/O Win32® APIs, the Transactional NTFS APIs should come naturally. The new Transactional NTFS APIs are unchanged from their non-transacted counterparts aside from the addition of a couple of parameters at the end, which are specifically for transactions. Figure 1 gives some examples of Win32 file APIs and their transacted counterparts. The complete list is much longer; nearly every file API in Win32 has a corresponding transacted version.

Figure 1 Win32 APIs and Their Transacted Counterparts (Close Figure 1)

Non-Transacted API | Transacted API
CreateFile | CreateFileTransacted
CopyFileEx | CopyFileTransacted
MoveFileWithProgress | MoveFileTransacted
DeleteFile | DeleteFileTransacted
CreateHardLink | CreateHardLinkTransacted
CreateSymbolicLink | CreateSymbolicLinkTransacted
CreateDirectoryEx | CreateDirectoryTransacted
RemoveDirectory | RemoveDirectoryTransacted

From an application perspective, there are three primary vehicles you can use when developing a TxF-aware application. You can use the new KTM directly as the transaction coordinator. This lets you develop applications that are solely using TxF file operations within their transactions.

The second option is to use DTC as the transaction coordinator. When you use this method, you need to go through DTC to get the handle for the kernel-level transaction object that you can then pass into the Transacted APIs.

The third way is actually an extension to the second approach where you are using System.Transactions from managed code to make use of DTC (or just the Lightweight Transaction Manager if there is only a single durable enlistment). In a moment, I’ll look at the challenges of this approach and what it really means to managed developers.

Right now, let’s take a closer look at these three approaches. Figure 2 illustrates all the major moving parts in the transaction platform in Windows Vista and Windows Server "Longhorn."

Figure 2 The Transaction Platform
Figure 2 The Transaction Platform(Click the image for a smaller view)
Figure 2 The Transaction Platform
Figure 2 The Transaction Platform (Click the image for a larger view)

At the API level, there are three main interfaces into TxF. The first way is to go directly to KTM via the new ktmw32.h header file. At the most basic level, this contains the transaction coordinator Win32 calls for KTM: CreateTransaction, CommitTransaction, and RollbackTransaction. In this approach, you use these calls directly and have KTM itself do the transaction coordination if you are only doing transacted file operations within the single transaction (see Figure 3).

Figure 3 TxF from Native Code with KTM (Close Figure 3)
hTx = CreateTransaction(NULL, NULL, 0, 0, 0, 0, L”MyTxn”);
hFile = CreateFileTransacted(myFileName,
            GENERIC_READ | GENERIC_WRITE,
            FILE_SHARE_READ | FILE_SHARE_WRITE,
            NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL,
            hTx, NULL, NULL);

... // work with the file using its file handle hFile

CommitTransaction(hTx);

The second interface is via DTC. With this approach, you use DTC as the transaction coordinator. To use TxF through DTC, you make use of COM and QueryInterface to grab the IKernelTransaction interface directly from DTC’s ITransaction interface. Once you have a reference to the new IKernelTransaction COM interface, you can call its GetHandle method in order to retrieve the kernel-level transaction handle that can then be used to pass to the various Transacted APIs in Win32 (see Figure 4).

Figure 4 TxF from Native Code with DTC (Close Figure 4)
hr = DtcGetTransactionManagerEx( NULL, NULL,
         IID_ITransactionDispenser, OLE_TM_FLAG_NONE,
         NULL, (void**) &pITransactionDispenser);
hr = pITransactionDispenser->BeginTransaction(
         NULL, ISOLATIONLEVEL_READCOMMITTED,
         ISOFLAG_RETAIN_BOTH, NULL, &pITransaction);
hr = pITransaction->QueryInterface(
         IID_IKernelTransaction,
         (void**) &pKernelTransaction);
hr = pKernelTransaction->GetHandle(&hTransactionHandle);
hAppend = CreateFileTransacted(TEXT(“test.txt”),
              FILE_APPEND_DATA, FILE_SHARE_READ,
              NULL, OPEN_ALWAYS,
              FILE_ATTRIBUTE_NORMAL, NULL,
              hTransactionHandle,    NULL, NULL);

... // work with the file using its file handle hAppend

hr = pITransaction->Commit(FALSE, XACTTC_SYNC_PHASEONE, 0);

If you are a managed developer, the solution isn’t quite as elegant. Currently, to use TxF from managed code, you have to use P/Invoke and COM interop. Don’t let that scare you away, though. For a large number of uses, the P/Invoke and COM interop code isn’t too bad. If you’re still too timid, you can grab some sample code from the MSDN® Magazine Web site. The sample has a simple managed wrapper around Transactional NTFS that you can freely use in your own applications. (While you’re playing with this sample code, you should also check out the screencast. It demonstrates using TxF with just KTM, with SQL, and with Windows Communication Foundation.) In the short term, a more robust version of this wrapper will be included in an upcoming release of the Windows VistaBridge samples as part of the Windows SDK. In the longer term, the product team is currently working on a plan to integrate these APIs directly into the Microsoft .NET Framework.

To use TxF from managed code, you can continue to use the functionality of the System.Transactions namespace. To dig down into TxF, you are essentially following the same path that you would when using DTC in native code (see Figure 4), but with the managed code equivalents. From an object level, it looks something like Figure 5.

Figure 5 TxF from Managed Code
Figure 5 TxF from Managed Code

The first thing you need to do is grab the IDtcTransaction COM interface from the current Transaction (this can be done using the TransactionInterop helper class in System.Transactions). Once you have the IDtcTransaction, you can cast it to an IKernelTransaction COM interface, which under the hood will QueryInterface for the IKernelTransaction COM interface. Then, once you have a reference to the currently running kernel-level transaction via IKernelTransaction, you call GetHandle on it to get a handle that you can pass to the various new Transacted Win32 APIs. For example, you can pass this transaction handle to CreateFileTransacted in order to obtain a SafeFileHandle for a new file. This SafeFileHandle can then be passed to the constructor of the StreamWriter class, which is then used to write to the file (see Figure 6).

Figure 6 TxF from Managed Code with System.Transactions (Close Figure 6)

NativeMethods.cs
    // IKernelTransaction COM Interface
[Guid(&quot;79427A2B-F895-40e0-BE79-B57DC82ED231&quot;)]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IKernelTransaction
{
    int GetHandle(out IntPtr pHandle);
}

[DllImport(KERNEL32,
   EntryPoint = &quot;CreateFileTransacted&quot;,
   CharSet = CharSet.Unicode, SetLastError = true)]
internal static extern SafeFileHandle CreateFileTransacted(
   [In] string lpFileName,
   [In] NativeMethods.FileAccess dwDesiredAccess,
   [In] NativeMethods.FileShare dwShareMode,
   [In] IntPtr lpSecurityAttributes,
   [In] NativeMethods.FileMode dwCreationDisposition,
   [In] int dwFlagsAndAttributes,
   [In] IntPtr hTemplateFile,
   [In] KtmTransactionHandle hTransaction,
   [In] IntPtr pusMiniVersion,
   [In] IntPtr pExtendedParameter);
...

Program.cs
...
using (TransactionScope scope = new TransactionScope())
{
   // Grab Kernel level transaction handle
   IDtcTransaction dtcTransaction =
      TransactionInterop.GetDtcTransaction(managedTransaction);
   IKernelTransaction ktmInterface = (IKernelTransaction)dtcTransaction;

   IntPtr ktmTxHandle;
   ktmInterface.GetHandle(out ktmTxHandle);

   // Grab transacted file handle
   SafeFileHandle hFile = NativeMethods.CreateFileTransacted(
      path, internalAccess, internalShare, IntPtr.Zero,
      internalMode, 0, IntPtr.Zero, ktmTxHandle,
      IntPtr.Zero, IntPtr.Zero);

   ... // Work with file (e.g. passing hFile to StreamWriter constructor)

   // Close handles
}

You should also know that TxF has read-committed consistency. This has impact on both writing and reading a file transactionally. When writing a file, only a single transaction can be writing to the file (note that we are talking about a single transaction, not a single writer). Non-transacted writers are blocked by the transacted writer, even if opened with share-write access. Additionally, a non-transacted writer prevents any transacted reader or writer from being opened on the file.

When reading from a file transactionally, there is an isolated view of the file for the life of the file handle. This isolated view of the file is based on the most recent contents of the file when the file handle was created. If a transacted writer is in the middle of modifying the file, those modifications won’t be picked up by a new transacted reader until the changes are actually committed. In addition, because of the read-committed consistency of Transactional NTFS, this isolated view of the transacted reader remains even if a transacted writer changes a file and commits those changes anytime during the life of the transacted reader. These transacted readers block non-transacted writers but only while their file handle is open. They do not block modification of the file for the life of the transaction. If you need a consistent view of the file throughout the life of the transaction, you need to keep the transacted reader’s file handle open.

Finally, developers should be aware that when using file handles for transacted file operations, there is a distinct difference between transacted and non-transacted file operations with regard to file handles. If you get a file handle from one of the new Transacted APIs, the file handle could become invalid at any time, whether from a transaction committing or a transaction rolling back.

Secondary Resource Managers

Secondary resource managers allow applications to manage their own resources running within a transaction, particularly their own transaction logs, as well as other possible file operations within a directory (and all subdirectories) specific to the application.

I have a good reason for discussing secondary resource managers in this article. When TxF was developed, there was an emphasis placed on availability over consistency. Since it is now possible within a transaction to modify files that are necessary to boot, the operating system has to guarantee that these files are always available. In order to guarantee these files are always available, the system drive’s transaction manager is required to be the transaction coordinator. Because of this, you cannot use a different transaction coordinator, like DTC for instance, as the "root" or superior transaction coordinator for these files. As a result, a transaction that needs to use DTC with other resource managers (such as SQL) cannot use TxF on the system drive. This means that if you want to use Transactional NTFS on the system drive within the same transaction as operations against SQL Server™, for example, you must use a different strategy.

The simplest way to get around this limitation is to place all of your data on a separate partition. If that is not possible, or you cannot guarantee that the data will be on a non-system partition, you can use a secondary resource manager. When using a secondary resource manager to manage transactional file operations for a given directory (and any subdirectories), you are informing KTM that the application will control resource management for any files residing there. This allows the default resource manager on the system volume to continue to be available and, hence, allows a transaction coordinator other than KTM to become the superior transaction coordinator.

At the command line, you can create and manage secondary resource managers using the FSUtil tool and the commands shown in Figure 7. However, for applications that need to use a secondary resource manager, it is better to control the secondary resource manager programmatically. To do this, you can use the DeviceIoControl Win32 API with specific dwIoControlCodes, outlined in Figure 8.

Figure 8 Manage Secondary Resource Managers Programmatically (Close Figure 8)

Control Codes | Purpose
FSCTL_TXFS_LIST_TRANSACTIONS | Lists all transactions currently involved in the specified resource manager.
FSCTL_TXFS_LIST_TRANSACTION_LOCKED_FILES | Lists all files currently locked by the specified transaction.
FSCTL_TXFS_GET_METADATA_INFO | Retrieves Transactional NTFS metadata for a file, as well as the ID of the transaction that has the file locked.
Figure 7 Manage Secondary Resource Managers from the Command Line (Close Figure 7)

Command | Purpose
fsutil resource create | Creates a secondary transactional resource manager.
fsutil resource info | Displays information relating to a transactional resource manager.
fsutil resource setautoreset | Sets whether a transactional resource manager will clean its transactional metadata the next time it starts.
fsutil resource setlog | Changes characteristics of a running transactional resource manager.
fsutil resource start | Starts a transactional resource manager.
fsutil resource stop | Stops a transactional resource manager.

When using secondary resource managers, the startup and recovery of the resource manager must be done by your application. The act of starting the resource manager creates the KTM objects to manage transactions, and opens up the Common Log File System (CLFS) log and metadata that is stored locally with your directories; the act of recovering the resource manager recovers all files to a consistent state within the secondary resource manager. The resource manager will stop when the last handle to the resource manager root goes away. As a result, files within a secondary resource manager are not automatically made consistent at system startup. Instead, that is done when your application starts and recovers the secondary resource manager. In addition, files are not automatically made consistent when the system is backed up with any software that uses the volume shadow copy service (VSS) unless you provide a VSS writer. Of course, if you restore an inconsistent secondary resource manager and then start and recover it, the files will be made consistent. Make sure that your backup software also backs up the TxF metadata using the FSCTL_TXFS_READ_BACKUP_INFORMATION and FSCTL_TXFS_WRITE_BACKUP_INFORMATION dwIoControlCodes.

As I mentioned, the use of secondary resource managers is only required if the files participating in the transaction are on the system drive and you want a technology other than KTM to be the superior transaction coordinator or you want to control the consistency and availability of the resource manager. Secondary resource managers become very powerful in this regard if you wish to have more direct control over the behavior of the resource manager in your application. If the files you are operating on are on a non-system drive, you do not need to worry about the use of secondary resource managers. We are currently working to lessen this restriction in Windows Server "Longhorn" so this requirement won’t be applied to the entire system drive, but only to the parts of the system drive that Windows absolutely needs to boot.

Management

For any application that is going to live in a production environment, it is important to think about how the application will be managed. Therefore, you need to know how you can monitor TxF itself. Figure 9 shows a list of control codes that can be used with DeviceIoControl to help you do just that.

Similar to the information on resource managers provided previously, you can get the same information as these control codes from the FSUtil command-line tool. The commands are as follows:

  • fsutil transaction list
  • fsutil transaction fileinfo
  • fsutil transaction query

What Happened to the Transacted Command Line?

If you were involved in the early betas for Windows Vista, you may recall seeing a transaction command available from the command line. This allowed an administrator to start a transaction from the command line and run an application, and then any file operations within that application would implicitly be transacted.

While testing application compatibility in Windows Vista, the team found that this implicit model was difficult for COM+ and managed developers to use safely. As a result, the model was changed to be opt-in and explicit to avoid this confusion. The downside to this change is that if you wish to use TxF in your app, your code will have to change to call the new APIs.

In Closing

Transactional NTFS is a truly innovative technology. It helps developers create more reliable applications on a more powerful platform. With TxF, developers can have atomic file operations that result in consistent file data, easing the test burden by reducing the number of error conditions to test for, and assisting in multi-user environments by providing isolation via ACID transaction semantics. Consider using TxF when developing apps that target Windows Vista and Windows Server "Longhorn." For more information, see the "Transaction Resources" sidebar.

I would like to thank Dana Groff, Christian Allred, and Jon Cargille for their help with this article.

Transaction Resources

Here’s a list I’ve compiled of resources relating to Transactional NTFS. I’ll be updating this list over time to reflect new resources, so be sure to watch for updates on my blog.

Web Sites:

  • Transactions Programming Forum
  • Transaction Management in Windows
  • Transactional NTFS (TxF)

Blogs:

  • Jim Johnson
  • Florin Lazar
  • Jason Olson

Specific Blog Posts:

  • An MSDTC Proxy Enhancement (MLTM)
  • Integration between System.Transactions and TxF
  • What Is TxF and Why Do I Care?

Media:

  • Transactional Vista: KTM and Friends
  • Vista Transactional File System
  • The Revolutionary Vista TxF Infrastructure
  • Developing Applications for Windows Vista with Transactional NTFS
  • Developer, Meet Server #1—Transactional NTFS
  • Developer, Meet Server #2—Transactional NTFS and WCF

Jason Olson is a Technical Evangelist at Microsoft working on Windows Server "Longhorn" in the Developer and Platform Evangelism division. Outside of work, he is a loving and dedicated husband, hobbyist game developer, and a blogger at his site, managed-world.com. He can be reached at jason.olson@microsoft.com.


Security: Applying Cryptography Using The CNG API In Windows Vista

Security

Applying Cryptography Using The CNG API In Windows Vista

Kenny Kerr
This article discusses:
  • Differences between CryptoAPI and CNG
  • Algorithm providers
  • Random numbers, encryption, signatures, and verification
  • Interop with .NET applications
This article uses the following technologies:
Windows Vista
Code download available at: CNG2007_07.exe (158KB)
Contents
Algorithm Providers
Random Number Generation
Hash Functions
Symmetric Encryption
Asymmetric Encryption
Signatures and Verification
Interop with .NET
Looking Forward

W
indows Vista™ introduces a new cryptography API to replace the old CryptoAPI, which has its roots in the early versions of Windows® NT and Windows 95. Cryptography Next Generation (CNG) is meant to be a long-term replacement for the CryptoAPI, providing substitutes for all of the cryptographic primitives it offered. CNG supports all of the algorithms provided by the CryptoAPI, but goes much further and includes many new algorithms and a much more flexible design, providing developers with greater control over how cryptographic operations are performed and how algorithms work together to perform various operations.

Figure 1 illustrates the design of CNG at a high level. This article focuses on BCrypt, the subset of CNG that provides the cryptographic primitives such as random number generation, hash functions, signatures, and encryption keys. In contrast, NCrypt is the subset of CNG that provides key storage facilities to support persisting asymmetric keys and hardware such as smart cards. Don’t read too much into the naming of BCrypt and NCrypt. BCrypt is simply the name of the header file and DLL that provides the base services for CNG. "B" stands for "base" in this case. Similarly, NCrypt is simply the name of the header file and DLL that provides the higher-level key storage functionality. "N" stands for "new."

Figure 1 CNG Architecture
Figure 1 CNG Architecture

The cryptographic primitives provided by BCrypt are available directly in kernel mode, providing for the first time a common cryptographic framework for both user-mode and kernel-mode applications. The key storage facilities provided by NCrypt, however, are only available to user-mode applications.

There are two main ways to view the cryptographic primitives provided by CNG. The first is as a set of logical objects that provide methods you may call and properties you may query and sometimes modify. These objects include algorithm providers, hash functions, keys, and secret agreements. Where do random number generators, signatures, and different types of keys come in? As it turns out, random number generation is handled directly by algorithm providers, and keys provide almost everything else. They can represent symmetric or asymmetric keys, and they are used to sign and verify hash signatures. Hash and key objects are derived from algorithm providers and secret agreements are derived from a pair of key objects—the public key of a principal and the private key of another principal wishing to communicate securely.

The other way to look at CNG is as a software router or intermediary for cryptographic operations. The CNG API is built on a set of logical cryptographic interfaces. As an example, you can use the hash interface without hard-coding a particular hash algorithm implementation. Most cryptography APIs take an algorithm-centric approach, whereas CNG takes an interface-centric approach. This provides greater agility both for the developer and for the application administrator who might need to replace an algorithm used by an application that is found to be flawed after that application has shipped. Figure 2 illustrates this interface-centric view of CNG.

Figure 2 CNG Interfaces
Figure 2 CNG Interfaces(Click the image for a smaller view)
Figure 2 CNG Interfaces
Figure 2 CNG Interfaces (Click the image for a larger view)

The CNG configuration system deserves an article of its own, but for now it is sufficient to know that CNG allows developers to request algorithms without having to specify algorithm providers. The providers are responsible for the algorithm implementation, and it is this aspect of CNG that allows administrators to reconfigure applications to use different implementations either declaratively in an application or through system policy.

Algorithm Providers

With that introduction out of the way, let’s see how we can use CNG to perform various common cryptographic operations. The first thing you need is an algorithm provider. All CNG objects defined by BCrypt are identified by a BCRYPT_HANDLE, and algorithm providers are no exception. The BCryptOpenAlgorithmProvider function loads an algorithm provider based on your choice of algorithm and optional implementation and returns a handle for use in subsequent calls to CNG functions. BCrypt also adopts the NTSTATUS type from the Windows Driver Kit (WDK) for indicating error information regardless of whether you are coding in user mode or kernel mode. Here’s how you load an algorithm provider into memory:

BCRYPT_HANDLE algorithmProvider = 0;

NTSTATUS status = ::BCryptOpenAlgorithmProvider(
                      &algorithmProvider, algorithmName,
                      implementation, flags);

if (NT_SUCCESS(status))
{
    // Use algorithm provider
}

In most cases you will pass zero for both the implementation and flags parameters. Passing zero for implementation indicates that the default algorithm provider should be loaded for the particular algorithm identified by the algorithmName parameter. The NT_SUCCESS macro should be familiar to those who have used the WDK in the past. It is a macro similar to SUCCEEDED used by COM developers and indicates whether the status value represents success or failure.

When you are finished with the algorithm provider, you must unload it by passing the handle returned by BCryptOpenAlgorithmProvider to the BCryptCloseAlgorithmProvider function as you see here:

  status = ::BCryptCloseAlgorithmProvider(
    algorithmProvider, flags);
  ASSERT(NT_SUCCESS(status));

No flags are currently defined for this function so you must pass zero for the flags parameter. Naturally, the assertion is not required, but is useful in weeding out bugs in your application.

The remainder of this article uses the NT_VERIFY macro to verify the result of CNG functions. This is just a placeholder to remind you that you need to check the status code that gets returned. If you are following along you can define the macros as follows:

#define NT_VERIFY(x) ASSERT(NT_SUCCESS(x))

Just be sure to use error handling that conforms to your project’s error handling strategy in production code.

Loading algorithm providers can be an expensive operation, so it is wise to consider the usage pattern of the providers in your application and reuse them as much as is reasonable. The remainder of this article makes use of algorithm providers to achieve various goals, so hold on to your handles!

Random Number Generation

Random number generation is provided courtesy of the BCryptGenRandom function. It could easily have been called random buffer generation as it allows you to fill any buffer with a random value. Let’s say you need to generate 10 random numbers. Figure 3 shows how you might accomplish that.

Figure 3 Random Numbers (Close Figure 3)
BCRYPT_HANDLE algorithmProvider = 0;

NT_VERIFY(::BCryptOpenAlgorithmProvider(
    &algorithmProvider,
    BCRYPT_RNG_ALGORITHM,
    0,                     // implementation,
    0));                   // flags

for (int i = 0; i < 10; ++i)
{
    UINT random = 0;

    NT_VERIFY(::BCryptGenRandom(
        algorithmProvider,
        reinterpret_cast<PUCHAR>(&random),
        sizeof(UINT),
        0));

    cout << random << endl;
}

In this example, I requested the default random number generation algorithm by specifying the BCRYPT_RNG_ALGORITHM algorithm identifier. The BCRYPT_RNG_FIPS186_DSA_ALGORITHM algorithm identifier is also available should you need an algorithm that meets Federal Information Processing Standards (FIPS) for use with the Digital Signature Algorithm (DSA).

As you can see from the example in Figure 3, the BCryptGenRandom function expects a pointer to a UCHAR to identify the buffer instead of the more practical pointer to void (any type). Fortunately, this is easily solved in a type-safe way with a little help from C++. Take a look at the following function template:

template <typename T>
__checkReturn
NTSTATUS GenRandom(BCRYPT_HANDLE algorithmProvider,
                   T& buffer, ULONG flags)
{
    ASSERT(0 != algorithmProvider);

    return ::BCryptGenRandom(algorithmProvider,
                             reinterpret_cast<PUCHAR>(&buffer),
                             sizeof(T), flags);
}

If you’re not familiar with Standard Annotation Language (SAL), the __checkReturn macro is simply a hint to analysis tools indicating that callers of the function should be checking the result of the function.

Although this function template is not suitable in all cases (such as if you wish to populate a buffer wrapped in a collection class), it does make generating a random number much more convenient in most cases as the casting is isolated within the function template. The template will also determine the size of the buffer automatically. Consider the following example of generating 10 random GUID values:

for (int i = 0; i < 10; ++i)
{
    GUID random = { 0 };

    NT_VERIFY(GenRandom(algorithmProvider, random, 0));

    // Use ‘random’ value
}

The BCryptGenRandom function supports an optional flag that allows you to provide additional entropy for the random number generation algorithm. The BCRYPT_RNG_USE_ENTROPY_IN_BUFFER flag indicates that the algorithm should use the value in the buffer passed to the function as additional entropy in calculating the random number that is then returned in the same buffer. Here is an example:

UINT random = 0;

for (int i = 0; i < 10; ++i)
{
    NT_VERIFY(GenRandom(algorithmProvider, random,
                        BCRYPT_RNG_USE_ENTROPY_IN_BUFFER));

    // Use ‘random’ value
}

Notice that I moved the definition of the random variable outside of the loop so that the value of the previous random number provides additional entropy for the next generated number.

Hash Functions

Like random number generation, hash functions play a critical role in enabling security measures and features in many aspects of modern computing. Hash functions are exposed as objects in CNG, but because the API needs to be accessible from kernel-mode code (which is mostly written in C), it takes a bit of up-front work to put the object back together. As I mentioned in the introduction, algorithm providers and hash functions are represented by objects in CNG. Since these are part of BCrypt, you can employ the BCryptGetProperty and BCryptSetProperty functions to query and set named properties for a particular object. Matching functions are provided for handling object properties with NCrypt.

The BCryptCreateHash function creates a new hash object. Before you call it, however, you need to have allocated a buffer that the hash function will use for processing. Here you can see one of the side effects of supporting both user mode and kernel mode in a single API—kernel mode is very sensitive regarding where memory is allocated and in particular whether the allocation is from paged or non-paged memory. Aside from the buffer, you are responsible for managing the handle and hash table resources for the hash object.

The size of the buffer required to create the hash object is a property of the algorithm provider and the BCRYPT_OBJECT_LENGTH identifier gives you the name of the property to query. Figure 4 shows how you might load the algorithm provider for the SHA-256 hashing algorithm and query the hash buffer size.

Figure 4 Using an Algorithm Provider (Close Figure 4)
BCRYPT_HANDLE algorithmProvider = 0;

NT_VERIFY(::BCryptOpenAlgorithmProvider(
    &algorithmProvider,
    BCRYPT_SHA256_ALGORITHM,
    0,                      // implementation,
    0));                    // flags

ULONG hashBufferSize = 0;
ULONG bytesCopied = 0;

NT_VERIFY(::BCryptGetProperty(
    algorithmProvider,
    BCRYPT_OBJECT_LENGTH,
    reinterpret_cast<PUCHAR>(&hashBufferSize),
    sizeof(ULONG),
    &bytesCopied, 0));

ASSERT(sizeof(ULONG) == bytesCopied);

BCryptGetProperty’s first parameter indicates the object to query. The second parameter indicates the name of the property. The third and fourth identify the destination buffer where the property value is stored as well as the size of the buffer. The bytesCopied parameter is useful in cases where you do not know beforehand what the expected buffer size is. No flags are currently defined for this function so you must pass zero for the flags parameter.

Having determined the size of the hash buffer, you need to commit a chunk of memory for the hash object. With the exception of kernel mode, it does not matter too much where you allocate this buffer, and you are free to use whatever storage is most appropriate as long as it is stable (in other words, you cannot use an unpinned managed byte array). Figure 5 provides a straightforward Buffer class that you can use for simple user-mode buffer allocations. This class will be used again when creating encryption key objects so it is worthwhile to have a Buffer class of some kind on hand.

With all of the prerequisites out of the way, you can now finally create the hash object with the BCryptCreateHash function as shown here:

Buffer hashBuffer;
NT_VERIFY(hashBuffer.Create(hashBufferSize));

BCRYPT_HANDLE hash = 0;

NT_VERIFY(::BCryptCreateHash(algorithmProvider, &hash,
                             hashBuffer.GetData(),
                             hashBuffer.GetSize(),
                             0,   // secret
                             0,   // secret size
                             0)); // flags

BCryptCreateHash’s first parameter indicates the algorithm provider that implements the hash interface. The second parameter receives the handle to the hash object. The third and fourth parameters identify the hash buffer and its size. Keyed hash algorithms use the secret parameters to identify the secret key. No flags are currently defined for this function so you must pass zero for the flags parameter. When you are finished with the hash object, you must destroy it using the BCryptDestroyHash function and then free the hash object buffer.

Now that you’ve seen how to create and destroy hash objects, let’s actually do something useful with them. After a hash object is created, you can call the BCryptHashData function to perform a one-way hash of a buffer. This function can be called repeatedly to combine additional buffers into the hash. Keep in mind that the size of a hash value is fixed and determined by the algorithm provider so no matter how many times you call BCryptHashData, the resulting hash value will always be the same size.

This example illustrates how to hash data read from a stream using the COM IStream interface:

BYTE buffer[256] = { 0 };
ULONG bytesRead = 0;

while (SUCCEEDED(stream->Read(buffer, _countof(buffer),
                              &bytesRead)) && 0 < bytesRead)
{
    NT_VERIFY(::BCryptHashData(hash, buffer, bytesRead, 0));
}

BCryptHashData’s first parameter is the handle returned from the BCryptCreateHash function. The second and third parameters indicate the buffer containing the data to hash as well as the size of the buffer. Notice that I am careful to pass bytesRead for the size of the buffer to ensure that I include only data actually read from the stream. Finally, no flags are yet defined for BCryptHashData so I must pass zero for the last parameter.

Once you pass all the data to the hash function, you can retrieve the resulting hash value by calling the BCryptFinishHash function. The size of the resulting hash value is dependent on the particular hash algorithm in use. If you do not know this size, you can query the hash object with the BCRYPT_HASH_LENGTH identifier. In the following example, I query the hash value size, create a buffer for it, and finally copy the hash value into the buffer using BCryptFinishHash:

ULONG hashValueSize = 0;
NT_VERIFY(::BCryptGetProperty(hash, BCRYPT_HASH_LENGTH, ...))

Buffer hashValue;
NT_VERIFY(hashValue.Create(hashValueSize));

NT_VERIFY(::BCryptFinishHash(hash,
                             hashValue.GetData(),
                             hashValue.GetSize(),
                             0)); // flags

The pattern is probably becoming pretty familiar to you. BCryptFinishHash’s first parameter is the hash object handle. The second and third parameters indicate the buffer that will receive the hash value as well as the buffer’s size. The last parameter indicates the flags, of which there are none at present.

Finally, hash objects can be duplicated. This is useful in the case where you want to produce two or more hash values based on some common data. You can start by creating a single hash object to hash the common data and then duplicate it one or more times, adding any deltas to the duplicates. Once a hash object has been duplicated, the two hash objects contain the same state but have no connection to one another and you are free to add unique data to each one, produce a hash value, or destroy one of the hash objects without affecting the other in any way.

The BCryptDuplicateHash function takes care of duplicating a hash object. You need to simply provide a handle to the hash object to duplicate as well as a new buffer that it will use for processing:

Buffer newHashBuffer;
NT_VERIFY(newHashBuffer.Create(hashBufferSize));
BCRYPT_HANDLE newHash = 0;

NT_VERIFY(::BCryptDuplicateHash(hash,
                                &newHash,
                                newHashBuffer.GetData(),
                                newHashBuffer.GetSize(),
                                0));

Symmetric Encryption

Symmetric encryption begins in much the same way as hash functions in that you need to create and prepare the symmetric key and its object buffer. The steps should be familiar: create an algorithm provider, determine the size of the key object buffer by querying the provider’s BCRYPT_OBJECT_LENGTH property, and allocate a buffer of that size. A call to the BCryptGenerateSymmetricKey function with a secret to initialize it creates the symmetric key. Figure 6 illustrates this process.

Figure 6 Creating a Symmetric Key Object
Figure 6 Creating a Symmetric Key Object

As I’ve already covered opening algorithm providers and creating buffers in previous sections, I’ll skip ahead and focus here on generating the symmetric key. This code illustrates a typical call to BCryptGenerateSymmetricKey:

BCRYPT_KEY_HANDLE key = 0;

NT_VERIFY(::BCryptGenerateSymmetricKey(algorithmProvider,
                                       &key,
                                       keyBuffer.GetData(),
                                       keyBuffer.GetSize(),
                                       secret.GetData(),
                                       secret.GetSize(),
                                       0)); // flags

The first parameter indicates the algorithm provider that implements a symmetric encryption algorithm. The second parameter receives the handle to the key object. The third and fourth parameters identify the key buffer and its size. The next two parameters indicate a buffer containing the secret key shared by the sender and receiver. This can be any byte array—and can even be empty—but is typically a hash of a password of some kind. No flags are currently defined for this function so you must pass zero for the flags parameter.

When you are finished with the key object, you must destroy it using the BCryptDestroyKey function and then free the key object buffer.

When you’ve created a key object, chances are you want to actually encrypt and decrypt some data, so let’s get to it. The aptly named BCryptEncrypt and BCryptDecrypt functions are used to encrypt and decrypt data for both symmetric and asymmetric keys. With the exception of padding schemes and initialization vectors, the functions are used in the same way regardless of the type of key in use. These functions may seem intimidating at first due to their long parameter list, but just keep in mind that they accept a set of common parameters that must match in order for data to be successfully decrypted. The sender and receiver in a symmetric encryption operation must share a few common properties. They need a key created with the same secret and matching property values. They need initialization vectors that are equal. And they need the same padding.

Symmetric encryption is surprisingly simple to use—the trouble comes when you consider sharing all these properties between parties. The first piece of information you need to determine is the size of a block of data for the algorithm:

ULONG blockSize = 0;
NT_VERIFY(::BCryptGetProperty(key, BCRYPT_BLOCK_LENGTH, ...))

The block size is useful for a number of reasons when working with block cipher algorithms, the most common form of symmetric algorithm. Block ciphers encrypt a fixed sized block of plaintext into a block of ciphertext of the same size. The block size also indicates the size of the initialization vector if used.

Once you have prepared the message to encrypt as well as the initialization vector, you can call the BCryptEncrypt function to encrypt the plaintext message. Of course you will need to determine the size of the buffer that will receive the ciphertext. To accomplish this you simply call BCryptEncrypt and pass zero for both the output and output size parameters. BCryptEncrypt will then return the necessary size.

ULONG ciphertextSize = 0;

NT_VERIFY(::BCryptEncrypt(key,
                          message.GetData(),
                          message.GetSize(),
                          0,   // padding info
                          iv.GetData(),
                          iv.GetSize(),
                          0,   // output
                          0,   // output size,
                          &ciphertextSize,
                          0)); // flags

The first parameter indicates the key to use for encryption. The second and third parameters identify the message to encrypt. The fourth parameter provides additional padding information for asymmetric encryption. It is not used by symmetric algorithms so you must pass zero for this parameter. The next two parameters indicate a buffer containing the initialization vector to use. The second-to-last parameter receives the size of the expected ciphertext and the last parameter accepts optional parameters. Only the BCRYPT_BLOCK_PADDING flag is used with symmetric algorithms and is handy when you don’t want to worry about padding a particular message to a multiple of the block size.

You can then create the ciphertext buffer and call BCryptEncrypt again, this time providing the buffer to receive the ciphertext:

Buffer ciphertext;
NT_VERIFY(ciphertext.Create(ciphertextSize));

NT_VERIFY(::BCryptEncrypt(key,
                          message.GetData(),
                          message.GetSize(),
                          0,   // padding info
                          iv.GetData(),
                          iv.GetSize(),
                          ciphertext.GetData(),
                          ciphertext.GetSize(),
                          &ciphertextSize,
                          0)); // flags

The decryption process is much the same. If you don’t know the size of the plaintext, you call the BCryptDecrypt function in much the same way as illustrated previously for determining the size of the ciphertext. Finally, you call BCryptDecrypt with the ciphertext, initialization vector, and the plaintext buffer to receive the resulting decrypted message:

Buffer plaintext;
NT_VERIFY(plaintext.Create(plaintextSize));

NT_VERIFY(::BCryptDecrypt(newKey,
                          ciphertext.GetData(),
                          ciphertext.GetSize(),
                          0,   // padding info
                          iv.GetData(),
                          iv.GetSize(),
                          plaintext.GetData(),
                          plaintext.GetSize(),
                          &plaintextSize,
                          0)); // flags

Be sure to specify the same flags when decrypting a message as you did when encrypting it.

Asymmetric Encryption

Asymmetric encryption solves the problems related to sharing secrets and initialization vectors by taking a public-key approach whereby both parties maintain a private and public key. The public keys are made available to anyone. The keys are related such that only the private key can be used to decipher data encrypted with the public key. Naturally, this additional power comes at a cost and asymmetric encryption turns out to be considerably more computationally expensive when compared to symmetric encryption. It is, nevertheless, invaluable if only to establish communication by providing a mechanism for parties to securely share symmetric encryption key information. Figure 7 illustrates the process by which asymmetric keys are created.

Figure 7 Creating an Asymmetric Key
Figure 7 Creating an Asymmetric Key

The following example illustrates calling the BCryptGenerateKeyPair function to generate a public and private key pair:

NT_VERIFY(::BCryptGenerateKeyPair(algorithmProvider,
                                  &key, keySize,
                                  0)); // flags

The first parameter indicates the algorithm provider that implements the asymmetric encryption algorithm. The second parameter receives the handle to the key object. The third parameter indicates the key size the algorithm should use. This value is expressed in bits, not bytes, and varies from one algorithm to another. For example, the RSA algorithm supports key sizes that are multiples of 64 between 512 and 16384 bits, inclusive. Generally speaking, the size of the key directly affects the performance of the algorithm, but more importantly affects the cost of defeating the encryption using a brute force attack. The key size is also important for another, less ominous reason: it indicates the block size that the algorithm will use. To determine the block size, simply divide the key size by 8.

Once you have generated the key pair with BCryptGenerateKeyPair, you may need to set various algorithm-specific key properties using the BCryptSetProperty function. Before you can use the key pair, however, you need to call the BCryptFinalizeKeyPair function to finalize the creation of the key object, as follows:

   NT_VERIFY(::BCryptFinalizeKeyPair(key, 0));

At this point, you can happily encrypt and decrypt data using the BCryptEncrypt and BCryptDecrypt functions discussed previously. About the only thing that’s slightly different in this case is that asymmetric keys ignore the initialization vector, and the padding scheme may be a bit more complex depending on which flags you choose. Assuming the message to be encrypted is provided in a buffer that is a multiple of the block size, you simply specify the BCRYPT_PAD_NONE flag as no padding is necessary. On the other hand, if you want to encrypt a message without having to first pad it to a multiple of the block size, then you have a few options. The simplest option is to specify the BCRYPT_PAD_PKCS1 flag, which tells the algorithm provider to pad the input buffer to a multiple of the block size using a random number based on the PKCS-1 standard.

Of course, the usefulness of asymmetric encryption is largely defeated if you can’t share the public portion of a key pair, so let’s take a look at how to do this. The BCryptExportKey and BCryptImportKeyPair functions can be used to export and import keys. Incidentally, BCryptExportKey can also be used to export symmetric keys, but you need to employ the BCryptImportKey function to import them since it needs to associate a buffer with the key object. BCryptExportKey and BCryptImportKeyPair can be used to export either the complete public and private key pair or just the public key. You might, for example, export the key pair and persist it for your own use at a later time, and export the public key separately to offer to those wanting to communicate with you.

The code in Figure 8 illustrates how to create a binary blob containing only the public key. As you can see, BCryptExportKey follows the same pattern we’ve seen before in that you first call it to determine the size of the blob and then you call it again to actually receive the public key information. The function’s third parameter is notable since it indicates both the algorithm and what exactly will be exported.

Figure 8 Creating a Public Key Binary Blob (Close Figure 8)
ULONG publicKeyBlobSize = 0;

NT_VERIFY(::BCryptExportKey(key,
                            0,                      // reserved
                            BCRYPT_RSAPUBLIC_BLOB,
                            0,                      // output,
                            0,                      // output size
                            &publicKeyBlobSize,
                            0));                    // flags

Buffer publicKeyBlob;
NT_VERIFY(publicKeyBlob.Create(publicKeyBlobSize));

NT_VERIFY(::BCryptExportKey(key,
                            0,                      // reserved
                            BCRYPT_RSAPUBLIC_BLOB,
                            publicKeyBlob.GetData(),
                            publicKeyBlob.GetSize(),
                            &publicKeyBlobSize,
                            0));                    // flags

In the example, I used BCRYPT_RSAPUBLIC_BLOB to indicate that I want to export only the public key portion of an RSA key pair. If you wanted to export both public and private keys, you simply use BCRYPT_RSAPRIVATE_BLOB instead. Don’t let the name fool you—it contains both the private and the public key. This is important to grasp since only the public key is used for encryption, so if you exported the private key and then imported it, you’d be able to encrypt a message, but it would use the public key and not the private key for encryption. The private key is only used for decryption.

Importing a key from a blob is straightforward:

BCRYPT_HANDLE publicKey = 0;

NT_VERIFY(::BCryptImportKeyPair(algorithmProvider,
                                0,   // reserved
                                BCRYPT_RSAPUBLIC_BLOB,
                                &publicKey,
                                publicKeyBlob.GetData(),
                                publicKeyBlob.GetSize(),
                                0)); // flags

The third parameter is again notable as it indicates the anatomy of the blob that is being imported. Just make sure the algorithm provider implements the algorithm being imported and it should work just fine. The resulting public key handle can then safely be used to encrypt messages that only the holder of the private key can decipher.

Signatures and Verification

A common use of asymmetric cryptography is the creation of digital signatures. They are used heavily by Windows, as well as by the Microsoft® .NET Framework, to certify the authenticity of messages, executables, and assemblies.

Unlike the asymmetric encryption and decryption process described already, signatures are created using a private key and verified using a public key. This allows someone holding a public key to verify that a given signature was generated by the holder of a private key without needing access to that private key.

The process of creating or verifying a signature is pretty straightforward given the groundwork already covered. Start by identifying the data you want to sign and then hash the data as described in the section on hash functions. The signature is then calculated by the BCryptSignHash function based on the hash value as well as a private key for a digital signature algorithm. Figure 9 provides a simple example.

Figure 9 Calculating a Signature (Close Figure 9)
ULONG signatureSize = 0;

NT_VERIFY(::BCryptSignHash(keyPair,
                           0,                  // padding info
                           hashValue.GetData(),
                           hashValue.GetSize(),
                           0,                  // output
                           0,                  // output size
                           &signatureSize,
                           0));                // flags

Buffer signature;
NT_VERIFY(signature.Create(signatureSize));

NT_VERIFY(::BCryptSignHash(keyPair,
                           0,                  // padding info
                           hashValue.GetData(),
                           hashValue.GetSize(),
                           signature.GetData(),
                           signature.GetSize(),
                           &signatureSize,
                           0));                // flags

As usual, the first call calculates the size of the resulting signature and the second actually computes the signature. Depending on the size of the hash value and the algorithm being used, you may need to provide additional padding information.

Verifying the signature is even simpler. The idea is to compute the hash independently and then pass this hash value, the public key of the signer, and the signature you received to the BCryptVerifySignature function for verification. BCryptVerifySignature returns STATUS_SUCCESS if the signature matches the hash value and STATUS_INVALID_SIGNATURE if it does not. Figure 10 gives you an idea what it might look like.

Figure 10 Verifying a Signature (Close Figure 10)
NTSTATUS status = ::BCryptVerifySignature(publicKey,
                                          0,              // padding info
                                          hashValue.GetData(),
                                          hashValue.GetSize(),
                                          signature.GetData(),
                                          signature.GetSize(),
                                          0);             // flags
switch (status)
{
    case STATUS_SUCCESS:
    {
        // The signature matches the hash.
        break;
    }
    case STATUS_INVALID_SIGNATURE:
    {
        // The signature does not match the hash.
        break;
    }
    default:
    {
        // An error occurred.
    }
}

Interop with .NET

The cryptography libraries included with the .NET Framework 2.0 and 3.0 provide a variety of managed algorithm implementations as well as wrappers around native implementations from the CryptoAPI. In many cases you will be able to interoperate with CNG data without any trouble, though in some cases the .NET Framework cryptography classes use inappropriate default algorithm properties that may force you to do some troubleshooting. The trick is to match every single key property between CNG and its managed equivalent, and then it should work just fine.

In some ways, working with CNG is quite a bit simpler than working with the managed cryptography classes as there are fewer abstractions and inherited defaults to think about. Effective cryptography relies on programmers being explicit about every single detail.

For symmetric encryption, keep the following properties of the .NET Framework SymmetricAlgorithm class in mind as you prepare to exchange ciphertext between CNG and the .NET Framework: BlockSize, IV, Key, KeySize, Mode, and Padding. Getting just one of these slightly wrong will lead to failures. Although the defaults used by CNG and the .NET Framework are not always the same, you can usually find common values for things like block sizes and padding schemes that are compatible across implementations.

Getting asymmetric encryption working across implementations is a bit more work as different algorithms employ vastly different properties, so finding a common list of properties to match up is not going to do it. Instead, you need to focus on how the different implementations import and export key information and provide the necessary translation yourself. Let’s look at a concrete example to illustrate the process of communicating a public key across implementations.

The RSA algorithm accepts the following general parameters: a public and private exponent, a modulus, a pair of prime numbers, another pair of exponents, and a coefficient. Although this may be interesting to cryptographers and mathematicians, most developers just want to know how to share the public portion of the key pair and, in the case of the RSA algorithm, you simply need to share the public exponent and the modulus values.

CNG provides the BCryptExportKey function to assist in exporting an encryption key to a binary blob that can be persisted or shared as necessary. Note that it follows the same pattern you’ve seen over and over where you need to call it a first time to determine the size of the blob and then again with the newly allocated buffer ready to receive the exported key information. The only thing that is worth mentioning is the third parameter, shown here:

NT_VERIFY(::BCryptExportKey(key,
                            0,   // reserved
                            BCRYPT_RSAPUBLIC_BLOB,
                            blob.GetData(),
                            blob.GetSize(),
                            &blobSize,
                            0)); // flags

This parameter indicates the blob type to produce. In this example, the BCRYPT_RSAPUBLIC_BLOB identifier indicates that the function should export only the public portion of the RSA key.

Now comes the fun part. You need to crack open that blob to find out what’s inside of it since the .NET Framework RSA classes use a different format for importing key information. The blob is actually a simple format starting with the BCRYPT_RSAKEY_BLOB structure as a header. This header indicates the type of blob as well as the extents of the various key parameters. Figure 11 illustrates the memory layout of a public key blob.

Figure 11 Memory Layout for Public Key Blob
Figure 11 Memory Layout for Public Key Blob

With this information in hand, it’s simply a matter of copying the necessary memory ranges and populating an RSAParameters structure, which is what the .NET Framework RSA classes expect when importing key information.

As you can see in Figure 12, I have used C++/CLI to directly convert a native RSA key blob into a managed RSAParameters structure. The structure’s Exponent and Modulus fields are assigned newly created managed byte arrays and then the values are copied directly into those byte arrays by first pinning them and then using the memcpy_s function. Suppose you had received the binary blob in C#, for example, you could have achieved the same results using the .NET Framework BinaryReader class. The rest of the process is straightforward:

Figure 12 Converting RSA Key Blob to .NET Structure (Close Figure 12)
BCRYPT_RSAKEY_BLOB* rsaBlob =
    reinterpret_cast<BCRYPT_RSAKEY_BLOB*>(blob.GetData());
RSAParameters rsaParams;

rsaParams.Exponent = gcnew array<BYTE>(rsaBlob->cbPublicExp);
{
    pin_ptr<BYTE> destination = &rsaParams.Exponent[0];
    const BYTE* source = reinterpret_cast<BYTE*>(rsaBlob);

    memcpy_s(destination,
             rsaParams.Exponent->Length,
             source + sizeof(BCRYPT_RSAKEY_BLOB),
             rsaBlob->cbPublicExp);
}

rsaParams.Modulus = gcnew array<BYTE>(rsaBlob->cbModulus);
{
    pin_ptr<BYTE> destination = &rsaParams.Modulus[0];
    const BYTE* source = reinterpret_cast<BYTE*>(rsaBlob);

    memcpy_s(destination,
             rsaParams.Modulus->Length,
             source + sizeof(BCRYPT_RSAKEY_BLOB) + rsaBlob->cbPublicExp,
             rsaBlob->cbModulus);
}

RSACryptoServiceProvider rsa;
rsa.ImportParameters(rsaParams);
array<BYTE>^ ciphertext = rsa.Encrypt(plaintext, false);

The RSACryptoServiceProvider class is the .NET Framework RSA algorithm implementation that is based on the CryptoAPI. The ImportParameters method imports the RSAParameters structure we prepared and the Encrypt method goes ahead and uses the public key to encrypt a message that can be decrypted only by using the private key.

Looking Forward

As I write this, Microsoft is hard at work on the next version of Visual Studio, code-named "Orcas," and along with that release we will see a new version of the .NET Framework. The .NET Framework 3.5 introduces a number of new algorithm implementations that are based on CNG. Once the .NET Framework 3.5 has been released, you can expect even greater interoperability between native and managed code, especially in the area of cryptography. These new additions to the framework provide drop-in replacements for many of the existing implementations as CNG becomes the core cryptographic provider for the Windows platform. Some brand-new algorithm implementations never before seen in the .NET Framework are also being introduced; they cover some of the elliptical curve algorithms provided through CNG’s key storage functions. These implementations fall under the subset of CNG known as NCrypt that I alluded to in the introduction to this article. For more information on all of this, see the "Cryptographic Next Generation Resources" sidebar.

Cryptographic Next Generation Resources

  • Microsoft Windows Cryptographic Next Generation Software Development Kit for Windows Vista
    Documentation, code, and tools to help you develop cryptographic applications and libraries targeting Windows Vista.
  • Cryptography API: Next Generation
    Documents and references to help you work with CNG.
  • Channel 9: Tomas Palmer and Tolga Acar—Crypto in Windows Vista
    Tomas Palmer and Tolga Acar sit down with Joe Smith for an interesting interview.
  • MSDN Virtual Labs: Security
    Learn about writing secure code.
  • TechNet Virtual Labs: Windows Vista
    Explore the improved security, management, and productivity in Windows Vista.

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. Reach Kenny at weblogs.asp.net/kennykerr.


Speak Up: Support Dictation With Text Services Framework

Speak Up

Support Dictation With Text Services Framework

Eric Brown
This article discusses:
  • Windows Speech Recognition and text services
  • Making apps TSF-aware
  • Text stores and document locks
  • Creating a TSF-aware control
This article uses the following technologies:
Windows Vista
Code download available at: Dictation2007_07.exe (1560KB)
Contents
Making Apps TSF-Aware
Text Stores and Document Locks
Connecting to TSF
Implementation Details
Interaction between Scintilla and TSF
Advanced Topics
Conclusion

O
ne of my favorite new features in Windows Vista™ is Windows® Speech Recognition, which allows you to operate your computer using only your voice, including dictating text into e-mail messages or other documents. Windows Speech Recognition uses the Text Services Framework (TSF) to insert, select, and correct dictated text. TSF is a scalable framework for the delivery of advanced text input technologies. It provides a standardized method for text services—such as voice recognition, handwriting recognition, spell checkers, and Japanese Input Method Editors—to communicate with applications and text controls. In particular, TSF allows bidirectional communication between applications and text services. This means that text services can read and write to an application’s document and an application can ask a text service to perform actions such as correcting text.

Windows Vista provides TSF support for Win32® standard edit controls, RichEdit controls, HTML editor controls, Windows Forms TextBox and RichTextBox controls, and Windows Presentation Framework TextBox or RichTextBox controls. The edit controls inside Win32 and Windows Forms ComboBoxes, ListBoxes, ListView controls, and TreeView controls are also natively supported. Many Microsoft applications, such as Microsoft® Word and Publisher, also support TSF.

If an application or text control does not fully support TSF, Windows Speech Recognition can still support dictation via its Dictation Everywhere functionality. Dictation Everywhere does not require any assistance from your application or controls, but it can only insert text; it cannot select or correct text after the text has been inserted. To avoid inserting incorrect text, the user must verify that every single dictated utterance is correct.

TSF is responsible for managing keyboard layouts, displaying the language bar (used to switch keyboard layouts), managing and switching the active Input Method Editor, and most importantly for this discussion, providing an abstraction layer that allows text services continuous access to the application’s document text.

In order for text services to read the application’s document text, however, the application or control must implement a COM interface. Figure 1 shows the relationship between TSF, text services, and your application or control.

Figure 1 TSF and Text Services
Figure 1 TSF and Text Services(Click the image for a smaller view)
Figure 1 TSF and Text Services
Figure 1 TSF and Text Services (Click the image for a larger view)

TSF consists of over 100 interfaces. Luckily, you really only need to use 4 of these—ITfThreadMgr, ITfDocumentMgr, ITfContext, and ITextStoreACPSink—to support dictation, and the only interface your control must implement is ITextStoreACP. Figure 2 provides more detail on these interfaces.

Figure 2 Dictation-Related TSF Interfaces (Close Figure 2)

Interface Name | Who Implements It? | What Does It Do?
ITfThreadMgr | Text Services Framework | Also known as the thread manager, this is the primary TSF interface. It must be created by all TSF-aware applications or controls. It manages focus and creates ITfDocumentMgr objects.
ITfDocumentMgr | Text Services Framework | Also known as the document manager, this interface manages a stack of one or (sometimes) two ITfContext objects.
ITfContext | Text Services Framework | This interface contains an instance of ITextStoreACP and manages the interactions between text services and text stores.
ITextStoreACPSink | Text Services Framework | Also known as the text store event handler, this interface handles change notifications from your application.
ITextStoreACP | Developer | Also known as the text store, this interface gets document text and properties, and sets document text.

Making Apps TSF-Aware

You make your application TSF-aware either by using only TSF-aware text controls or by integrating TSF support directly into it. In most cases, you would only add direct TSF support if your application is directly responsible for displaying and editing text (such as Microsoft Publisher or Microsoft Word). If your app uses a control to display and edit the text, then you can implement TSF support in the control (if that control doesn’t already support it).

If your application uses only unmodified (or subclassed) plain edit controls, RichEdit controls, HTML editor controls, or is based on Windows Presentation Foundation, your application is already TSF-aware and dictation will work without modification to your application. If your application uses other controls, it may be possible to make these controls TSF-aware with very little effort.

If the control superclasses a Rich Edit control and it uses Rich Edit 4.1 or higher, call the following with your control’s window handle after it’s created:

SendMessage(hwnd, EM_SETEDITSTYLE, SES_USECTF, SES_USECTF);

If the control superclasses a plain edit control, or it superclasses a Rich Edit control that isn’t based on Rich Edit 4.1 or higher, you can identify your control as being compatible by adding a DWORD value to the following registry entry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CTF\KnownClasses\<RealClassName> = <flags>

<RealClassName> is the class name for your control, and <flags> can be either 1 for controls that are superclassed plain edit controls or 2 for controls that are superclassed Rich Edit controls.

If your application uses a text control that isn’t based on a Rich Edit or plain edit control, or if your application is responsible for rendering text itself, you’ll have to implement TSF support in your text control or application manually. This is a lot more work, but it can be implemented in a straightforward manner.

Text Stores and Document Locks

TSF assumes that a text store is a uniform stream of 16-bit Unicode characters, starting at offset 0. If your document doesn’t match this model, you have a couple of options. First, you can hide some characters in the stream. For example, if your document model has embedded formatting characters (HTML markup, for example), then you can tag those characters as being hidden, and TSF will hide those characters from all text services.

If your document model has multiple streams per logical document (such as Word documents), then you could create one text store per stream. If your document model has distinct editing areas (such as tables), you can separate each area with a region marker. Your application must notify the TSF manager whenever the document text changes, the selection changes, or the layout changes. This is discussed in more detail later in the article.

When you add TSF support, text services can modify your document via the ITextStoreACP methods. Since these calls can arrive at any time, interspersed with user input from the keyboard or mouse, you will need some sort of document locking so that your document is always in a consistent state.

The TSF manager will call ITextStoreACP::RequestLock when a text service wishes to read or write from the document. For example, Microsoft Global Input Method Editors (IMEs) read the document to look at the insertion context, and write to the document in order to insert the new text. While the lock is granted, your application must not call ITextStoreACPSink methods; if you do, the TSF code will get very confused.

You will also need to make sure that external events do not change the document while the lock is granted. If your control has a COM interface, you must block or postpone any COM calls that would change the document.

ITextStoreACP::RequestLock has two parameters. The first parameter (dwLockFlags) contains the lock request. It indicates whether TSF is requesting a read-only (TS_LF_READ) or read/write (TS_LF_READWRITE) lock. It also indicates whether the lock request must be granted immediately (TS_LF_SYNC), or if it may be postponed until later. The second parameter (phrSession) contains the return code of the lock request.

Your application grants the lock by calling ITextStoreACPSink::OnLockGranted using the saved interface pointer that you got from ITextStoreACP::AdviseSink. Locks can be either synchronous (ITextStoreACPSink::OnLockGranted is called within this call to RequestLock) or asynchronous (ITextStoreACPSink::OnLockGranted is called sometime later).

There are only a few instances where your application must grant a synchronous lock request. Your application must grant a synchronous lock if your application (or control) asks a text service to do something such as correcting text (via ITfFunctionProvider, the use of which is outside the scope of this article). Your application must also grant a synchronous lock whenever you call any method on the ITfKeystrokeMgr interface. Your application must also grant a synchronous lock when you do not use ITfKeystrokeMgr, and the lock request comes from within GetMessage or PeekMessage.

Aside from these exceptions, your application is not required to grant a synchronous lock request; it is perfectly allowable to only grant asynchronous lock requests. It is also perfectly allowable to treat asynchronous lock requests as synchronous lock requests.

The MSDN® documentation is a little confusing when it describes the meaning of phrSession. Figure 3 should clarify what *phrSession should contain on return from ITextStoreACP::RequestLock.

Figure 3 *phrSession Return Values (Close Figure 3)

Lock Type Requested | Lock Type Supported | *phrSession Should Contain
Synchronous | Synchronous | Return value from ITextStoreACPSink::OnLockGranted
Synchronous | Asynchronous | TS_E_SYNCHRONOUS
Asynchronous | Asynchronous | TS_S_ASYNC
Asynchronous | Synchronous | Return value from ITextStoreACPSink::OnLockGranted

If your control or application supports multiple threads, or if it reenters the message pump (for example, it calls a COM method while processing a message), then you must handle TSF locks asynchronously. One way to handle the lock request is to post a message to a message queue inside ITextStoreACP::RequestLock and then grant the lock (call ITextStoreACPSink::OnLockGranted) once the document is stable. Otherwise, it’s likely going to be easier to support locking the document synchronously.

Connecting to TSF

To illustrate how the various TSF methods work together, I’m going to present a working example based on the Scintilla text editor component available at scintilla.org. The changes that I made to Scintilla are in red.

The first thing your code must do is to register itself with the TSF framework. This must be done at least once per thread; if your control can be created on separate threads, then it is preferable to do this once per control instance. You do this by creating an ITfThreadMgr object via CoCreateInstance and activating the object. Activating the thread manager marks your control as being TSF-aware and returns a client ID that you need to create a context object.

Once you have a thread manager, you also need to create ITfDocumentMgr and ITfContext objects. You need one instance of each object per active document. Since the Scintilla component is window-based, I create a thread manager, document manager, and context in the Initialise method, which gets called in the WM_CREATE handler, as shown in Figure 4.

Figure 4 Initializing the Component for TSF (Close Figure 4)
void ScintillaWin::Initialise()
{
    // Initialize COM.  If the app has already done this it will have
    // no effect.  If the app hasn’t, we really shouldn’t ask them to
    // call it just so this internal feature works.
    ::OleInitialize(NULL);
    // Register with TSF...
    HRESULT hr = CoCreateInstance (CLSID_TF_ThreadMgr, NULL,
        CLSCTX_INPROC_SERVER, IID_ITfThreadMgr, (void**)&cpThreadMgr);
    if (SUCCEEDED(hr))
    {
        hr = cpThreadMgr->Activate(&tid);
    }
    // TSF: create context for this window
    if (SUCCEEDED(hr))
    {
        hr = cpThreadMgr->CreateDocumentMgr(&cpDocMgr);
    }
    if (SUCCEEDED(hr))
    {
        hr = cpDocMgr->CreateContext(tid, 0,
            reinterpret_cast<ITextStoreACP *>(&acp),
            &cpContext, &ecTextStore);
    }
    if (SUCCEEDED(hr))
    {
        hr = cpDocMgr->Push(cpContext);
    }
}

When the window is destroyed, the thread manager, document manager, contexts, and other objects need to be released. I do this in the Finalise method, which gets called just before object destruction (see Figure 5).

Figure 5 Releasing TSF Objects (Close Figure 5)
void ScintillaWin::Finalise()
{
    ScintillaBase::Finalise();
    SetTicking(false);
    SetIdle(false);
    DestroySystemCaret();
    ::RevokeDragDrop(MainHWND());
    // TSF: Cleanup
    cpDocMgr->Pop(TF_POPF_ALL);
    cpDocMgr = NULL;
    cpThreadMgr->Deactivate();
    cpThreadMgr = NULL;
    cpContext = NULL;
    cpSinkUnk = NULL;
    cpTextStoreACPSink = NULL;
    ::OleUninitialize();
}

Implementation Details

There are a couple of important design decisions I made while implementing ITextStoreACP for Scintilla. My first decision was to not support display attributes. This simplified the code without affecting dictation, which doesn’t use display attributes. Not supporting embedded objects was also obvious, as Scintilla doesn’t support embedded objects anyway.

The last decision to make was how to handle the mismatch between Scintilla’s 8-bit per character document stream and the 16-bit per character stream that ITextStoreACP expects. I resolved this by treating the trailing characters (DBCS trail bytes, or UTF-8 trailing bytes) as hidden white space within a 16-bit character stream. This complicated the implementation of ITextStoreACP::GetText, but greatly simplified the implementation of every other routine, as it maintained the one-to-one mapping between Unicode and DBCS characters.

Once that decision was made, the implementation of the other ITextStoreACP methods is quite straightforward. You can find the entire implementation in the source code download for this article.

Scintilla already had a notification mechanism that fired some sort of event whenever TSF needed notification. All I had to do was add the appropriate TSF notification in Scintilla’s notification routines as shown in Figure 6.

Figure 6 Adding TSF Notification Handlers (Close Figure 6)
void ScintillaWin::NotifyFocus(bool focus)
{
    ::SendMessage(::GetParent(MainHWND()), WM_COMMAND,
        MAKELONG(GetCtrlID(), focus ? SCEN_SETFOCUS : SCEN_KILLFOCUS),
        reinterpret_cast<LPARAM>(MainHWND()));
    //TSF:  Update focus
    if (!!cpThreadMgr)
    {
        cpThreadMgr->SetFocus(focus ? cpDocMgr : NULL);
    }
}
void ScintillaWin::ClaimSelection()
{
    //TSF:  The selection has changed - notify the sink,
    //      if there’s no lock held.
    if (!!cpTextStoreACPSink && dwLock == NONE)
    {
        cpTextStoreACPSink->OnSelectionChange();
    }
}

sptr_t ScintillaWin::WndProc(unsigned int iMessage,
    uptr_t wParam, sptr_t lParam)
{
    ... // lots of existing code not shown

    case WM_WINDOWPOSCHANGED:
       // TSF: Update position
       if (!!cpTextStoreACPSink && dwLock == NONE)
       {
           cpTextStoreACPSink->OnLayoutChange(TS_LC_CHANGE, 0);
       }

    ... // lots more existing code not shown
}

void ScintillaWin::NotifyParent(SCNotification scn)
{
    scn.nmhdr.hwndFrom = MainHWND();
    scn.nmhdr.idFrom = GetCtrlID();
    ::SendMessage(::GetParent(MainHWND()), WM_NOTIFY,
        GetCtrlID(), reinterpret_cast<LPARAM>(&scn));
    // TSF: update sinks
    if (!!cpTextStoreACPSink && dwLock == NONE)
    {
        if (scn.nmhdr.code == SCN_MODIFIED)
        {
            bool fNotify(false);
            static TS_TEXTCHANGE chg;
            // Lots of notifications get routed here.
            // We’re only interested in insert/delete
            // insert operations have before/after
            // notifications, so we save up info
            if (scn.modificationType & SC_MOD_BEFOREINSERT)
            {
                chg.acpStart = scn.position;
            }
            if (scn.modificationType & SC_MOD_INSERTTEXT)
            {
                fNotify = true;
                chg.acpOldEnd = scn.position;
                chg.acpNewEnd = scn.position + scn.length;
            }
            if (scn.modificationType & SC_MOD_BEFOREDELETE)
            {
                chg.acpStart = scn.position;
            }
            if (scn.modificationType & SC_MOD_DELETETEXT)
            {
                fNotify = true;
                chg.acpOldEnd = scn.position + scn.length;
                chg.acpNewEnd = scn.position;
            }
            if (fNotify)
            {
                cpTextStoreACPSink->OnTextChange(0, &chg);
            }
        }
        else if (scn.nmhdr.code == SCN_UPDATEUI)
        {
            cpTextStoreACPSink->OnLayoutChange(TS_LC_CHANGE, 0);
        }
    }
}

The last thing that needs to be considered is document locking. Keeping TSF from modifying the document while the user is working on it turned out to be trivial; Scintilla runs in a single-threaded apartment, and message dispatch is atomic (Scintilla doesn’t pump messages while processing a single message). This also meant I could handle lock requests synchronously—there was no way that I could get a TSF lock request while processing another message.

Making sure that TSF requests have the appropriate lock is also trivial. The lock can be checked at all entry points to ITextStoreACP. The hardest part is making sure that the user can’t modify the document while a text service is holding the lock. I chose to implement this lock by queuing incoming Windows messages while the lock was held, and then draining the queue after the lock was released.

The other thing that needs to be queued is the notifications that Scintilla generates while the lock is held. These notifications can cause very strange behavior if they are delivered to the host application while the lock is held.

Interaction between Scintilla and TSF

It may not be clear how events in the Windows Speech Recognition Speech Text Service (such as dictating some text) cause changes in Scintilla, or how events in Scintilla cause changes in the Speech Text Service. Let’s take a look at the interactions between the Windows Speech Recognition Speech Text Service, TSF, and Scintilla. Figure 7 shows how Scintilla creates the TSF objects and how the Speech Text Service becomes aware that Scintilla has a text store.

Figure 7 Creating the TSF Objects
Figure 7 Creating the TSF Objects(Click the image for a smaller view)
Figure 7 Creating the TSF Objects
Figure 7 Creating the TSF Objects (Click the image for a larger view)

Figure 8 shows how the Speech Text Service inserts text into a Scintilla document. This diagram only shows the objects that are related to Scintilla; there are other (TSF-managed) objects that are not shown here. The key insight here is that TSF insulates the application from the text service, and vice versa. The application controls when changes can be made by granting lock requests, and the text service gets notified when relevant changes are made.

Figure 8 Inserting Text into a Document
Figure 8 Inserting Text into a Document(Click the image for a smaller view)
Figure 8 Inserting Text into a Document
Figure 8 Inserting Text into a Document (Click the image for a larger view)

Figure 9 shows how Scintilla and TSF interact when the user types some text into a Scintilla document. Note that TSF locks the document after a text change. TSF does this in order to guarantee that TSF can atomically update its internal data structures. Again, TSF insulates the text service from the application while still allowing text services to be aware of document changes.

Figure 9 Locking a Document for Input
Figure 9 Locking a Document for Input(Click the image for a smaller view)
Figure 9 Locking a Document for Input
Figure 9 Locking a Document for Input (Click the image for a larger view)

Advanced Topics

It is possible to wrap TSF support around a control that you don’t have source for. There are four areas that need special attention. First, you must be able to send ITextStoreACPSink notifications at the appropriate time. If the control doesn’t provide those notifications, it is much more difficult to add TSF support. Second, implementing ITextStoreACP::GetACPFromPoint requires that you know how the control lays out text; this can be tricky without source. Third, locking the document can also be challenging; the best way is to wrap the third-party control in an abstraction layer so that you can postpone or deny incoming change requests while the control is locked. Finally, if the control supports asynchronous changes (such as loading from a URL), wrapping TSF support around that control will be a real challenge.

It is possible to implement ITextStoreACP in managed code using P/Invoke to call the appropriate methods. However, since you are calling unmanaged code, you need to have full trust (or at least SecurityPermissionFlag.UnmanagedCode) to allow the calls.

Debugging a text service can be quite difficult on a single machine. If your program hits a breakpoint while running under the debugger, the Text Services Manager will attempt to call into your application (because of the status notification). Since your application is halted, the Text Services Manager halts, which blocks the entire system. I strongly recommend that you debug your text store implementation on another machine via remote debugging.

Conclusion

Adding dictation support to your application, even adding full TSF support, is not as difficult as it may seem at first. Windows Vista, via Windows Speech Recognition and TSF, does most of the heavy lifting.

The download for this article contains the complete edited code. For more information about the Scintilla control used as an example here, see www.scintilla.org. To learn more, see the definitive reference for TSF on the MSDN® Web site at msdn2.microsoft.com/ms629032.aspx.

Eric Brown has been a software developer for over 20 years. He has a passion for speech recognition and dictation. He blogs at blogs.msdn.com/tsfaware.


Editor's Note: What Is Mobility?

Editor's Note
What Is Mobility?
Joshua Trupin

What do you think of when you hear the word "mobility?" It differs by generation. One person might think of a motorized scooter while another reflects on his own car and how it frees him from the horrors of safe, reliable, fiscally sound mass transit. Geeks of a certain age remember a Kaypro computer that was "portable"—though it felt like a cinder block with a handle. Today, the definition of mobile is both narrowing and broadening at once.

"But how is that even possible?" you might ask, scratching your head. It’s quite possible, and it’s a matter of consumer expectations on both ends.

No longer is mobility merely a function of location. Just because you can use a device away from the office doesn’t mean that it’s in any way a rich and rewarding experience. Look at Pocket PCs, for instance. Sure, you can use them for e-mail and associated tasks, but unless you’re always online you have to intentionally sync them up when you’re at base. Now look at a Windows Mobile phone, such as the Treo. It looks the same, sure, but this device automatically connects and keeps your e-mail up-to-date wherever you are. The definition of truly mobile computing has narrowed.

At the same time, the number of devices that fit into this category has expanded enormously. Smartphones are the norm now—and if you don’t believe us, take a look at what people are using on trains, in stores, and while behind the wheel of the car next to yours. With the growth in this category, mobile devices have finally moved from niche to mainstream in the past year. If you have one of these gadgets, chances are you are wondering how you ever lived without it. From sending text messages to someone in the next room to playing Freecell while driving, they offer unmatched utility.

With Ultra-Mobile PCs, there’s more continuity in the marketplace now. You can choose just about any size device that suits you. However, with all those machines out there, there’s a lot of stuff you should think about when writing your app, whether you’re designing it for mobile devices or for regular desktop machines.

To that end, this issue of MSDN Magazine features two articles to help you design programs that will serve the full range of form factors, from the smallest Smartphone to the bulkiest laptop. The first is "Make Your WPF Apps Power-Aware," by Andre Michaud. Andre discusses power management support for apps that employ Windows Presentation Foundation on both Windows XP and Windows Vista, and how to write your applications to take full advantage of it so that portable devices can enjoy better battery life.

Daniel Moth takes an in-depth look at the .NET Compact Framework in his article, "Write Code Once For Both Mobile And Desktop Apps." Specifically, he explores how you can share your code between a standard desktop version of your app and a version that runs on a smart device. Can desktop apps be modified for mobile devices? Can mobile apps run on a desktop machine? Can you even reuse your codebase across the platforms? If you’re easily startled, you better sit down before reading the shocking answer!

With the newest devices and rapid improvement in existing forms, mobile computing is on a roll right now. How is it working for you? Let us know, and join the discussion on the MSDN Magazine blog. We look forward to chatting with you. —J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Christian Allred, Claudio Caldato, Jon Cargille, Stephan Doll, Shawn Farkas, Donovan Follette, Matt Gibbs, Dushyant Gill, Richard Greenberg, Dana Groff, Brian Grunkemeyer, Solom Heddaya, Elisa Johnson, Ladi Prosek, Travis Querec, Rob Relyea, and Andrew Tucker.


Toolbox: Find Bottlenecks, Code Similarities, and More

Toolbox
Find Bottlenecks, Code Similarities, and More
Scott Mitchell

Contents
Find Performance and Memory Bottlenecks
Quickly Find Similar Code in Multiple Files
A Simple, Lightweight Color Picker
Learn the Essentials of Security

Find Performance and Memory Bottlenecks

This summer my wife and I are embarking on a three-week, 215-mile backpacking trip. After formulating the initial pack list, we spent several days evaluating equipment and paring down the list to reduce our pack weights. To maximize our efforts, we focused primarily on the heaviest items—the tent, sleeping bags, bear canister, and so on—and virtually ignored the lighter items. It’s a classic case of the old 80/20 rule, that 20 percent of the supplies constitute 80 percent of the overall weight.

The same tactics should be employed when optimizing an application’s performance. While there may be many thousands of lines of code in an application, chances are its performance is primarily affected by a small handful. It is important to identify where the time- and memory-based bottlenecks reside and focus your attention accordingly. Identifying these code regions is a fairly straightforward task with the help of a code profiler, which is a specialized application that monitors and records the time and/or memory resources used by another application.

ANTS Profiler Identifies Time- and Memory-Based Bottlenecks
ANTS Profiler Identifies Time- and Memory-Based Bottlenecks (Click the image for a smaller view)
ANTS Profiler Identifies Time- and Memory-Based Bottlenecks
ANTS Profiler Identifies Time- and Memory-Based Bottlenecks (Click the image for a larger view)

One such third-party code profiler is Red Gate Software’s ANTS Profiler 2.7. ANTS Profiler is a cinch to use. Upon starting, a three-step Profiler Project Wizard walks through the setup process, first asking whether you want to profile performance or memory. Profiling performance will show the number of times each line of code is executed along with how much total time was spent executing each one. It also summarizes the slowest lines of code and methods. Profiling memory works by taking periodic snapshots of the heap, objects in memory, the total memory footprint, and other memory-related metrics at various points in the program’s execution.

Next, the wizard asks if you want to profile a Microsoft® .NET Framework desktop application, an ASP.NET Web application, a .NET Windows® service, or a COM+ server application. ANTS Profiler can profile .NET Framework applications for versions 1.0, 1.1, and 2.0. Finally, the wizard asks for the location or name of the application to profile, at which point the application to profile is launched. During the program’s execution, ANTS Profiler will automatically capture either execution or memory usage information.

After closing the profiled application, the ANTS Profiler report is displayed, providing a high-level summary along with detailed metrics. When profiling performance, ANTS Profiler shows the source code with a column listing the number of times each line of code was executed and the total execution time. Lines of code that experienced an exceptionally long execution time are flagged. When profiling memory, the detailed report shows all objects in a grid that can be sorted and filtered. Selecting a particular object shows where it exists in the code tree.

When optimizing an application, it is essential to focus on the code that’s causing the greatest impact on performance. Code profilers use empirical evidence to help identify trouble spots. For an easy-to-use, feature-rich profiler, check out ANTS Profiler.

Price: $295 standard edition, $495 professional edition.

www.red-gate.com/products/ants_profiler

Quickly Find Similar Code in Multiple Files

One of the challenges inherent in any non-trivial project, especially those involving multiple developers, is code repetition. For example, Bob may need to format output, parse input, or perform a calculation the same way in multiple places in the system. Ideally, such similar code would be refactored into a method that would then be called from the multiple locations, but Bob may be in a hurry or has made a mental note to come back and fix this issue at a later date. But before Bob refactors his code, imagine what would happen if Alice was reviewing the code and found and fixed a bug. Unless Alice knows that Bob’s code has been duplicated elsewhere, chances are she won’t make the bug fix in both locations. In short, duplicate code compromises the maintainability and updateability of an application.

Fortunately, there are tools that will quickly identify code repetition. Simian 2.2 by Red Hill Consulting is a similarity analyzer that searches for duplicate syntax in multiple files. The program understands a variety of programming language syntaxes—C#, T-SQL, JavaScript, and Visual Basic®, among others—and can also find repetitions in plain-text files. There are a variety of configuration options that allow the user to fine-tune the rules that identify duplicate code. For example, the threshold setting dictates how many identical lines of code must be present to qualify as a duplicate.

Simian is a lightweight tool designed to efficiently find duplicate code. It lacks a graphical interface, but can be used through the command line or accessed programmatically. Its results are output as plain text, but can be formatted through one of the built-in format types (such as XML). While Simian’s lean interface and limited reporting features add a bit to the learning curve, they help keep the product focused and efficient.

Simian is capable of searching for duplicate code in both large and small projects. For instance, Red Hill Consulting reports that Simian compared the Java Development Kit’s 1.3 million lines of raw source code spread across 4,136 files in approximately 25 seconds. In my own tests, I found Simian to finish in well under one second when analyzing projects with around 15,000 lines of raw source code spread over 200 files or so.

Duplicate code impacts a project’s maintainability and updateability. You can use Simian to find duplicate code across multiple files quickly. Since it can be executed from the command line, Simian can be plugged into the build process to provide warnings or stop the build altogether in the face of repetitive code.

Price: $99 for a personal license; $499 for a project license.

www.redhillconsulting.com.au/products/simian

A Simple, Lightweight Color Picker

If you have ever designed the front end of a Web site or worked on the user interface for a commercial desktop application, you have most likely needed to know the particular RGB settings of a specific onscreen color. Perhaps you need to match the form’s background color to the color of your company’s logo or want to color a Web page’s text the same color as it appears in the PDF brochure. If you do not know the exact color specifications, you can use a color picker to discover them. Color pickers report the RGB combination of a particular color, typically the color currently underneath the mouse pointer.

Standalone color picker programs are nice when you’re not already working inside a graphic editing program. It is a hassle to have to open the image or take a screenshot and paste it in the graphic editing program just to determine the specifications for a particular onscreen color.

Pixie's Simple Interface Suplies Detailed Color Info
Pixie's Simple Interface Suplies Detailed Color Info (Click the image for a smaller view)
Pixie's Simple Interface Suplies Detailed Color Info
Pixie's Simple Interface Suplies Detailed Color Info (Click the image for a larger view)

One of the best freeware color pickers I have come across is Pixie 1.0, by Nattyware. When launched, Pixie appears as a small square in the center of the desktop, showing the current color under the mouse pointer in hexadecimal, HTML, RGB, and other formats, as well as the current mouse pointer’s X and Y coordinates. In addition to its onscreen user interface, Pixie can also be minimized to the task tray.

Pixie has a handful of actions that are mapped to keyboard shortcuts. To magnify the area under the mouse, simply press Ctrl+Alt+Z. Use Ctrl+Alt+X to bring up the standard Windows color palette. Once you’ve placed your mouse’s pointer over the color of interest, hit Ctrl+Alt+C to copy the color’s information to the clipboard.

Pixie’s utility lies in its simplicity and minimalist interface. If you find yourself needing to determine the specs for an onscreen color, give Pixie a try.

Price: Free; source code, $25.

www.nattyware.com/pixie.html

Learn the Essentials of Security

Security requires planning and domain knowledge and its results are not easily measurable. Furthermore, since security spans physical, technological, and policy levels, it is easy for a developer to bury her head in the sand and assume someone else will implement or has implemented the necessary security precautions. Such attitudes, however, are an invitation to disaster. Security is an important aspect of any application and all developers should have at least a cursory understanding of key security concepts, potential attack vectors, and techniques for protecting against common threats.

Foundations of Security: What Every Programmer Needs to Know (Apress, 2007), by Neil Daswani, Christoph Kern, and Anita Kesavan, provides a solid overview of security fundamentals at a level appropriate for developers who may have little to no background in security. The book starts with three chapters on security design principles, covering concepts such as authentication, authorization, encryption, and message integrity.

(Click the image for a smaller view)

The meat of the book is found in the second section, which explores secure programming techniques. Here you will find discussions on common attack vectors—buffer overflows, client-state manipulation, SQL injection attacks, techniques for securely storing passwords, cross-site scripting (XSS) attacks, and so on—along with practices for stopping these types of attacks. The code samples in this section are written in Java, but don’t let that dissuade you from reading the book as they are provided mainly as pseudocode-like reference material.

The final section explores cryptography with chapters on symmetric encryption, asymmetric encryption, key management, Message Authentication Codes (MACs), and signatures.

Security is an important part of every project that is often avoided at the expense of your users’ private data and your company’s bottom line. Foundations of Security: What Every Programmer Needs to Know is an accessible introduction to security that every developer should consider reading.

Price: $39.99.

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out: Digging into IDisposable

CLR Inside Out
Digging into IDisposable
Shawn Farkas

Contents
Disposable Objects
The Disposable Pattern
Managed vs. Native Resources
Managed Resource Cleanup
Deriving from a Disposable Type
Dispose and Security
SafeHandles
Conclusion

One of the major productivity benefits that the common language runtime (CLR) offers developers of managed code is that the garbage collector (GC) makes sure any memory allocated on the managed heap is cleaned up after it is no longer needed. Developers are thus saved countless hours of debugging difficult problems arising from memory leaks, from trying to use released memory, and from double-freeing memory. However, while the garbage collector is great at making sure that memory does not leak, it doesn’t have any knowledge about other resources that need to be freed. For instance, the garbage collector doesn’t know how to close a file handle or how to release memory allocated outside of the managed heap with an API such as CoAllocTaskMem.

Objects that manage these types of resources must ensure that they are released when they’re no longer needed. You can accomplish this by overriding the Finalize method of System.Object, which lets the garbage collector know that the object would like to participate in its own cleanup (in C# you use the C++ destructor syntax, ~MyObject, rather than directly overriding the method). If a class has a finalizer, then before objects of that type are collected the garbage collector will call the object’s finalizer and allow it to clean up any resources that it may be holding onto.

One problem with this system is that the garbage collector does not run deterministically and, as a result, your object may not be finalized for a long time after the last reference to it has gone away. If your object holds onto an expensive or rare resource, such as a database connection, this may not be acceptable. For instance, if your object has 1 of only 10 available connections open, it should release that connection as soon as possible, rather than waiting for the garbage collector to call the finalize method.

Disposable Objects

To avoid having to wait an indeterminate amount of time for the garbage collector to act, types that own resources should implement the IDisposable interface; then consumers of that type can release those resources in a timely manner. Implementing IDisposable is a hint to consumers of your object that they should call Dispose as soon as they are finished with it, which enables the object to release resources as soon as they are no longer needed. In fact, C# and Visual Basic® both provide a "using" keyword to make this process as painless as possible. In C#, if you have a class, MyClass, which implements IDisposable, then the following code will cause the compiler to generate code similar to that in Figure 1.

Figure 1 Generated Code (Close Figure 1)
MyClass myClass = GetMyClass();
try
{
    myClass.DoSomething();
}
finally
{
    IDisposable disposable = myClass as IDisposable;
    if (disposable != null) disposable.Dispose();
}

using (MyClass myClass = GetMyClass())
{
    myClass.DoSomething();
}

The try/finally block ensures that even if the code within the using block throws an exception, MyClass’s Dispose method will be given an opportunity to clean up the object. The compiler-generated code does two interesting things before calling the Dispose method. First, it checks to ensure that the disposable object is not null—so if the expression assigned to the variable happens to evaluate to null, a using block will not cause a NullReferenceException in your app. It also calls Dispose through an IDisposable reference, which allows types that explicitly implement IDisposable to still be cleaned up properly.

Although the using block will work with classes that do have an explicit IDisposable implementation, I recommend that classes never implement the interface this way. If you explicitly implement IDisposable, developers who are exploring your object model using IntelliSense® in Visual Studio® will not notice that the object has a Dispose method and therefore may not take advantage of early cleanup, causing your object to hold onto its resources until the garbage collector runs the finalizer.

It’s important to note that from the garbage collector’s point of view, IDisposable is just another interface. The garbage collector does not treat disposable objects differently from objects that do not have an IDisposable implementation; specifically, the garbage collector will never call your Dispose method for you. The only cleanup method that the garbage collector will call is the object’s finalizer; if code is not written to explicitly call the Dispose method, then the method will never be called.

The Disposable Pattern

You’ve seen that the "using" constructs in C# and Visual Basic make cleaning up disposable objects easy. On the other side of that equation are developers who need to create disposable types. Since the IDisposable interface defines only a single method, it seems as though developers of disposable types should be able to quickly and easily implement the interface. However, as indicated in Joe Duffy’s IDisposable guidelines, there are many subtle details to implementing a truly robust disposable type.

One key observation about a Dispose method is that the cleanup work it does is very similar to the cleanup work that would be done by the object’s finalizer. Since avoiding code duplication is always a good thing, it’s better to share this cleanup code between the two locations rather than to copy and paste it between them. This also has the benefit of keeping all of the type’s cleanup code in one location. The obvious way to share this code is to put all of the cleanup code into the Dispose method and simply have the object’s finalizer call into Dispose:

public class MyClass : IDisposable
{
    ~MyClass()
    {
        Dispose();
    }

    public void Dispose()
    {
        // Cleanup
    }
}

Though this certainly accomplishes the goal of keeping all of the cleanup code in one place, there is a problem with implementing a finalizer this way. While some of the cleanup code may be shared between Dispose and the finalizer, there is actually a set of resources that you would want to release in a Dispose method, but you would not want to release during finalization.

Managed vs. Native Resources

Since finalizers will not run in a deterministic order, you cannot guarantee that objects your object references have not already been finalized when your finalizer is run. For example, consider the following classes:

public class Database : IDisposable
{
    private NetworkConnection connection;
    private IntPtr fileHandle;

    // ...
}

public class NetworkConnection : IDisposable
{
    private IntPtr connectionHandle;

    // ...
}

The Database class contains two resources, one of type NetworkConnection and an IntPtr, which represents a file handle. While there is still an active reference to an object of type Database, the garbage collector will not collect the NetworkConnection object referenced by the connection member variable. However, if the garbage collector is calling the Database object’s finalizer, then you know that there were no more live references to the object and it became eligible for collection.

It’s possible that prior to the execution of the database finalizer, the NetworkConnection object may have already been finalized if there were no other live references to it. Using an object that has already been finalized is a bad idea as the object will likely not be in a usable state; since there are no guarantees as to the order of finalization, the Database finalizer should not reference the connection field.

However, if you’re cleaning up as a result of a call to the Dispose method, you know that you are not being finalized since there must have been a live reference to the object for Dispose to have been called. Your Database object is still alive and it is acting as a GC root for the NetworkConnection referenced by the connection field, preventing the NetworkConnection object from being finalized. Therefore, in a Dispose method it is safe to access finalizable member variables.

The story is different with the fileHandle field, however. This handle field is just an IntPtr, which is not something that the garbage collector will finalize. This means it is safe to access the fileHandle field in both the finalizer and the Dispose method.

Generalizing this principle, in a Dispose method it’s safe to clean up all resources that an object is holding onto, whether they are managed objects or native resources. However, in a finalizer it is only safe to clean up objects that are not finalizable, and generally the finalizer should only be releasing native resources.

It’s interesting to notice that in this example the connection field is not considered a native resource even though it clearly contains a native connection handle. While the internal implementation of the NetworkConnection class does contain a native resource, from the perspective of the Database class NetworkConnection is a managed resource since the NetworkConnection type is responsible for managing the lifetime of its own resources. Within the NetworkConnection class, however, connectionHandle is a native resource, as it will not be cleaned up unless code is written to explicitly do so.

Given that you want to clean up all of your resources if you’re being disposed and a subset of these resources if you’re being finalized, you can factor your cleanup code into a method that is called by both the finalizer and Dispose method. This new method will then make sure that all resources are cleaned up if you are disposing and that only the native resources are cleaned up if the object is being finalized. This requires you to create a disposable implementation that follows the pattern in Figure 2.

Figure 2 Basic IDisposable Pattern (Close Figure 2)
public class DisposableClass : IDisposable
{
    ~DisposableClass()
    {
        Dispose(false);
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            // Clean up all managed resources
        }

        // Clean up all native resources
    }
}

You’re still meeting the original goal of avoiding code duplication and keeping cleanup code together, as all of the cleanup code is now in an override of the Dispose method. I’ll refer to this method as the cleanup method for the rest of this article in order to avoid confusion with the other Dispose method. The cleanup method takes a Boolean parameter indicating if you are in the dispose code path or the finalization code path. In both code paths you should make sure to clean up the native resources your class holds onto. Additionally, in the Dispose code path you should also clean up managed resources.

The cleanup method does not need to be public, since you don’t want consumers of your class to call it directly. Instead, they can access your cleanup logic via the IDisposable interface. While consumers of your class will not need to directly invoke your cleanup method, any subclasses will want to call it, so it is marked as protected rather than private (more on subclassing disposable types later).

Since Dispose will release all resources that an object holds onto, there’s no reason for the garbage collector to finalize the object. To tell the garbage collector that the object does not need to be finalized, even though it has a finalizer, call GC.SuppressFinalize after the cleanup code has executed in your Dispose method.

In fact, if a type does not own any native resources, the code executed by the finalizer will not do anything at all. In those cases, the class should not define a finalizer method.

In the Database example, the new cleanup method might look like Figure 3. If the Database object is being disposed, it cleans up both the connection managed resource and the fileHandle native resource. However, if the object is being finalized, it cleans up only the fileHandle resource. Notice that the cleanup code is safe to call multiple times. The first call through the Dispose method will release any resources the Database object is holding onto; future calls to Dispose will find that the resources are already released and exit without attempting to free them. Dispose implementations should be coded so they will succeed even if they are called many times, even though the extra calls are unnecessary. Other methods on an object may throw an ObjectDisposedException if they are called after the object is disposed.

Figure 3 Example Database Cleanup Method (Close Figure 3)
protected virtual void Dispose(bool disposing)
{
    if (disposing)
    {
        if (connection != null)
            connection.Dispose();
    }

    if (fileHandle != IntPtr.Zero)
    {
        CloseHandle(fileHandle);
        fileHandle = IntPtr.Zero;
    }
}

It’s also interesting to notice that the cleanup code does not throw any exceptions. Generally developers expect to be able to clean up an object without causing an exception. This is because there’s really no good way to recover from an exception during a cleanup code path. If an object fails while cleaning up, the only choices a consumer of that object has is to retry and hope the second attempt succeeds, or to give up and allow that some resources may not be released properly.

In order to ensure that your cleanup code does not throw an exception, you should be sure to check any object references you access in the cleanup method to ensure that they are not null, and to only call methods that do not fail. This becomes vitally important when the object is being finalized. While an exception during a Dispose call simply leaves developers in a bad place, an unhandled exception on the finalizer thread by default tears down the entire process in version 2.0 of the CLR.

Even if you’re not running in the finalizer thread, throwing an exception during cleanup can prevent other resources from being cleaned up as you would expect. For example, take a look at Figure 4. Here, DisposableClass contains resources of type Foo and Bar. If Foo.Dispose throws an exception while executing, this DisposableClass will never get a chance to dispose of the Bar object.

Figure 4 The Dangers of Exceptions in Dispose (Close Figure 4)
public class Disposableclass : IDisposable
{
    Foo a;
    Bar b;

    // ...

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            if (a != null) a.Dispose();
            if (b != null) b.Dispose();
        }
    }
}

Managed Resource Cleanup

So, if managed resources are not cleaned up when an object is being finalized, what prevents them from leaking? Well, these objects should follow the IDisposable pattern if they contain resources that need to be cleaned up. Since an object’s finalizer cannot tell if its managed resources have been finalized yet, it cannot clean them directly. However, if these types contain resources that need to be released, they should also have finalizers that the garbage collector can run; if they only contain managed resources, then no finalizer is necessary.

Let’s look at an example of this. Imagine a Cookie class used by a managed control to maintain its state across browser sessions. The Cookie class might be implemented by using IsolatedStorage, which itself might be implemented using a FileStream. The FileStream may contain a Win32® file handle, so the final classes look like the code shown in Figure 5.

Figure 5 Implementing IDisposable in a Class Hierarchy (Close Figure 5)
public class DisposableBase : IDisposable
{
    ~DisposableBase()
    {
        Dispose(false);
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        // ...
    }
}

public class DisposableSubclass : DisposableBase
{
    protected override void Dispose(bool disposing)
    {
        try
        {
            if (disposing)
            {
                // Clean up managed resources
            }

            // Clean up native resources
        }
        finally
        {
            base.Dispose(disposing);
        }
    }
}

If the control forgets to dispose of its Cookie object, the Cookie will not have a finalizer to clean up its IsolatedStorageFileStream, since this stream is a managed resource. That’s fine, because the garbage collector will collect the stream object for you. Again, IsolatedStorageFileStream does not need a finalizer because it only contains managed resources. This is still fine since the FileStream object will contain a finalizer, as it does own a native resource. When the garbage collector finalizes the FileStream, it will close the handle. Even though nobody disposed of the root object in this object graph, no resources were leaked because every class involved followed the disposable pattern.

Deriving from a Disposable Type

When you derive from a disposable type, and that derived type does not introduce any new resources, then nothing special needs to be done. The base type IDisposable implementation will take care of cleaning up its resources and your subclass can be blissfully ignorant of the details. However, it’s also common to have a subclass that does contain new resources that need to be cleaned up. In this case, your class needs to release its resources, while making sure that the base type’s resources also get released. Do this by overriding the cleanup method, releasing your resources, and then calling the base type to clean up its resources, as in Figure 6.

Figure 6 Overriding Dispose (Close Figure 6)
public class FileStream : IDisposable
{
    private IntPtr fileHandle;

    // ...

    protected void Dispose(bool disposing)
    {
        if (fileHandle != IntPtr.Zero)
        {
            CloseHandle(fileHandle);
            fileHandle = IntPtr.Zero;
        }
    }
}

public class IsolatedStorageFileStream : IDisposable
{
    private FileStream file;

    // ...

    protected void Dispose(bool disposing)
    {
        if (disposing)
        {
            if (file != null)
                file.Dispose();
        }
    }
}

public class Cookie : IDisposable
{
    private IsolatedStorageFileStream isolatedStore;

    // ...

    protected void Dispose(bool disposing)
    {
        if (disposing)
        {
            if (isolatedStore != null)
                isolatedStore.Dispose();
        }
    }
}

Notice that the derived type only has to override the cleanup method, as the inherited version of the public Dispose method and finalizer do exactly what you need. The exception to this rule is if the base type does not have a finalizer because it did not own any native resources and the derived type happened to own native resources. In this case, you would have to add a finalizer to the derived type.

In the derived type, you want to make sure to clean up your resources before you call your base type’s cleanup method. This way, objects get torn down in the reverse order that they are constructed. If you called the base type’s cleanup method first, you might get into a state where the base type has torn down part of the object that you depend upon to complete your cleanup method.

Dispose and Security

One subtlety of writing cleanup code relates to the security context it can run in. All cleanup code must be able to run no matter what identity the executing thread is impersonating. Since the garbage collector runs finalizers on a dedicated thread, any impersonation active on the thread that instantiated the disposable object will not be in place when the finalizer is run. There’s also no way to ensure that between object construction and calling Dispose, the code consuming your type hasn’t reverted the thread’s impersonation, or even impersonated another identity all together.

One example of a class that handles impersonation incorrectly is the RSACryptoServiceProvider class in the Microsoft® .NET Framework 2.0. If an RSACryptoServiceProvider object creates an ephemeral key to work with, the finalizer of the object will attempt to delete the key; this key is stored in the profile of the user the thread creating the key is running as. The problem arises if an RSACryptoServiceProvider object is created while a thread is impersonating a user, and the object is not disposed. In this situation, the finalizer will run in the context of a different user than the object constructor. When the finalizer attempts to delete the key, it will cause an exception because the user the finalizer thread is running as will likely not have access to the user profile where the key was stored.

The fact that finalizers run on a separate thread also means that you can’t rely on checking permissions of the call stack in your Dispose method. If the cleanup method is being called by your object’s finalizer, there will be no code above it on the call stack and any Code Access Security checks are meaningless. (Security demands should generally be avoided in cleanup code anyway, since they could lead to an exception.)

SafeHandles

The .NET Framework 2.0 introduced the SafeHandle class to help with resource management. A SafeHandle wraps a native resource and provides several benefits over using an IntPtr to hold onto the resource. Full details of SafeHandle are outside the scope of this article, but more information can be found in Brian Grunkemeyer’s excellent post on the Base Class Library (BCL) team blog.

A strong argument can be made that with the introduction of SafeHandle, there is no longer any reason for a type not derived from SafeHandle to contain raw native resources. Instead, objects should only contain managed resources, including subclasses of SafeHandle for any native resources they use.

Following this pattern has several benefits. First, no classes other than the SafeHandles need to have finalizers, as only the SafeHandles will own native resources. Secondly, it creates a clean model where each SafeHandle owns exactly one native resource, and managed objects can then compose various SafeHandles to achieve their needs.

Conclusion

The CLR’s garbage collector frees you from having to worry about managing memory that’s allocated on the managed heap; however, you still need to clean up other types of resources. Managed classes use the IDisposable interface to allow their consumers to release potentially expensive resources before the garbage collector finalizes the object. By following the disposable pattern and being aware of issues to look out for, classes can ensure that all of their resources get cleaned up properly and that no problems will occur when the cleanup code is run either directly via a Dispose call or via the finalizer thread.

Send your questions and comments to clrinout@microsoft.com.

Shawn Farkas is a software design engineer on the CLR security team at Microsoft. He works on security policy, cryptography, and ClickOnce. See his blog at blogs.msdn.com/shawnfa.


Data Points: ADO.NET Entity Framework Overview

New information has been added to this article since publication.
Refer to the Editor's Update below.

Data Points
ADO.NET Entity Framework Overview
John Papa

Code download available at: DataPoints2007_07.exe (1590KB)
Contents
Entity Framework Components
Entity Data Model
Generating an Entity Data Model
Dissecting the CSDL
Mapping to the Store
Defining Inheritance
Mapping across Multiple Tables
Using EntityClient
Using Object Services
Using LINQ to Entities
Wrapping Up

[ Editor's Update - 6/19/2007: The ADO.NET Entity Framework and Tools will ship during the first half of 2008 as an update to the Visual Studio 2008 release.]

ADO.NET in the next release of Visual Studio® code-named "Orcas" features the new Entity Framework. It allows developers to focus on data through an object model instead of through a logical/relational data model. The Entity Framework helps abstract the logical data schema into a conceptual model and allows for multiple ways to interact with the conceptual model through Object Services and a new data provider called EntityClient. This month’s column discusses what the Entity Framework is, how it fits into an application, and how it can be designed and programmed against.

The Entity Framework abstracts the logical database structure using a conceptual layer, a mapping layer, and a logical layer. In this column, I review the purpose of each of these layers. I also introduce EntityClient and a new language, Entity SQL, that can interact with the conceptual layer’s Entity Data Model (EDM).

An alternative to EntityClient is Object Services. Specifically, Object Services in the Entity Framework can help alleviate the amount of data-access code developers need to write. I also discuss and demonstrate how to use Object Services with Entity SQL and with LINQ to Entities to interact with and retrieve conceptual entities from the EDM.

Note that all of the examples in this article work with ADO.NET and the Entity Framework as they exist in Beta 1 of "Orcas."

Entity Framework Components

The Entity Framework allows developers to write less data-access code, reduces maintenance, abstracts the structure of the data into a more business-friendly (and less normalized) manner, and facilitates the persistence of data. When used with LINQ to Entities (to be discussed later), it can also help to reduce the number of compile time errors since it generates strongly typed classes from the conceptual model.

The Entity Framework generates a conceptual model that developers can write code against. This model can be interacted with directly using a new data provider called EntityClient and a new language (similar to T-SQL) called Entity SQL. EntityClient has a similar model to the familiar ADO.NET objects, using EntityConnection and EntityCommand objects to return a DbDataReader. Another option for developers is to employ Object Services using either an ObjectQuery object with Entity SQL or using LINQ to Entities. Object Services allows developers to take advantage of the conceptual model’s generated classes, which offer such features as strongly typed objects and persistence (see Figure 1).

Figure 1 Overview of the Entity Framework
Figure 1 Overview of the Entity Framework

These data-access techniques let a developer interact with the conceptual entities of the EDM. The layers of the EDM exist as XML files; at this time the EDM can be generated using a command line tool (EDMGEN.EXE), manually, or with a wizard in Visual Studio.

Entity Data Model

The core of the Entity Framework is in its models. The Entity Framework supports a logical store model that represents the relational schema from a database. A relational database often stores data differently than an application uses the data. This typically forces developers to retrieve the data in the same structure as the database contains it. Developers often then feed the data into business entities that are more suited for handling business rules. In this example, the schema from the relational database is represented in a logical model and the business entities represent the conceptual model. The Entity Framework bridges this gap between the models using a mapping layer. Thus there are three layers active in the Entity Framework’s model:

  • Conceptual layer
  • Mapping layer
  • Logical layer

These three layers allow data to be mapped from a relational database to a more object-oriented business model. The Entity Framework provides the means to define these layers using XML files. It also generates a series of classes based on the schema of the conceptual model. You can program against these classes directly to interact with the data. This provides a level of abstraction so developers can program against the conceptual model instead of the relational model. The Entity Framework maps all commands coded against the conceptual model into the logical model (see Figure 2).

Figure 2 Designing the Entity Data Model
Figure 2 Designing the Entity Data Model(Click the image for a smaller view)
Figure 2 Designing the Entity Data Model
Figure 2 Designing the Entity Data Model (Click the image for a larger view)

The conceptual model is defined in an XML file using the Conceptual Schema Definition Language (CSDL). The CSDL defines the entities and the relationships as the application’s business layer knows them. The logical model, which represents the database schema, is defined in an XML file using the Store Schema Definition Language (SSDL). For example, you might have an entity in a conceptual model that actually derives its data from multiple tables in a database. The conceptual model and the logical model can associate entities one-to-one. However, the power of the EDM is that it does not have to link them one-to-one. The mapping layer, which is defined using the Mapping Schema Language (MSL), maps the other two layers between each other. This mapping is what allows developers to code against the conceptual model and have those instructions mapped into the logical model.

Generating an Entity Data Model

You can generate an EDM using a database as a starting point. You can then modify the XML manually (or possibly using a modeling tool that may be available in a future release of Visual Studio). When you add an ADO.NET EDM to your project, the wizard walks you through the process of creating the EDM.

After choosing to generate the EDM from the sample Northwind database that comes with "Orcas," you are prompted with a list of objects that it can model, as shown in Figure 3. I chose to bring all the tables into the model so the wizard generates CSDL, SSDL, and MSL files for all of the tables in Northwind. Of course, this is a pure one-to-one table-to-entity mapping. I can adjust this mapping to suit my business needs by combining entities or perhaps through inheritance.

Figure 3 The EDM Wizard
Figure 3 The EDM Wizard(Click the image for a smaller view)
Figure 3 The EDM Wizard
Figure 3 The EDM Wizard (Click the image for a larger view)

The wizard also generates a series of classes from the CSDL that represent the domain model. Some of these classes are shown in Figure 4 through the Class View window. Due to the one-to-one mapping with which I started, there is a class for every table. The EDM wizard picked up on the relationship between Customers and Orders in the database and created a corresponding association for them in the conceptual model. Therefore, the Customers class contains a navigation property, called Orders, that allows developers to navigate from a Customers instance down to any associated Orders instances for a customer.

Figure 4 Classes
Figure 4 Classes

Dissecting the CSDL

The metadata residing within the CSDL file contains a list of entities, represented by the EntityType element, and of relations, represented by the Association element, which is an AssociationType (note that this is an inconsistency in the naming scheme within Beta 1). A fragment of the CSDL file I generated earlier is shown in Figure 5. The entities contain a list of scalar properties that define the entity. The Key attribute indicates which properties are the key for the entity. Compound keys are indicated by separating each property name with a space. Entities can also contain a special type of property called a NavigationProperty. This defines how to navigate from one entity following the associations to another entity.

Figure 5 EntityType Definition in CSDL (Close Figure 5)
<EntityType Name=”Customers” Key=”CustomerID”>
    <Property Name=”CustomerID” Type=”String” Nullable=”false”
MaxLength=”4000” FixedLength=”true” />
    <Property Name=”CompanyName” Type=”String” Nullable=”false”
MaxLength=”4000” />
    <Property Name=”ContactName” Type=”String” MaxLength=”4000” />
    <Property Name=”ContactTitle” Type=”String” MaxLength=”4000” />
    <Property Name=”Address” Type=”String” MaxLength=”4000” />
    <Property Name=”City” Type=”String” MaxLength=”4000” />
    <Property Name=”Region” Type=”String” MaxLength=”4000” />
    <Property Name=”PostalCode” Type=”String” MaxLength=”4000” />
    <Property Name=”Country” Type=”String” MaxLength=”4000” />
    <Property Name=”Phone” Type=”String” MaxLength=”4000” />
    <Property Name=”Fax” Type=”String” MaxLength=”4000” />
    <NavigationProperty Name=”Orders”
Relationship=”NorthwindModel.FK_Orders_Customers”
FromRole=”Customers” ToRole=”Orders” />
  </EntityType>

The AssociationType between the Customer and its Orders is defined in this CSDL fragment:

  <Association Name=”FK_Orders_Customers”>
     <End Role=”Customers” Type=
         ”NorthwindModel.Customers”
         Multiplicity=”0..1” />
     <End Role=”Orders” Type=
         ”NorthwindModel.Orders” Multiplicity=”*” />
  </Association>

The End elements of the AssociationType indicate the participants of the association. In this example, a Customers entity is associated with an Orders entity. A Customers entity can also be related to any number of Orders entities, as defined by the multiplicity.

While the EntityType and AssociationType elements define the types of domain entities and relationships, the EntitySet and AssociationSet elements define the scope of them. All "sets" that should be logically grouped together are contained within an EntityContainer element. (For the full CSDL, see the NorthwindEntities.csdl file in the accompanying download.)

The following CSDL fragment shows the EntityContainer and some of its contents:

<EntityContainer Name=”NorthwindEntities”>
    <EntitySet Name=”Customers”
        EntityType=”NorthwindModel.Customers” />
    <EntitySet Name=”Orders”
        EntityType=”NorthwindModel.Orders” />
    <AssociationSet Name=”FK_Orders_Customers”
            Association=”NorthwindModel.FK_Orders_Customers”>
        <End Role=”Customers” EntitySet=”Customers” />
        <End Role=”Orders” EntitySet=”Orders” />
    </AssociationSet>

</EntityContainer>

This fragment shows the EntitySets for the EntityTypes Customers and Orders. The AssociationSet FK_Orders_Customers is also declared here. This fragment is what defines the entities Customers and Orders as well as the relationship between them.

Mapping to the Store

The SSDL file defines the structure of the relational data in the database. It also uses the EntityType and AssociationType XML elements, in this case to declare the structures of tables and foreign keys that exist in the database, respectively. The SSDL file’s namespace is set based on the name of the database used in the EDM while its EntityContainer element is named after the database schema. The EntityContainer contains a series of EntitySet and AssociationSet elements, which declare the instances of the tables and relationships represented by the EntityType and AssociationType. For every EntitySet in the SSDL file there is a single corresponding table in the database.

If you generate an EDM from a database and immediately open the CSDL and SSDL files without making modifications to them, you will find that they are strikingly similar. This is because the models are generated directly from the database so the conceptual model is mapped directly to the logical store. The MSL file contains the direct map from the CSDL to the SSDL. All queries written against the EDM are translated into generated SQL commands. The Entity Framework also supports using stored procedures instead of generated SQL queries.

An EntityContainerMapping element is used to map a model (the CSDL) to the store (the SSDL). The attribute StorageEntityContainer indicates the name of the EntityContainer in the store while the EdmEntityContainer attribute indicates the corresponding EntityContainer in the model. Mapping a model’s EntitySet to a store’s EntitySet requires an EntitySetMapping element. The Name attribute defines the name of the EntitySet in the model while the TableName attribute defines the name of the corresponding EntitySet in the store. Each property in the model is mapped to the store through a ScalarProperty element. Figure 6 shows an MSL fragment.

Figure 6 Mapping the Inherited Entity in the MSL (Close Figure 6)
<cs:EntitySetMapping cs:Name=”Products”>
  <cs:EntityTypeMapping cs:TypeName=”NorthwindModel.Products”>
    <cs:TableMappingFragment cs:TableName=”Products”>
      <cs:ScalarProperty cs:Name=”ProductID” cs:ColumnName=”ProductID” />
      <cs:ScalarProperty cs:Name=”ProductName”
          cs:ColumnName=”ProductName” />
      <cs:ScalarProperty cs:Name=”QuantityPerUnit”
          cs:ColumnName=”QuantityPerUnit” />
      <cs:ScalarProperty cs:Name=”UnitPrice” cs:ColumnName=”UnitPrice” />
      <cs:ScalarProperty cs:Name=”UnitsInStock”
          cs:ColumnName=”UnitsInStock” />
      <cs:ScalarProperty cs:Name=”UnitsOnOrder”
          cs:ColumnName=”UnitsOnOrder” />
      <cs:ScalarProperty cs:Name=”ReorderLevel”
          cs:ColumnName=”ReorderLevel” />
      <!--<cs:ScalarProperty cs:Name=”Discontinued”
          cs:ColumnName=”Discontinued” />-->
      <cs:Condition cs:ColumnName=”Discontinued” cs:Value=”0”/>
    </cs:TableMappingFragment>
  </cs:EntityTypeMapping>
<!--</cs:EntitySetMapping>
<cs:EntitySetMapping cs:Name=”DiscontinuedProducts”>-->
  <cs:EntityTypeMapping cs:TypeName=
        ”NorthwindModel.DiscontinuedProducts”>
    <cs:TableMappingFragment cs:TableName=”Products”>
      <cs:ScalarProperty cs:Name=”ProductID” cs:ColumnName=”ProductID” />
      <cs:ScalarProperty cs:Name=”ProductName”
          cs:ColumnName=”ProductName” />
      <cs:ScalarProperty cs:Name=”QuantityPerUnit”
          cs:ColumnName=”QuantityPerUnit” />
      <cs:ScalarProperty cs:Name=”UnitPrice” cs:ColumnName=”UnitPrice” />
      <cs:ScalarProperty cs:Name=”UnitsInStock”
          cs:ColumnName=”UnitsInStock” />
      <cs:ScalarProperty cs:Name=”UnitsOnOrder”
          cs:ColumnName=”UnitsOnOrder” />
      <cs:ScalarProperty cs:Name=”ReorderLevel”
          cs:ColumnName=”ReorderLevel” />
      <cs:Condition cs:ColumnName=”Discontinued” cs:Value=”1”/>
    </cs:TableMappingFragment>
  </cs:EntityTypeMapping>
</cs:EntitySetMapping>

Defining Inheritance

The EDM also supports models that are not purely one-to-one with the database. For example, using the Northwind database, you can create a class called DiscontinuedProducts, which inherits all properties from the Products class but only contains products whose Discontinued column is 1. (Note that the DiscontinuedProducts class could also add additional properties.) This is a simplistic inheritance scenario, but it demonstrates how to implement inheritance using the EDM.

The first step in creating the DiscontinuedProducts class in the conceptual model is to open the CSDL file and create a new EntityType, name it DiscontinuedProducts, and set its BaseType attribute to NorthwindModel.Products (the schema and base EntityType name). The derived EntityType inherits the properties of the Products EntityType including its Keys. Therefore, I do not specify the Key attribute or the properties for the new derived EntityType. I also comment out the Discontinued property from the Products EntityType. The additional code needed to do all of this in the CSDL file looks like this:

<EntityType Name=”DiscontinuedProducts”
    BaseType=”NorthwindModel.Products”/>

The next step in this process is to open the MSL file and find the Products EntitySetMapping element and remove its TypeName and TableName attributes. These will now be set for each specific EntityType. Then create an EntityTypeMapping child element and set its TypeName to NorthwindModel.Products. For each EntityType that inherits from a base EntityType, the EntitySetMapping must include an EntityTypeMapping element. Create a child of the EntityTypeMapping element called TableMappingFragment and set its TableName attribute to Products. Basically, these steps move the mapping from the EntitySetMapping element down to a more granular level.

Comment out the Discontinued property mapping and add a Condition element that indicates that only records with Discontinued equal to 0 will be included. Copy the entire EntityTypeMapping XML fragment, change the Name attribute to DiscontinuedProducts, and change the condition to a value of 1. The new fragment of the MSL file is shown in Figure 6.

Mapping across Multiple Tables

Another way that the EDM can vary from the pure one-to-one model to store mapping is to map a single entity in the model to multiple tables in the store. The Contacts table and the ContactNameSplit table in the sample Northwind database have a one-to-one relationship between them and can be combined into a single entity in the model. For this example, I will create an entity in the model that includes all columns from the Contacts table and the Title and Name columns from the ContactNameSplit table.

The first change is to modify the Contacts EntityType in the CSDL file to include two additional properties: Name and Title.

The next set of changes is a bit more verbose. These two new properties in the model must now be mapped to the store with the MSL file. The EntitySetMapping element for the Contacts EntitySet must be changed to represent a mapping to multiple tables. For this example, I modified the existing EntitySetMapping tag for the Contacts EntitySet by removing the TableName and TypeName attributes. These attributes are only declared in the EntitySetMapping element if the EntitySet in the model maps one-to-one with an EntitySet in the store.

Since the mapping from the model EntitySet to the store EntitySet for Contacts has been removed, a replacement must be created. This replacement is a child element called EntityTypeMapping. Two of these must be created—one each to represent the Contacts and the ContactNameSplit from the store. The EntityTypeMapping elements define the TypeName attribute for each of the EntitySets.

Inside each of the EntityTypeMapping elements is a child element called TableMappingFragment. This element contains the TableName attribute, which corresponds to the store’s EntitySet. The TableMappingFragment is where all of the ScalarProperty elements are defined that map the model’s properties to the store. Figure 7 shows the revised Contacts EntitySetMapping that now maps the Contacts and ContactSplitName tables from the store into a single EntitySet in the model.

Figure 7 Combining Tables into an Entity (Close Figure 7)
<cs:EntitySetMapping cs:Name=”Contacts”>
  <cs:EntityTypeMapping cs:TypeName=”NorthwindModel.Contacts”>
    <cs:TableMappingFragment cs:TableName=”Contacts”>
      <cs:ScalarProperty cs:Name=”ContactID”
          cs:ColumnName=”ContactID” />
      <cs:ScalarProperty cs:Name=”ContactType”
          cs:ColumnName=”ContactType” />
      <cs:ScalarProperty cs:Name=”CompanyName”
          cs:ColumnName=”CompanyName” />
      <cs:ScalarProperty cs:Name=”ContactName”
          cs:ColumnName=”ContactName” />
      <cs:ScalarProperty cs:Name=”ContactTitle”
          cs:ColumnName=”ContactTitle” />
      <cs:ScalarProperty cs:Name=”Address” cs:ColumnName=”Address” />
      <cs:ScalarProperty cs:Name=”City” cs:ColumnName=”City” />
      <cs:ScalarProperty cs:Name=”Region” cs:ColumnName=”Region” />
      <cs:ScalarProperty cs:Name=”PostalCode”
          cs:ColumnName=”PostalCode” />
      <cs:ScalarProperty cs:Name=”Country” cs:ColumnName=”Country” />
      <cs:ScalarProperty cs:Name=”Phone” cs:ColumnName=”Phone” />
      <cs:ScalarProperty cs:Name=”Extension”
          cs:ColumnName=”Extension” />
      <cs:ScalarProperty cs:Name=”Fax” cs:ColumnName=”Fax” />
      <cs:ScalarProperty cs:Name=”PhotoPath”
          cs:ColumnName=”PhotoPath” />
    </cs:TableMappingFragment>
  </cs:EntityTypeMapping>
  <cs:EntityTypeMapping cs:TypeName=”ContactNameSplit”>
    <cs:TableMappingFragment cs:TableName=”ContactNameSplit”>
      <cs:ScalarProperty cs:Name=”ContactID” cs:ColumnName=”ID” />
      <cs:ScalarProperty cs:Name=”Name” cs:ColumnName=”Name” />
      <cs:ScalarProperty cs:Name=”Title” cs:ColumnName=”Title” />
    </cs:TableMappingFragment>
  </cs:EntityTypeMapping>

Using EntityClient

Accessing the Entity Framework’s conceptual model can be done through any of three different mechanisms (see Figure 1). Here I will introduce EntityClient, the new .NET data provider.

EntityClient is abstracted from the logical store as it communicates with the conceptual model using its own text-based language called Entity SQL. All Entity SQL queries executed using EntityClient are compiled into command trees that are sent to the store. The conversion of the queries from Entity SQL through the conceptual model and down to the store are handled by the Entity Framework.

The classes in EntityClient are similar to the classes in common ADO.NET providers. For example, EntityClient queries are executed with an EntityCommand object, which requires an EntityConnection object to connect to the EDM. While EntityClient interacts with the entities in the EDM, it does not return instances of the entities but instead returns all results in a DbDataReader object. EntityClient can return a standard set of rows and columns or it can return a representation of more-complex hierarchical data through a DbDataReader.

Figure 8 shows an example of using EntityClient to connect to the conceptual model and retrieve a list of Customers from London. The EntityConnection can accept a full connection string to the conceptual layer or the name of the connection string in the App.Config file. The connection string contains a list of the metadata files (the CSDL, MSL, and SSDL files) as well as the database-specific connection string information for the store. An example of the full connection string for Figure 8 is shown here:

Figure 8 Entity Client Using Entity SQL to Get to EDM (Close Figure 8)
string city = “London”;
using (EntityConnection cn =
    new EntityConnection(“Name=NorthwindEntities”))
{
    cn.Open();
    EntityCommand cmd = cn.CreateCommand();
    cmd.CommandText =
         “SELECT VALUE c FROM NorthwindEntities.Customers “ +
         “AS c WHERE c.City = @city”;
    cmd.Parameters.AddWithValue(“city”, city);
    DbDataReader rdr = cmd.ExecuteReader(
        CommandBehavior.SequentialAccess);
    while (rdr.Read())
        Console.WriteLine(rdr[“CompanyName”].ToString());
    rdr.Close();
}

“metadata=.\NorthwindEntities.csdl|.\NorthwindEntities.ssdl|.
\NorthwindEntities.msl;provider=System.Data.SqlClient;provider connection string=’Data Source=DDVPC01
\SQLEXPRESS;Initial Catalog=Northwind;
Integrated Security=True’”

The code in Figure 8 shows how to create an EntityConnection object and execute an EntityCommand on it. The query written with Entity SQL is referring to the Customers EntitySet in the EDM. Note that the syntax is intentionally similar to T-SQL. (If you want a good reference for the Entity SQL syntax, see the MSDN® documentation for the Beta 1 release of "Orcas.")

As shown in Figure 8, EntityParameter objects can also be added. As of Beta 1, EntityClient does not support DML queries; it is, however, being worked on for the future. Still, DML can be executed using other means, such as LINQ to Entities.

Using Object Services

Another way to interact with the data represented by the EDM is to use Object Services. Object Services provides a way to load objects and navigate any relationships defined in the EDM. As illustrated in Figure 1, Object Services uses EntityClient to get at the data. Object Services adds identity resolution, which is a manual process when using a DataSet. It also provides object persistence and change tracking through events to allow an explicit load and save. This reduces the round-trip to the server.

Object Services allows lists of objects to be returned directly—both projections and defined entities can be returned. For example, using Object Services, you could retrieve a List<Customers> as defined in the EDM. The Customers objects can be examined, the values changed, and then the data saved back to the database.

If you use projections with Object Services, the data returned will not be an updateable object. Because projections return specific properties of entities and not the entire entity, Object Services cannot save updates to the projected data back to the database. If you intend to update the data, a better option is to return the entire entity rather than using a projection.

You can use Object Services to execute queries using Entity SQL or you can write queries using LINQ to Entities. The following example demonstrates querying with Object Services and Entity SQL to retrieve a list of Customers:

string city = “London”;
ObjectQuery<Customers> query = northwindContext.CreateQuery<Customers>(
    “SELECT VALUE c FROM Customers AS c WHERE c.City = @city”,
     new ObjectParameter(“city”, city));
foreach (Customers c in query) Console.WriteLine(c.CompanyName);

In the EDM, the EntityContainer is represented by a class that inherits from ObjectContext (northwindContext in this example). The ObjectContext class implements the ObjectQuery<T> interface, allowing it to create the queries using Entity SQL or LINQ.

The CreateQuery method accepts a parameterized Entity SQL statement that defines a query that will retrieve a list of Customers entities. The actual SQL statement that hits the database is executed when ObjectQuery<Customers> is iterated through using a foreach statement.

Using LINQ to Entities

Dynamic queries can be written in Entity SQL and used with Object Services to interact with the EDM entities. But the Entity Framework can also be used with strongly typed EDM classes using LINQ to Entities. For example, in the sample just shown, querying with Object Services and Entity SQL can be modified to be executed using LINQ to Entities, like this:

string city = “London”;
var query = from c in northwindContext.Customers
            where c.City == city
            select c;
foreach (Customers c in query) Console.WriteLine(c.CompanyName);

This code sample replaces all of the text-based syntax of Entity SQL with the strongly typed LINQ syntax supported by C# 3.0. For more information on LINQ and support for it in both C# and Visual Basic®, see the June 2007 issue of MSDN Magazine at msdn.microsoft.com/msdnmag/issues/07/06.

Now recall when I created an EntityType in the EDM, called DiscontinuedProducts, that inherits from Products. The inheritance features of the EDM can be used with LINQ and Object Services to retrieve a list of discontinued products. Note that the following example does not specify where discontinued equals 1. Instead, the type for the product entity is checked against the derived EntityType DiscontinuedProducts, which in turn uses the condition I created in the mapping (MSL file) to generate an appropriate SQL statement to only retrieve discontinued products:

var query = from p in northwindContext.Products
            where p is DiscontinuedProducts
            select p;
foreach (Products p in query) Console.WriteLine(p.ProductName);

You can also write queries that take advantage of the built-in relationships in the EDM (defined by AssociationSet). For example, a list of Orders for Customers in London can be retrieved using the following LINQ query expression:

var query = from o in northwindContext.Orders
            where o.Customers.City == “London”
            select o;
foreach (Orders o in query) Console.WriteLine(o.OrderID);

This code sample starts with the Orders entity and uses its navigation property, called Customers, to be able to examine the City property. The Orders from customers not in London are filtered out entirely. Since a list of Orders entities are returned, the entities can also be modified and their changes saved to the database. Saving changes to the database can be done through the SaveChanges method.

The following example gets the list of customers in London and sets the Country property for each of them to United Kingdom. The changes are persisted to the StateManager, but they are not written to the database until the SaveChanges method is executed:

var query = from c in northwindContext.Customers
            where c.City == “London”
            select c;
foreach (Customers c in query) c.Country = “United Kingdom”;
northwindContext.SaveChanges();

You can also create a new instance of an entity and add it to the EDM using ObjectContext’s AddObject method. The following example shows how to add a new category to the Categories table in the database:

Categories newCat = new Categories();
newCat.CategoryName = “Other”;
northwindContext.AddObject(newCat);
northwindContext.SaveChanges();
int newCatID = newCat.CategoryID;

First an instance of the Categories entity is created and its CategoryName property is set. Then the new category is added to the EDM using the AddObject method. When the SaveChanges method is invoked, a SQL statement is generated that inserts the new Category into the database and returns the CategoryID for the new row.

When entities have relationships to other entities, you may want to associate the new entity with an existing entity. For example, you can create a new Orders entity and set its Customers property to an existing Customers entity. While the Orders table in the database has a CustomerID column, the EDM represents the relationship through an object oriented approach using a Customers property to navigate to the related Customers entity:

Orders newOrder = new Orders();
newOrder.OrderDate = DateTime.Today;
Customers cust = northwindContext.Customers.Where(
    “it.CustomerID = ‘ALFKI’”).First();
newOrder.Customers = cust;
northwindContext.AddObject(newOrder);
northwindContext.SaveChanges();

Wrapping Up

The Entity Framework allows developers to focus on the data through an object model instead of the logical/relational data model. Once the EDM is designed and mapped to a relational store, the objects can be interacted with using a variety of techniques including EntityClient, ObjectServices, and LINQ.

While the traditional objects such as the DataSet, DataAdapter, DbConnection, and DbCommand are still supported in the upcoming release of ADO.NET available in Visual Studio "Orcas," the Entity Framework brings major additions that open ADO.NET to new and exciting possibilities.

Send your questions and comments for John mmdata@microsoft.com.

John Papa is a senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences such as VSLive or blogging at codebetter.com/blogs/john.papa or at www.johnpapa.net.


Cutting Edge: Context-Sensitive Feedback with AJAX

Cutting Edge
Context-Sensitive Feedback with AJAX
Dino Esposito

Code download available at: CuttingEdge2007_07.exe (170KB)
Contents
Built-In Tools for Update Progress
Limitations of UpdateProgress
The Progress Indicator Pattern
The Progress Monitor Server API
Building a Sample Page
The Progress Monitor Client API
The Event Sink for Client Calls
Wrapping Up a Sample Page
Tricks for UpdatePanel Controls
Summary

When users of any computer application start a potentially lengthy operation, best practices dictate that the user interface should be updated to indicate that work is in progress and that results may not be available for a while. This is easy to accomplish in a desktop application, but it can be rather difficult in a Web scenario. In Web applications, displaying text such as "Please wait" just before an operation begins is easy, but if you want to provide some useful feedback, such as the estimated time to completion or the percentage of work finished, it’s a bit more difficult to accomplish. This is because most lengthy tasks in Web apps execute on the server, and there are no built-in mechanisms for pushing state information to the client.

In my September and October 2006 Cutting Edge columns, I discussed progress bars in ASP.NET Web applications. In the first column, I focused on a pure ASP.NET 2.0 solution using ASP.NET script callbacks. In the second column, I built a solution based on ASP.NET AJAX Extensions. At the time of that writing, ASP.NET AJAX was still in a pre-beta stage and, as you probably know, a good number of things have changed since then. In this month’s column, I revisit the topic and build a server and client ASP.NET AJAX infrastructure to start and control a potentially lengthy server-side task.

Built-In Tools for Update Progress

ASP.NET AJAX Extensions support two different programming models: partial rendering and direct remote method calls. Partial rendering is an evolution of the traditional, control-based programming model of ASP.NET. It lets you define regions within the page that can be updated independently from one another and from the page as a whole. Postback events that relate to a particular region, or to a subset of regions, trigger the generation of markup for the controls in the region(s). As a result, only a fraction of the page is updated or refreshed. No new page is displayed when these controls are refreshed (unless you explicitly navigate to another page)—but the Document Object Model (DOM) of the currently displayed page is updated.

With partial rendering, you still use codebehind methods and server-side event handlers to define the expected behavior of a page. The partial refresh of pages is entirely managed by the ASP.NET AJAX runtime environment. Although a few client events are publicly exposed, partial rendering is essentially a black-box process. Users click buttons, the system determines that a postback is necessary, and it uses the client-side AJAX infrastructure to make an asynchronous request. When data is ready, the DOM of the current page is automatically updated.

Because the server-side postback event may take a while to complete, ASP.NET AJAX provides the UpdateProgress control to keep the user informed. Figure 1 lists a sample ASP.NET AJAX page that uses partial rendering and displays a progress screen.

Figure 1 Page with UpdatePanel and UpdateProgress Controls (Close Figure 1)
<%@ Page CodeFile=”LongUpdates.aspx.cs” Inherits=”LongTask” %>

<html>
<head runat=”server”>
    <title>Lengthy Operations and Updates</title>
    <style type=”text/css”>
        #UpdateProgress1
        {
            width: 30%; background-color: #FFC080; height:15%;
            top: 40%; left: 35%; position: absolute;
            border: solid 1px black;
        }
        #ProgressTemplate1
        {
            font-weight: bold; font-size: 12pt; color: navy;
            font-family: Verdana; background-color: #ffff99;
        }
    </style>
</head>

<body>
<form id=”form1” runat=”server”>
        <asp:ScriptManager ID=”ScriptManager1” runat=”server” />

        <asp:UpdatePanel ID=”UpdatePanel1” runat=”server”
                         UpdateMode=”Conditional”>
            <ContentTemplate>
                <asp:Button runat=”server” ID=”Button1”
                  Text=”Start Task ...” onclick=”Button1_Click” />
                <hr />
                <asp:Label runat=”server” ID=”Label1” /><br />
            </ContentTemplate>
        </asp:UpdatePanel>

        <hr />

        <asp:UpdateProgress runat=”server” ID=”UpdateProgress1”>
            <ProgressTemplate>
                <div ID=”ProgressTemplate1”>
                   <img alt=”” src=”Images/indicator.gif” />
                   <span id=”Msg”>Please, wait ... </span>
                </div>
                <br /><br />
            </ProgressTemplate>
        </asp:UpdateProgress>
    </form>
</body>
</html>

The UpdateProgress control is designed to automatically provide visual feedback in the browser window when the contents of one or more UpdatePanel controls are updated. As a page developer, you don’t have to worry about showing and hiding the control explicitly. All that you have to do is define the visual template to show to the user and its style and position. The ASP.NET AJAX client infrastructure takes care of everything else, showing and hiding the template as appropriate (see Figure 2).

Figure 2 UpdateProgress Control in Action
Figure 2 UpdateProgress Control in Action(Click the image for a smaller view)
Figure 2 UpdateProgress Control in Action
Figure 2 UpdateProgress Control in Action (Click the image for a larger view)

The UpdateProgress control is easy to use and particularly suited for relatively fast operations that complete in just a few seconds. For longer operations—those that take more than a few seconds—you should look for a more comprehensive solution.

Limitations of UpdateProgress

UpdateProgress is a powerful control with many handy properties, but it can’t obtain server process information because no interaction between the UpdateProgress control and the code being executed is supported. Plus, UpdateProgress can’t display a percentage progress bar or a list of messages describing the current stage of the server operation.

Another reason for looking elsewhere is that the UpdateProgress control is tightly coupled with both the partial rendering engine and the UpdatePanel control itself, so it won’t work if you use your ASP.NET AJAX pages to invoke a server method either via local Web services or page methods. You can reuse the same visual template, however, and show and hide it via script code. For example, the progress template of Figure 1 becomes a simple <div> tag as shown here:

<div id=”UpdateProgress1” style=”display:none”>
    <div id=”ProgressTemplate1”>
        <img alt=”” src=”images/indicator.gif” />
        <span id=”Msg”>Please, wait ... </span>
    </div>
</div>

Figure 3 shows the JavaScript code required to manage the progress template when a remote call is made to a server method. Let’s see how to build a client and server infrastructure that returns timely and informative feedback when you execute partial updates and remote method calls.

Figure 3 Show and Hide a Visual Progress Template (Close Figure 3)
<script type=”text/javascript”>
function startTask()
{
    // Display the progress screen
    enableProgress(true);

    // Invoke the remote method (ExecuteTask in codebehind class)
    PageMethods.ExecuteTask(taskCompleted, taskFailed);
}

function taskCompleted(results, context, methodName)
{
    // Hide the progress screen and update the UI
    enableProgress(false);
    $get(“Label1”).innerHTML = results;
}

function taskFailed(results, context, methodName)
{
    // Hide the progress screen
    enableProgress(false);
}

function enableProgress(shouldShow)
{
    var template = $get(“UpdateProgress1”);
    if (shouldShow)
        template.style.display = “”;
    else
        template.style.display = “none”;
}
</script>

The Progress Indicator Pattern

Whatever approach you take to execute a partial page refresh, no information is pushed to the client during the operation. As a result, there’s no easy way for a client to grab status information and update a progress bar. You can, however, create a custom server operation that can write its current status to a known location as it makes progress. A second operation would then read periodically from that location and report the status to the client.

As an example, imagine a server operation that goes through a number of distinct steps. At the beginning of each step, the server code updates a durable store with some task-related information. Concurrently, a client-controlled monitoring service reads the text and brings it back to the client for display. This behavior is associated with one of the most popular AJAX patterns—the Progress Indicator pattern. (See www.ajaxpatterns.org for further information on common AJAX design patterns.)

But there’s another common scenario to consider. Say the user triggers a server operation that will loop through a sequence of records. If you accomplish all the work in a stored procedure using one large, set-based transaction, you’re issuing just one database request, resulting in great performance. However, if giving feedback to the user is essential, you might want to break the original transaction into multiple steps controlled by the middle-tier. The state of the transaction could then be properly exposed to the monitoring service before each step is processed and you could inform users that the system has completed, say, 35 percent of the work. Figure 4 provides a graphical view of the Progress Indicator pattern.

Figure 4 Progress Indicator Pattern
Figure 4 Progress Indicator Pattern(Click the image for a smaller view)
Figure 4 Progress Indicator Pattern
Figure 4 Progress Indicator Pattern (Click the image for a larger view)

Implementing the Progress Indicator pattern in ASP.NET AJAX is a three-step process. You define the API that reads and writes the status information from a persistent store, such as a database, a disk file, or a shared block of memory. Next, you provide an event sink for the client to connect via XMLHttpRequest to the server and read the current status. Finally, you set up a client JavaScript API that represents the monitoring service that periodically connects to the server to measure progress. Note that the pattern should be implemented to work regardless of the technique you use to start the partial page update—be it the UpdatePanel control, a page, or a Web service method.

The Progress Monitor Server API

To read and write status information from a known container, you need a server-side, contract-based API. In Figure 5 you see the source code of a ProgressMonitor class that implements a given interface. The interface features two methods—GetStatus and SetStatus—to read and write the status of the task. The choice of the data container is ultimately up to the class author. The ProgressMonitor class in Figure 5 saves data to the ASP.NET cache. The SetStatus method adds a new item and gives it an absolute expiration policy of five minutes. The value can be modified to suit your needs. You can also make it an external parameter and have the class read the cached item duration from the configuration file. An explicit expiration date is helpful so that the ASP.NET cache doesn’t fill up with too many unnecessary items.

Figure 5 ProgressMonitor Server Class (Close Figure 5)
namespace Samples.Server
{
    public interface IProgressMonitor
    {
        void SetStatus(int taskID, object message);
        string GetStatus(int taskID);
    }

    public class ProgressMonitor : IProgressMonitor
    {
        // Sets the current status of the task
        public void SetStatus(int taskID, object message)
        {
            HttpContext.Current.Cache.Insert(
                taskID.ToString(),
                message,
                null,
                DateTime.Now.AddMinutes(5),
                Cache.NoSlidingExpiration);
        }

        // Reads the current status of the task
        public string GetStatus(int taskID)
        {
            object o = HttpContext.Current.Cache[taskID.ToString()];
            if (o == null)
                return String.Empty;

            return (string) o;
        }
    }
}

The ProgressMonitor class must be visible to the ASP.NET server-side code that runs and controls the lengthy task. The code below shows an ASP.NET AJAX page method exposed to the JavaScript client. When invoked, the method triggers the server-side progress monitor to store its status to a server data container—in this case, the ASP.NET cache.

[WebMethod]
public static string ExecuteTask(int taskID)
{
   ProgressMonitor progMonitor = new ProgressMonitor();
   progMonitor.SetStatus(taskID, “5% done”);
   DoStep1();
   progMonitor.SetStatus(taskID, “25% done”);
   DoStep2();
   ...
   progMonitor.SetStatus(taskID, “100% done”);
}

Similar code is required in the body of any ASP.NET AJAX Web service method exposed to the client for direct calls. As another example, here’s a server event handler for when an UpdatePanel control is used to trap postbacks and other partial page updates.

protected void Button1_Click(object sender, EventArgs e)
{
    ProgressMonitor progMonitor = new ProgressMonitor();
    int taskID = RetrieveTaskID();

    progMonitor.SetStatus(taskID, “15% done”);
    DoStep1();
    ...
}

The SetStatus method takes a value and stores it in the internal data container. The value can be a string as well as a number. Generally speaking, you can express the status of a task using any Microsoft® .NET Framework object. However, if you use something other than a primitive type, you should then guarantee that it can be correctly transformed into a JavaScript object. This means that status information must be serialized to a JavaScript Object Notation (JSON) stream and a JavaScript class with an analogous structure must be available on the client. Finally, you need to preprocess the returned JSON string to get an instance of that class properly initialized.

Because two distinct software entities—the server-side progress monitor and the client-side monitoring service—operate on the status of the same task, a unique identifier for the task is required. The task ID is generated as a random number on the client and passed as an additional argument to the remote method. The task ID is used as the key to set and retrieve status information in whatever data container you decide to use.

Building a Sample Page

To see how it all works, let’s consider a sample ASP.NET AJAX page with a button. The user clicks the button and a remote method call is made. To monitor the progress, you need the server method to use the ProgressMonitor class and receive a task ID from the client. Here’s the JavaScript code that starts the remote method:

function startTask() {
    // Start the remote task using a page method
    var taskID = progressManager.getTaskID();
    PageMethods.ExecuteTask(taskID, taskCompleted, taskFailed);

    // Enable the monitoring service
    updateUI(true);
    progressManager.startMonitor(taskID, 2000, updateProgress,
        updateProgressCompleted);
}

ExecuteTask is the name of the ASP.NET AJAX page method to invoke from the client. This method takes a task ID in addition to all input parameters it needs and the usual callbacks for success and failure. In the preceding code snippet, the updateUI JavaScript function is a local function that brings up the progress screen.

The progressManager object in the listing is implemented as a JavaScript class and is based on the Microsoft AJAX library. It represents the client-side monitoring service. The getTaskID method returns the ID for the task being started. The task ID is a random number generated using a JavaScript helper class. The startMonitor method sets a timer that periodically sends a remote call to the server page. The server code receives the task ID, connects to the data container (the ASP.NET cache), reads the current status of the specified task, and returns any value found. Once status information has been downloaded to the client, the updateProgress callback is invoked to update the client UI. The updateProgressCompleted callback is invoked when the task is completed to clear the progress screen.

The progressManager object is initialized when the ASP.NET AJAX page is loaded. You use the pageLoad JavaScript function to define any code you want to execute upon page loading once the Microsoft AJAX library has been fully initialized and set up.

var progressManager = null;
function pageLoad() {
   progressManager = new Samples.Progress();
}

If you need to accomplish ASP.NET AJAX-specific tasks upon loading, the pageLoad method is more useful than the body’s onload event. The Samples.Progress class is the client-side monitoring service that periodically checks the status of the task and updates the user interface.

The Progress Monitor Client API

The Samples.Progress class is defined across a couple of JavaScript files—progress.js and random.js. Both are to be bound to the page either using the <script> tag or the ScriptManager control, as shown here:

<asp:ScriptManager ID=”ScriptManager1” runat=”server”
     EnablePageMethods=”true”>
  <Scripts>
      <asp:ScriptReference path=”random.js” />
      <asp:ScriptReference path=”progress.js” />
  </Scripts>
</asp:ScriptManager>

Both script files take advantage of the Microsoft AJAX client library. Their classes are defined in the same namespace and built using the extended set of object-oriented capabilities provided by the Microsoft AJAX client library. The classes use the prototype model and require an explicit constructor (for more information on the prototype model, see Ray Djajadinata’s article on JavaScript in the May 2007 issue of MSDN® Magazine.). The random.js file defines the Samples.Random class. The class has only one method—the one that generates a random number using the JavaScript native Math object, as you see here:

Type.registerNamespace(‘Samples’);
Samples.Random = function Samples$Random() {
    Samples.Random.initializeBase(this);
}
function Samples$Random$getNumber(minNumber, maxNumber) {
    var num = minNumber + Math.floor(Math.random() * maxNumber);
    return num;
}
Samples.Random.prototype = {
    getNumber:  Samples$Random$getNumber
}
Samples.Random.registerClass(‘Samples.Random’);

The Samples.Progress class, on the other hand, does much more (see Figure 6). The class has three essential public methods—getTaskID, startMonitor, and stopMonitor—plus a fourth method that gets invoked only when an UpdatePanel control is used to trigger the potentially lengthy task. As mentioned, getTaskID returns a randomly generated number used to uniquely identify the task. If you don’t trust the JavaScript Math object, you can place an additional server-side call to a page method or a Web service and then ask for a GUID.

Figure 6 Samples.Progress JavaScript Class (Close Figure 6)
//
// Samples.Progress ::
// Client-side monitoring service
//

// Define the namespace of the class
Type.registerNamespace(‘Samples’);

// Constructor
Samples.Progress = function Samples$Progress()
{
    Samples.Progress.initializeBase(this);
    this._timerID = null;
    this._taskID = null;
    this._msInterval = null;
    this._progressCallback = null;
    this._callback = null;
}

// Start the timer to periodically check the status of the ongoing task
function Samples$Progress$startMonitor(taskID, msInterval,
    progressCallback, progressCompletedCallback) {
    if (arguments.length !== 4) throw Error.parameterCount();

    // Update internal members
    _taskID = taskID;
    _msInterval = msInterval;
    _progressCallback = progressCallback;
    _progressCompletedCallback = progressCompletedCallback;

    this._startTimer();
}

// Stop the timer
function Samples$Progress$stopMonitor() {
    window.clearTimeout(_timerID);
    if (_progressCompletedCallback !== null)
        _progressCompletedCallback();
}

// Get task ID
function Samples$Progress$getTaskID(taskID) {
    return Samples.Random.getNumber(0, 10000000);
}

// Start the timer to control progress
function Samples$Progress$_startTimer() {
    this._callback = Function.createDelegate(this, this._checkProgress);
    _timerID = window.setTimeout(this._callback, _msInterval);
}

// Modify the request to add the task ID to a hidden field
//(for UpdatePanel pages)
function Samples$Progress$modifyRequestForTaskId(request, taskID,
    hiddenField) {
    var body = request.get_body();
    var token = “&” + hiddenField + “=”;
    body = body.replace(token, token + taskID);
    request.set_body(body);

    return request;
}

// Timer function(s)
function Samples$Progress$_checkProgress() {
    PageMethods.GetCurrentStatus(_taskID, this._onFeedbackReceived,
        this._onFeedbackFailed, this);
}

function Samples$Progress$_onFeedbackReceived(results, context) {
    context._startTimer();

    if (_progressCallback !== null)
        _progressCallback(results);
}

function Samples$Progress$_onFeedbackFailed(results) {
}

// Class prototype
Samples.Progress.prototype =
{
    getTaskID:              Samples$Progress$getTaskID,
    startMonitor:           Samples$Progress$startMonitor,
    stopMonitor:            Samples$Progress$stopMonitor,
    modifyRequestForTaskId: Samples$Progress$modifyRequestForTaskId,
    _startTimer:            Samples$Progress$_startTimer,
    _checkProgress:         Samples$Progress$_checkProgress,
    _onFeedbackReceived:    Samples$Progress$_onFeedbackReceived,
    _onFeedbackFailed:      Samples$Progress$_onFeedbackFailed
}

// Register the new class
Samples.Progress.registerClass(‘Samples.Progress’);

The startMonitor method takes the ID of the task to monitor and the desired interval in milliseconds. In addition, the method accepts a couple of callbacks—one to update the user interface with status information and one to reset the user interface at the end of the operation.

Implemented through the window’s setTimeout function, the timer calls back an internal method when the interval has elapsed. The method is named _checkProgress and looks like this:

function Samples$Progress$_checkProgress() {
    PageMethods.GetCurrentStatus(_taskID,
           this._onFeedbackReceived,
           this._onFeedbackFailed,
           this);
}

The method does one key thing: it calls the event sink on the server that retrieves the current status of the task. The event sink is a publicly exposed method that the JavaScript client can invoke. In this implementation, I assume it is a page method named GetCurrentStatus. The call to the event sink is a classic ASP.NET AJAX remote method invocation. Hence, it requires callbacks for success and failure and may optionally carry a context object. No special action is required in case of failure; it just won’t update the user interface. Instead, whenever significant status information is downloaded to the client, you need to update the user interface and restart the timer for the next update, as you see here:

function Samples$Progress$_onFeedbackReceived(results, context) {
    context._startTimer();

    if (_progressCallback !== null)
        _progressCallback(results);
}

The success callback receives the Samples.Progress object through the context parameter and restarts the timer. After that, it just invokes the page-defined callback and updates the progress bar. In the ASP.NET page, you start the monitoring service when the user clicks to begin the operation and stop the service from within any callback (success or failure) that runs after the operation has completed.

The Event Sink for Client Calls

Like the remote operation, the monitoring service is based on a remote call that’s repeated periodically. In addition to the endpoint for the potentially lengthy operation, you also need a second public method on the server that can be called from JavaScript to obtain status information. As you know, there are three ways for an ASP.NET AJAX page to expose client-callable endpoints: local Web services, page methods, and UpdatePanel controls. You can use any of these techniques to implement the task to monitor and the monitoring service.

However, note that you can’t use an UpdatePanel bound to a Timer control to periodically refresh the progress bar if you’re already using another UpdatePanel to execute the remote task. By design, UpdatePanel updates execute one at a time under the control of the same page request manager object. This manager object aborts the pending request when a new request is posted. As a result, if you trigger the long task from an UpdatePanel control, and use another timer-based UpdatePanel to provide feedback, the first request for an updated status will abort the operation you want to monitor!

Furthermore, two UpdatePanel updates can’t run concurrently because the view state might be updated improperly if they did. Therefore, to preserve a consistent state of the page, calls going through UpdatePanels must be serialized.

I suggest you consider using an UpdatePanel to start the monitored task, but use a page-specific method to concurrently read its status. You don’t have to put the client-callable page method in each and every page that includes a potentially lengthy operation. It will suffice to inherit any monitorable page from the same base class, as you see here:

public class UpdateProgressPage : System.Web.UI.Page
{
   static ProgressMonitor _progMonitor = new ProgressMonitor();

   [WebMethod]
   public static string GetCurrentStatus(int taskID)
   {
      return _progMonitor.GetStatus(taskID);
   }
}

Note that the name of the public method—GetCurrentStatus—is hardcoded in the progress.js file. If you change the interface of the UpdateProgressPage class, you likewise need to edit the source code of the progress.js script.

Wrapping Up a Sample Page

Let’s outline a step-by-step procedure to add monitoring capabilities to ASP.NET AJAX pages. As an example, imagine that the task is controlled from the codebehind of the page or from some sort of middle-tier where you have access to the progress monitor server API.

You begin by adding the lengthy method to the page as a public page method or as a linked Web service method. Next, you add progress.js and random.js files to the page using <script> tags or the ScriptManager control. The page class must inherit from UpdateProgressPage or expose a GetCurrentStatus static Web method with the following prototype:

public string GetCurrentStatus(int taskID)

At this point, you add the script code to start and monitor the operation. A key element is the callback function you pass to startMonitor to refresh the user interface with the current status. What this function does depends on the markup in the page that is used to show the progress. If all that you have is a <span> tag to display a message, then the following is a good example of a callback:

function updateProgress(perc) {
    $get(“Msg”).innerHTML = perc + “ done.”;
}

The callback function receives a value that indicates the progress made. It can be a string, a percentage, or whatever else you specify through the progress monitor server API. Suppose you signal progress using the following code:

progMonitor.SetStatus(taskID, “5”);

The updateProgress callback function will receive a value of "5". In this case, you’re passing the percentage of work done; as an alternative, you can pass text describing the current stage.

The updateProgress callback is entirely responsible for updating the user interface. If you want to display a classic progress meter, you have to code it here. Here’s one way to do so:

function updateProgress(perc) {
   var table = “<table width=100%><tr><td>{2}%</td></tr>” +
               “<tr><td bgcolor=blue width=’{0}%’>&nbsp;</td>” +
               “<td width=’{1}%’></td></tr></table>”;
   table = String.format(table, perc, 100-perc, perc);
   $get(“Msg”).innerHTML = table;
}

You build a dynamic <table> tag and split one of its rows in two cells. The leftmost cell takes a share of the row equivalent to the work done and is rendered with a different color. Figure 7 shows the final result. The Web Development Helper tool at the bottom of the browser window logs all requests from the browser. (You can get it from Nikhil Kothari’s blog at www .nikhilk.net.) As you can see, the monitoring service issues calls for feedback every two seconds. (See the Timestamp column.) The response of each request consists of a number that indicates the percentage of work done. This number is passed to updateProgress and used in the building of a dynamic HTML table.

Figure 7 Context-Sensitive AJAX Progress Bar in Action
Figure 7 Context-Sensitive AJAX Progress Bar in Action(Click the image for a smaller view)
Figure 7 Context-Sensitive AJAX Progress Bar in Action
Figure 7 Context-Sensitive AJAX Progress Bar in Action (Click the image for a larger view)

The full source code for this column is available from the MSDN Magazine Web site. In the project, you’ll find three ASP.NET AJAX pages, each covering a different scenario. You’ll see the lengthy operation implemented as a page method, as a method on a local Web service, and also triggered within a partial update operation. In all cases, the monitoring service relies on a page method inherited from the page base class.

Tricks for UpdatePanel Controls

To top off this month’s column, let me point out an issue that arises if you use an UpdatePanel control to start and monitor a remote task. As long as you start the operation using script, you generate and pass the task ID explicitly. The server code receives the task ID as an argument and proceeds. When an UpdatePanel control is used, the user starts a task by simply clicking on a postback control, such as a button. So how would you pass a task ID to the server? A hidden field is always a good choice for sending custom data from the client to the server. To add a hidden field, you can either add it explicitly to the page markup or proceed programmatically as below:

void Page_Load(object sender, EventArgs e)
{
    ScriptManager.RegisterHiddenField(UpdatePanel1,
          “HiddenField1”, “”);
}

Why use this code instead of a call to ClientScript.RegisterHiddenField as in regular ASP.NET 2.0 pages? If you use the method on the ScriptManager class, you are guaranteed that the hidden field is maintained during partial postbacks. On the server, you retrieve the task ID using the following code:

int taskID;
Int32.TryParse(Request[“HiddenField1”], out taskID);

But how would you start monitoring? Thankfully, the ASP.NET AJAX client component responsible for partial updates has a client eventing model. When a partial update begins and ends, two JavaScript events are fired—beginRequest and endRequest:

function pageLoad() {
   progressManager = new Samples.Progress();
   reqManager = Sys.WebForms.PageRequestManager.getInstance();
   reqManager.add_beginRequest(onBeginRequest);
   reqManager.add_endRequest(onEndRequest);
}

The code associated with onBeginRequest executes just before the partial page update request is sent over. Here’s a sample handler:

function onBeginRequest(sender, args) {
    // Get the task ID
    taskID = progressManager.getTaskID();

    // Must pass the taskID via the hidden field.
    progressManager.modifyRequestForTaskId(args.get_request(),
         taskID, “HiddenField1”);

    // Too late for this to work
    progressManager.startMonitor(taskID, 2000,
         updateProgress, updateProgressCompleted);
}

When the beginRequest event is fired, it is too late to set the contents of the hidden field to the task ID. More precisely, when the event fires, the body of the request has been already prepared and won’t be further updated. Hence, setting the hidden field at this time is useless. To send the task ID value through the hidden field, you have to modify the body of the request.

function modifyRequestForTaskId(request, taskID, hiddenField) {
    var body = request.get_body();
    var token = “&” + hiddenField + “=”;
    body = body.replace(token, token + taskID);
    request.set_body(body);

    return request;
}

The request body is a plain string that contains a token of the form &HiddenField=. You replace the token with another one that contains the task ID: &HiddenField=xyz. Finally, in the endRequest event handler you call the stopMonitor method of the Samples.Progress class.

Summary

Monitoring ongoing tasks is relatively easy in Windows® applications, but that hasn’t been the case at all in Web applications. However, the AJAX programming model makes this task much easier to accomplish because ASP.NET AJAX Extensions provide ready-made tools for building the infrastructure you need using client and a server APIs.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Introducing Microsoft ASP.NET 2.0 AJAX Extensions (Microsoft Press, 2007). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos. For more tips on this topic, keep an eye on his freshly reactivated blog.


Service Station: WCF Bindings In Depth

Service Station
WCF Bindings In Depth
Aaron Skonnard

Code download available at: ServiceStation2007_07.exe (162KB)
Contents
What is a Binding?
Built-In Bindings
Configuring Bindings
Defining a Custom Binding
Making Custom Bindings Easy to Use
Sharing Binding Descriptions
Conclusion

The Windows® Communication Foundation programming model makes it easy for developers to configure services with a variety of wire formats and message protocols. The binding is the secret sauce behind this simplicity. This month I show you in detail how the built-in bindings work, how to configure them, and how to define custom bindings of your own.

What is a Binding?

Ultimately, Windows Communication Foundation is a framework for building services that process XML messages. Windows Communication Foundation allows you to transmit messages using different transport protocols (such as HTTP, TCP, and MSMQ) and using different XML representations (such as text, binary, or MTOM, which is commonly referred to as the message encoding in Windows Communication Foundation). In addition, you can enhance specific messaging interactions with a suite of SOAP protocols, such as the various WS-* specifications around security, reliable messaging, and transactions. All three of these communication concepts—the transport, the message encoding, and the suite of protocols—are central to what happens on the wire at run time.

In Windows Communication Foundation, all of these communication details are handled by the channel stack (see Figure 1). Just as its name implies, the channel stack is a stack of channel components that all messages pass through during runtime processing. The bottom-most component is the transport channel. This implements the given transport protocol and reads incoming messages off the wire. The transport channel uses a message encoder to read the incoming bytes into a logical Message object for further processing.

Figure 1 Bindings and Channel Stacks
Figure 1 Bindings and Channel Stacks(Click the image for a smaller view)
Figure 1 Bindings and Channel Stacks
Figure 1 Bindings and Channel Stacks (Click the image for a larger view)

After that, the message bubbles up through the rest of the channel stack, giving each protocol channel an opportunity to do its processing, until it eventually reaches the top and Windows Communication Foundation dispatches the final message to your service implementation. Messages undergo significant transformation along the way.

Although the Windows Communication Foundation channel architecture provides a consistent and flexible messaging foundation, it would be too tedious for most developers to work with it directly. When building a channel stack, you have to think carefully about the ordering of the components and whether or not they are compatible with one another. So instead, Windows Communication Foundation provides a simpler abstraction—the concept of endpoints—for indirectly configuring the underlying channel stack.

The idea is that you configure services with one or more endpoints in order to accommodate different communication scenarios. For each endpoint, you specify an address, a binding, and a contract. The address simply specifies the network address where you want to listen for messages while the contract specifies what the messages arriving at the specified address should contain. It’s the binding, however, that provides the recipe for building the channel stack needed to properly process the messages. When you load a service, Windows Communication Foundation follows the instructions outlined by the binding description to create each channel stack. The binding binds your service implementation to the wire through the channel stack in the middle (Figure 1 illustrates this).

This communication model allows you to think about the messaging features you need, in simple terms, while also allowing you to ignore the complexities of actually making it happen in the Windows Communication Foundation runtime environment.

For more background on message processing fundamentals, see my April 2007 column on WCF Messaging Fundamentals.

Built-In Bindings

Within the Windows Communication Foundation programming model, bindings are represented by the System.ServiceModel.Channels.Binding class. All binding classes must derive from this base class. Figure 2 summarizes the built-in binding classes that come with Windows Communication Foundation out of the box. Each class derives from Binding and defines a different channel stack configuration through its implementation. These built-in bindings address the most common messaging scenarios that you’ll run into today. Transaction flow is always disabled by default—the table shows the protocols that are used by default when you choose to enable transaction flow. Also note that MsmqIntegrationBinding doesn’t use a Windows Communication Foundation message encoding—instead it lets you choose a pre-Windows Communication Foundation serialization format.

Figure 2 Windows Communication Foundation Built-In Bindings (Close Figure 2)

Binding Class Name | Transport | Message Encoding | Message Version | Security Mode | Reliable Messaging | Transaction Flow (disabled by default)
BasicHttpBinding | HTTP | Text | SOAP 1.1 | None | Not Supported | Not Supported
WSHttpBinding | HTTP | Text | SOAP 1.2 WS-Addressing 1.0 | Message | Disabled | WS-AtomicTransactions
WSDualHttpBinding | HTTP | Text | SOAP 1.2 WS-Addressing 1.0 | Message | Enabled | WS-AtomicTransactions
WSFederationHttpBinding | HTTP | Text | SOAP 1.2 WS-Addressing 1.0 | Message | Disabled | WS-AtomicTransactions
NetTcpBinding | TCP | Binary | SOAP 1.2 | Transport | Disabled | OleTransactions
NetPeerTcpBinding | P2P | Binary | SOAP 1.2 | Transport | Not Supported | Not Supported
NetNamedPipesBinding | Named Pipes | Binary | SOAP 1.2 | Transport | Not Supported | OleTransactions
NetMsmqBinding | MSMQ | Binary | SOAP 1.2 | Message | Not Supported | Not Supported
MsmqIntegrationBinding | MSMQ | Not Supported (uses a pre-WCF serialization format) | Not Supported | Transport | Not Supported | Not Supported
CustomBinding | You Decide | You Decide | You Decide | You Decide | You Decide | You Decide

For example, the BasicHttpBinding was designed for scenarios where interoperability is of utmost importance. As a result, BasicHttpBinding uses HTTP for the transport and text for the message encoding. An important aspect of the message encoding is the expected message version, which happens to be SOAP 1.1 for BasicHttpBinding. As for additional protocols, BasicHttpBinding is capable of using transport or message security, but both are disabled by default. The other WS-* protocols are not supported with this binding. As a result, this binding produces a simple channel stack capable of interoperating with any other basic Web services implementation—and this is a great choice when your number-one priority is to make things work.

Then there’s WSHttpBinding. This was also designed for interoperability while incorporating the richer Web services protocols for security, reliable messaging, and transactions. As a result, WSHttpBinding also uses HTTP for the transport and text for the message encoding, but it uses SOAP 1.2 along with WS-Addressing 1.0 for the message version—they are needed to carry the additional Web services protocol headers. The binding enables message-based security (via WS-Security and friends) and is capable of supporting WS-ReliableMessaging and WS-AtomicTransactions if you choose to enable them. WSHttpBinding produces a more sophisticated channel stack and will most likely be constrained to enterprise scenarios where integration across frameworks and platforms is required.

And that brings me to NetTcpBinding. Unlike the two HTTP bindings, the various "Net" bindings were not designed for interoperability. In fact, each was designed for optimizing a different communication scenario when you can safely assume you have the Microsoft® .NET Framework 3.0 installed on both sides (this explains why the binding names are prefixed with "Net").

NetTcpBinding uses TCP for the transport, binary for the message encoding, and SOAP 1.2 for the message version. It enables transport security by default and can support transactions if enabled. As you can see, the configuration of this binding focuses on creating a channel stack that will perform better in Windows environments, giving you a great option for replacing your various COM+ and .NET remoting investments.

All you have to do is choose a particular binding based on your communication needs (see Figure 2) and Windows Communication Foundation takes care of producing the appropriate channel stack. Programmatically you choose a binding by providing an instance of the desired binding class in your call to ServiceHost.AddServiceHost:

using (ServiceHost host = new ServiceHost(typeof(ChatService)))
{
    host.AddServiceEndpoint(typeof(IChat),
        new BasicHttpBinding(), “http://localhost:8080/chat”);
    host.AddServiceEndpoint(typeof(IChat),
        new WSHttpBinding(), “http://localhost:8080/chat/secure”);
    host.AddServiceEndpoint(typeof(IChat),
        new NetTcpBinding(), “net.tcp://localhost:8081/chat”);

    host.Open();

    ... // remaining code ommitted for brevity
}

Or you can specify the binding name when defining endpoints in the host’s configuration file. When you specify an endpoint via configuration, the binding name must be written in camel case (for instance basicHttpBinding, wsHttpBinding, or netTcpBinding). Figure 3 illustrates how to configure the same three endpoints in the host’s configuration file.

Figure 3 Configuring Endpoints in Host’s Configuration File (Close Figure 3)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”ChatService”>
        <endpoint
           address=”http://localhost:8080/chat”
           binding=”basicHttpBinding”
           contract=”IChat”/>
        <endpoint
           address=”http://localhost:8080/chat/secure”
           binding=”wsHttpBinding”
           contract=”IChat”/>
        <endpoint
           address=”net.tcp://localhost:8081/chat”
           binding=”netTcpBinding”
           contract=”IChat”/>
      </service>
    </services>
  </system.serviceModel>
</configuration>

The remaining bindings in Figure 2 target other common communication scenarios. For example, when you need bidirectional communication over HTTP, you can use WSDualHttpBinding, or when you want to implement asynchronous durable messaging, you can use NetMsmqBinding. When you need peer-to-peer capabilities for rich client applications, you can turn to NetPeerTcpBinding.

You’ll be hard-pressed to come up with communication scenarios that aren’t already addressed by one of these built-in bindings. However, it’s very likely that you’ll need to configure one of the built-in bindings in order to accommodate a specific integration issue.

Configuring Bindings

You can configure any of the built-in bindings by taking advantage of the various constructors and properties found on the classes. Once you’ve instantiated a binding object, you can modify its public writable properties before you pass the object to ServiceHost.AddServiceEndpoint.

The following example illustrates how to configure a BasicHttpBinding object to use MTOM and transport security (HTTPS):

BasicHttpBinding basicHttpBinding = new BasicHttpBinding();
basicHttpBinding.MessageEncoding = WSMessageEncoding.Mtom;
basicHttpBinding.Security.Mode = BasicHttpSecurityMode.Transport;

host.AddServiceEndpoint(
    typeof(IChatService),
    basicHttpBinding,
    “http://localhost:8080/chat”);
...

You can accomplish the same thing entirely within configuration using the <bindings> configuration section. This mechanism allows you to define numerous named binding configurations for any of the built-in binding classes. Then you can apply a particular binding configuration to an endpoint through the <endpoint>’s bindingConfiguration attribute. Figure 4 illustrates how to create a binding configuration named basicConfig that’s also set up to use MTOM and transport security.

Figure 4 Creating a Binding Configuration (Close Figure 4)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”ChatService”>
        <endpoint address=”http://localhost:8080/chat”
          binding=”basicHttpBinding”
          bindingConfiguration=”basicConfig”
          contract=”ChatLibrary.IChat” />
        ...
      </service>
    </services>
    <bindings>
      <basicHttpBinding>
        <binding name=”basicConfig” messageEncoding=”Mtom”>
          <security mode=”Transport”/>
        </binding>
      </basicHttpBinding>
      ...
    </bindings>
  </system.serviceModel>
</configuration>

Notice how the <endpoint> definition refers to basicHttpBinding for the binding name and basicConfig for the binding configuration. You can define multiple binding configurations for each known binding class when multiple endpoints need to use different binding configurations. Take a look at Figure 5 for an example. In this configuration file, I’ve created two binding configurations for BasicHttpBinding—both enable transport security but one uses text while the other uses MTOM. The first endpoint is configured to use basicConfig1, the second basicConfig2. Both endpoints use the same binding class but two different configurations.

Figure 5 Binding Configuration Examples (Close Figure 5)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”ChatService”>
        <endpoint address=”text”
           binding=”basicHttpBinding”
           bindingConfiguration=”basicConfig1”
           contract=”IChat” />
        <endpoint address=”mtom”
           binding=”basicHttpBinding”
           bindingConfiguration=”basicConfig2”
           contract=”IChat” />
        <endpoint address=”secure”
           binding=”wsHttpBinding”
           bindingConfiguration=”wsConfig”
           contract=”IChat” />
        <endpoint address=””
           binding=”netTcpBinding”
           bindingConfiguration=”tcpConfig”
           contract=”IChat” />
      </service>
    </services>
    <bindings>
      <basicHttpBinding>
        <binding name=”basicConfig1” messageEncoding=”Text”>
          <security mode=”Transport”/>
        </binding>
        <binding name=”basicConfig2” messageEncoding=”Mtom”>
          <security mode=”Transport”/>
        </binding>
      </basicHttpBinding>
      <wsHttpBinding>
        <binding name=”wsConfig” transactionFlow=”true”>
          <security mode=”TransportWithMessageCredential”>
            <message clientCredentialType=”UserName”/>
          </security>
          <reliableSession enabled=”true” ordered=”true”/>
        </binding>
      </wsHttpBinding>
      <netTcpBinding>
        <binding name=”tcpConfig” transactionFlow=”true”
            transactionProtocol=”WSAtomicTransactionOctober2004”>
          <security mode=”None”/>
          <reliableSession enabled=”true” />
        </binding>
      </netTcpBinding>
    </bindings>
  </system.serviceModel>
</configuration>

Figure 5 also contains a configuration for WSHttpBinding, named wsConfig, which does several things. It enables transaction flow and ordered reliable messaging. It also changes the security mode to TransportWithMessageCredential and specifies a client credential type of UserName—this means the channel stack will be configured to use transport security (HTTPS) but incoming messages will be expected to contain WS-Security <UsernameToken> elements.

The final binding configuration shown in Figure 5 changes the defaults for NetTcpBinding. In this example, the binding configuration named tcpConfig enables transaction flow and changes the transaction protocol from OleTransactions (the default) to the October 2004 version of WS-AtomicTransaction. It also disables transport security, which is on by default, and enables WS-ReliableMessaging.

The Windows Communication Foundation binding configuration mechanism can accommodate most of the integration scenarios. However, if luck still finds you stuck dealing with an obscure scenario that you cannot resolve, you can always define a custom binding to meet your exact needs.

Defining a Custom Binding

You define a custom binding by deriving a class from System.ServiceModel.Channels.Binding. Your implementation has one primary responsibility: to produce an ordered collection of BindingElement objects (an object of type BindingElementCollection). The runtime asks the binding instance to do this by calling its CreateBindingElements method at run time. At this point, the binding instance should look at its current configuration, create the various binding element objects it needs, and order them properly in the collection before returning it.

The Windows Communication Foundation runtime walks through the binding element collection and uses it to create the underlying channel stack (this is essentially what I mean by the "recipe"). There’s a one-to-one mapping between the binding elements and the objects that end up in the channel stack at run time.

Each of the built-in binding classes derives from Binding and provides a unique implementation of CreateBindingElements. You should whip out .NET Reflector (available at aisto.com/roeder/dotnet) and view the implementation of each class. You’ll learn exactly what binding elements each one uses and the order in which they are placed in the collection. This is valuable because you’ll be doing the exact same thing when you create a custom binding.

Before you can create a custom binding, however, you must become familiar with the binding element classes that are available to you. A binding element is a class that derives from System.ServiceModel.Channels.BindingElement. Windows Communication Foundation ships with numerous BindingElement classes that are ready to use.

Figures 6 through 9 describe most of the built-in BindingElement classes that are currently found in Windows Communication Foundation. Figure 6 lists the classes that represent the different transport protocols supported by Windows Communication Foundation. Figure 7 describes the classes that represent different types of transport security. Figure 8 describes the classes that represent the different message encodings. And finally, Figure 9 describes the classes that represent the layered protocols for security, reliable messaging, and transactions.

Figure 9 Protocol Binding Elements (Close Figure 9)

Protocol | Class | Element
Transaction Flow | TransactionFlowBindingElement | <transactionFlow/>
Reliable Messaging | ReliableSessionBindingElement | <reliableSession/>
Security | SecurityBindingElement | <security/>
Figure 8 Message Encoding Binding Elements (Close Figure 8)

Message Encoding | Class | Element
Text | TextMessageEncodingBindingElement | <textMessageEncoding/>
MTOM | MtomMessageEncodingBindingElement | <mtomMessageEncoding/>
Binary | BinaryMessageEncodingBindingElement | <binaryMessageEncoding/>
Figure 7 Transport Security Binding Elements (Close Figure 7)

Transport Security | Class | Element
Windows | WindowsStreamSecurityBindingElement | <windowsStreamSecurity/>
SSL | SslStreamSecurityBindingElement | <sslStreamSecurity/>
Figure 6 Transport Binding Elements (Close Figure 6)

Transport | Class | Element
HTTP | HttpTransportBindingElement | <httpTransport/>
HTTPS | HttpsTransportBindingElement | <httpsTransport/>
TCP | TcpTransportBindingElement | <tcpTransport/>
Named pipes | NamedPipeTransportBindingElement | <namedPipeTransport/>
MSMQ | MsmqTransportBindingElement | <msmqTransport/>
MSMQ | MsmqIntegrationBindingElement | <msmqIntegration/>
P2P | PeerTransportBindingElement | <peerTransport/>

There is another BindingElement not listed in any of these figures. It’s called CompositeDuplexBindingElement (<compositeDuplex/>) and it’s something of a special case because it doesn’t fit into any of these groupings. You use it when you need to create a binding that supports duplex communications.

You create a custom binding by instantiating CustomBinding and adding the desired BindingElement objects to its Elements collection. However, the order in which you add them is very important. Here are the order details:

  • Transaction Flow (Not Required)
  • Reliable Messaging (Not Required)
  • Message Security (Not Required)
  • Composite Duplex (Not Required)
  • Message Encoding (Required)
  • Transport Security (Not Required)
  • Transport (Required)

You add to the collection from top to bottom. Only the transport binding element is officially required when defining a custom binding. Message encoding is required for each binding, but if you don’t specify one, Windows Communication Foundation will add a default encoding for you. The default encoding for HTTP(S) is text and for all other transports it is binary.

Here’s an example of creating and using one of the simplest custom bindings possible:

...
CustomBinding myHttpBinding = new CustomBinding();
myHttpBinding.Name = “myHttpBinding”;
myHttpBinding.Elements.Add(new HttpTransportBindingElement());

host.AddServiceEndpoint(typeof(IChat), myHttpBinding,
    “http://localhost:8080/chat/custom”);
...

The only BindingElement object I added was an instance of HttpTransportBindingElement. In this case, Windows Communication Foundation will choose to use the text message encoding when it builds the channel stack since it’s the default encoding for HTTP. This particular example produces a binding that ends up being identical to the built-in BasicHttpBinding.

However, you now have direct access to the underlying BindingElement objects, which gives you more flexibility and control over the resulting channel stack configuration. When you use one of the built-in binding classes, you’re limited to the facade it provides over the underlying BindingElementCollection.

Consider the example shown in Figure 10. It configures the individual BindingElement objects before adding them to the collection. The resulting binding uses a different message version than normal, SOAP 1.1 along with the August 2004 version of WS-Addressing, which may come in handy when you need to integrate with older Web services frameworks. It also uses a customized HTTP transport instance.

Figure 10 Configuring the Individual BindingElement Objects (Close Figure 10)
...
// instantiate message encoding element and configure
TextMessageEncodingBindingElement text =
    new TextMessageEncodingBindingElement();
text.MessageVersion = MessageVersion.Soap11WSAddressingAugust2004;

// instantiate transport element and configure
HttpTransportBindingElement http = new HttpTransportBindingElement();
http.TransferMode = TransferMode.Streamed;
http.UseDefaultWebProxy = true;

CustomBinding myHttpBinding = new CustomBinding();
myHttpBinding.Name = “myHttpBinding”;
myHttpBinding.Elements.Add(text);
myHttpBinding.Elements.Add(http);

host.AddServiceEndpoint(typeof(IChat), myHttpBinding,
    “http://localhost:8080/chat/custom”);
...

Of course, you can accomplish the same thing entirely within configuration by creating a configuration for <customBinding> as illustrated in Figure 11. This technique works like configuring any other built-in binding. You define a configuration for customBinding and then apply it to an endpoint. The difference is that in this case you’re defining the entire binding definition within the configuration. You don’t get anything for free when using CustomBinding.

Figure 11 Creating a Configuration for <customBinding> (Close Figure 11)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”ChatService”>
        <endpoint
          address=”custom”
          binding=”customBinding”
          bindingConfiguration=”myBasicHttpBindingConfiguration”
          contract=”IChat” />
      </service>
    </services>
    <bindings>
      <customBinding>
        <binding name=”myHttpBindingConfiguration”>
          <textMessageEncoding
            messageVersion=”Soap11WSAddressingAugust2004”/>
          <httpTransport
            useDefaultWebProxy=”true” transferMode=”Streamed”/>
        </binding>
      </customBinding>
    </bindings>
  </system.serviceModel>
</configuration>

Figure 12 shows a few more custom binding examples that use some of the other BindingElement classes shown in the figures. The myWSHttpBindingConfiguration configuration is similar to the built-in WSHttpBinding except it uses the binary message encoding and it enables transaction flow and ordered reliable messaging. The myNetTcpBindingConfiguration configuration is like NetTcpBinding except it uses the text message encoding and enables transaction flow.

Figure 12 Custom Binding Configuration Examples (Close Figure 12)
<configuration>
  <system.serviceModel>
    ...
    <bindings>
      <customBinding>
        <binding name=”myBasicHttpBindingConfiguration”>
          <textMessageEncoding
            messageVersion=”Soap11WSAddressingAugust2004”/>
          <httpTransport allowCookies=”true”
            useDefaultWebProxy=”true” transferMode=”Streamed”/>
        </binding>
        <binding name=”myWSHttpBindingConfiguration”>
          <transactionFlow/>
          <reliableSession ordered=”true”/>
          <security authenticationMode=”SspiNegotiated”/>
          <binaryMessageEncoding/>
          <httpTransport/>
        </binding>
        <binding name=”myNetTcpBindingConfiguration”>
          <transactionFlow/>
          <textMessageEncoding/>
          <windowsStreamSecurity/>
          <tcpTransport/>
        </binding>
      </customBinding>
    </bindings>
</configuration>

Making Custom Bindings Easy to Use

Although CustomBinding is convenient for one-off customizations, you should define a custom Binding class when you need to make it easier to reuse. When you derive from Binding, you need to override its two abstract members: CreateBindingElements and Scheme. Your implementation of CreateBindingElements is where you create and return the collection of BindingElement objects.

Suppose you want to create a custom HTTP binding that always uses the binary message encoding—call it NetHttpBinding—and assume that you want to let the users of this binding configure two properties on the underlying HttpTransportBindingElement instance—TransferMode and UseDefaultWebProxy. The code in Figure 13 demonstrates how you can accomplish this.

Figure 13 Custom HTTP Binding (Close Figure 13)
public class NetHttpBinding : Binding
{
    private BinaryMessageEncodingBindingElement binary =
        new BinaryMessageEncodingBindingElement();
    private HttpTransportBindingElement http =
        new HttpTransportBindingElement();

    public override BindingElementCollection CreateBindingElements()
    {
        return new BindingElementCollection(
            new BindingElement[] { binary, http });
    }

    public TransferMode TransferMode
    {
        get { return http.TransferMode; }
        set { http.TransferMode = value; }
    }

    public bool UseDefaultWebProxy
    {
        get { return http.UseDefaultWebProxy; }
        set { http.UseDefaultWebProxy = value; }
    }

    public override string Scheme { get { return “http”; } }
}

With the code in place, developers can simply instantiate your NetHttpBinding class and apply the instance to an endpoint as illustrated here:

...
NetHttpBinding netHttp = new NetHttpBinding();
netHttp.TransferMode = TransferMode.Streamed;
netHttp.UseDefaultWebProxy = true;

host.AddServiceEndpoint(typeof(IChat), netHttp,
    “http://localhost:8080/chat/nethttp”);
...

This is much easier than dealing with CustomBinding. But if you want to make it even easier, you may want to make it possible to configure the NetHttpBinding via the Windows Communication Foundation configuration section. In order to do this, you need to implement a few more configuration-related classes. Figure 14 provides a complete example of the classes you’ll need.

Figure 14 Custom Configuration Classes for NetHttpBinding (Close Figure 14)
public class NetHttpBindingConfigurationElement : StandardBindingElement
{
    public NetHttpBindingConfigurationElement(string configurationName)
        : base(configurationName) { }

    public NetHttpBindingConfigurationElement()
        : this(null) { }

    protected override Type BindingElementType
    {
        get { return typeof(NetHttpBinding); }
    }

    [ConfigurationProperty(“transferMode”,
         DefaultValue = TransferMode.Buffered)]
    public TransferMode TransferMode
    {
        get { return ((TransferMode)(base[“transferMode”])); }
        set { base[“transferMode”] = value; }
    }

    [ConfigurationProperty(“useDefaultWebProxy”, DefaultValue = false)]
    public bool UseDefaultWebProxy
    {
        get { return ((bool)(base[“useDefaultWebProxy”])); }
        set { base[“useDefaultWebProxy”] = value; }
    }

    protected override ConfigurationPropertyCollection Properties
    {
        get
        {
            ConfigurationPropertyCollection properties = base.Properties;
            properties.Add(new ConfigurationProperty(“transferMode”,
                typeof(TransferMode), TransferMode.Buffered));
            properties.Add(new ConfigurationProperty(“useDefaultWebProxy”,
                typeof(bool), true));
            return properties;
        }
    }

    protected override void InitializeFrom(Binding binding)
    {
        base.InitializeFrom(binding);
        NetHttpBinding netHttpBinding = ((NetHttpBinding)(binding));
        this.TransferMode = netHttpBinding.TransferMode;
        this.UseDefaultWebProxy = netHttpBinding.UseDefaultWebProxy;
    }

    protected override void OnApplyConfiguration(Binding binding)
    {
        if (binding == null)
        {
            throw new System.ArgumentNullException(“binding”);
        }
        if (binding.GetType() != typeof(NetHttpBinding))
        {
            throw new System.ArgumentException(
                “Invalid binding type – expected NetHttpBinding”);
        }
        NetHttpBinding netHttpBinding = ((NetHttpBinding)(binding));
        netHttpBinding.TransferMode = this.TransferMode;
        netHttpBinding.UseDefaultWebProxy = this.UseDefaultWebProxy;
    }
}

public class NetHttpBindingSection :
    StandardBindingCollectionElement<NetHttpBinding,
        NetHttpBindingConfigurationElement>
{
}

With these classes in place, you can add a binding extension mapping to the configuration file so Windows Communication Foundation knows that netHttpBinding goes with the configuration classes shown in Figure 14. Now you can simply specify netHttpBinding for the binding name, and configure it within the <bindings> section like any other binding, as shown in Figure 15. (The MSDN® documentation available at msdn2.microsoft.com/aa967561.aspx provides more complete custom binding examples.)

Figure 15 Configuring netHttpBinding (Close Figure 15)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”ChatService”>
        <endpoint
          address=”nethttp”
          binding=”netHttpBinding”
          bindingConfiguration=”myNetHttpBindingConfiguration”
          contract=”ChatLibrary.IChat” />
      </service>
    </services>
    <bindings>
      <netHttpBinding>
        <binding
          name=”myNetHttpBindingConfiguration”
          transferMode=”Streamed”
          useDefaultWebProxy=”true”/>
      </netHttpBinding>
    </bindings>
    <extensions>
      <bindingExtensions>
        <add name=”netHttpBinding”
          type=”NetHttpBindingSection, NetHttpBindingLibrary” />
      </bindingExtensions>
    </extensions>
  </system.serviceModel>
</configuration>

I’ve covered all the major options and components related to implementing a custom binding. The only thing I haven’t yet discussed is how to write a custom BindingElement class, which may be necessary if you can’t find one that fits your exact needs. For example, say you need to implement a custom transport. In such a case, you’ll need to implement the transport channel (which is non-trivial) and a corresponding BindingElement class in order to make it easy to use within a binding. Implementing custom channel components and binding elements are advanced extensibility topics that I’ll save for a future column.

Sharing Binding Descriptions

Bindings are all about what happens on the wire. When you choose a particular binding, configure one in a special way, or define a custom binding, it’s vital for consumers to be able to discover the exact binding configuration so they can properly integrate with the endpoint. Services can share their binding configurations with consumers by exposing metadata in the form of Web Services Description Language (WSDL) and WS-Policy.

You can enable metadata on a Windows Communication Foundation service through the <serviceMetadata> behavior. This allows developers to browse to the service description using a Web browser. You can also make the metadata available programmatically by enabling MEX (IMetadataExchange) on a separate endpoint.

Once you have metadata enabled, Windows Communication Foundation automatically produces the appropriate WSDL definition for the service upon request. The binding configuration for each endpoint is translated into WS-Policy statements and embedded within the WSDL definition to specify the exact configuration. If you browse to a Windows Communication Foundation-generated WSDL definition, you’ll find these policy statements toward the top of the file.

As long as the client is capable of doing everything found in the WS-Policy statement, it should be able to produce the necessary client-side configuration. When using Windows Communication Foundation on the client side, svcutil.exe inspects the policy statements found in the WSDL definition and translates them into a corresponding binding configuration section in the client configuration file (see Figure 16). The net result is that the client ends up with an equivalent channel stack for dispatching outgoing messages.

Figure 16 Sharing Binding Congfigurations via Metadata
Figure 16 Sharing Binding Congfigurations via Metadata(Click the image for a smaller view)
Figure 16 Sharing Binding Congfigurations via Metadata
Figure 16 Sharing Binding Congfigurations via Metadata (Click the image for a larger view)

The following code illustrates how a Windows Communication Foundation client can automatically access all of the endpoints exposed by a service via MEX at run time:

...
ServiceEndpointCollection endpoints =
    MetadataResolver.Resolve(typeof(IChat),
    new EndpointAddress(“http://localhost:8080/chat/mex”));
foreach (ServiceEndpoint se in endpoints)
{
    ChannelFactory<IChat> cf = new ChannelFactory<IChat>(
        se.Binding, se.Address);
    IChat client = cf.CreateChannel();
    client.SendMessage(msg);
}
...

With this code available on the client, you can add new endpoints to the service, each with a unique binding configuration. Each time you run the client it will invoke all of the endpoints using the correct configuration. This shows that the client is able to automatically discover each endpoint’s binding configuration.

One thing that can potentially break interoperability, even when using Windows Communication Foundation on both sides, is when the service employs a custom BindingElement that the client doesn’t have access to (such as a custom transport). Obviously, using any type of custom channel component requires agreement between the client and service, and both sides will require an appropriate implementation.

This brings me to my final word of caution. It’s easy to configure yourself out of interoperability by getting too fancy with a binding configuration for no good reason. Make sure you know what you’re enabling and why. If you don’t have a good reason for something, stay with the defaults.

That said, when you do run into situations where interoperability fails when using the defaults, your first goal should be to figure out what’s getting in the way and then you can configure the binding (or create a custom binding) that works around the problem at hand.

Conclusion

The Windows Communication Foundation binding architecture provides an elegant model for configuring what happens on the wire. You simply specify a binding and Windows Communication Foundation uses that as the recipe for building the channel stack at run time.

Windows Communication Foundation ships with numerous built-in bindings that come preconfigured to address today’s most common communication needs. However, when the built-in bindings don’t fit the bill, you can configure them or create custom bindings of your own, making it possible to accommodate any tough integration scenario you encounter.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a cofounder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight’s Applied Web Services 2.0, Applied BizTalk Server 2006, and Applied Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. You can reach him at pluralsight.com/aaron.


Bugslayer: Wait Chain Traversal

Bugslayer
Wait Chain Traversal
John Robbins

Code download available at: Bugslayer2007_07.exe (175KB)
Contents
Basics of Wait Chain Traversal
WAITCHAIN_NODE_INFO Structure Pain
Unsafe
LockWatcher
Mixed Debugging on x64
Wrap Up

Having concentrated on writing development tools for years, I always look at new operating system releases with an eye toward seeing if there’s anything new I can use to solve difficult development problems. For example, Windows® XP introduced vectored exception handling, which allows you to intercept the exception handling chain for logging and diagnostic purposes. In the November 2006 issue of MSDN® Magazine, I discussed my attempt at using that functionality to write an exception logger for Microsoft® .NET-based applications.

With Windows Vista™, Microsoft didn’t disappoint. In fact, Windows Vista has a very interesting new API called Wait Chain Traversal (WCT), which allows you to determine when and why a process is deadlocked. That sounded extremely exciting to me, so of course I had to jump in with both feet. The good news is that WCT will report exactly what synchronization object you are deadlocking on. The bad news is that it only reports a limited set of synchronization primitives. Even with that limitation, it’s still a very useful API and something you’ll want to have in your debugging toolkit.

In this column, I want to discuss the WCT API, its usage, and its limitations. As part of this column, I give you a tool that pinpoints all the deadlocks supported by WCT. Because I insisted on writing the tool in .NET, I also get to show you a descent into the depths of interop despair and how I was able to get the WCT API working from the .NET Framework.

Basics of Wait Chain Traversal

Figure 1 lists the synchronization primitives that the WCT API supports for deadlock reporting. While there are a few other synchronization primitives you might deadlock on, such as an event, the list covers a majority of the problem areas for most applications. If you’re focused on .NET, you might be wondering if a sync block is supported. If you haven’t heard of sync blocks before, that’s because you don’t use them directly; sync blocks are the internal construct used to implement the lock keyword along with the Monitor class. As the common language runtime (CLR) provides the implementation for sync blocks, they are not known to the native WCT API.

Figure 1 WCT-Supported Synchronization Primitives (Close Figure 1)

Synchronization Primitive | Description
ALPC | The internal Windows mechanism used for Remote Procedure Calls
COM | COM calls across threads and processes
Critical Sections | The lightweight, per process, native synchronization type
Mutexes | The handle-based synchronization type generally used to coordinate multiple process synchronization
SendMessage | Sending a message to another thread or window blocks until that message is processed
Process and Thread Handles | Waiting on processes and threads

The WCT API works its magic by looking at whether a thread is waiting in a blocking call. If the thread is on a blocking call, it looks up the object the thread is blocking on and determines, if appropriate, the name of the object and which thread or process owns that object. For example, if thread A is waiting on process B to complete execution and process B calls SendMessage to process A, you have a deadlock that WCT can detect and report.

The only problem I see with the WCT API is that it does not report deadlocks when using WaitForMultipleObjects, which reduces its usefulness in many applications. For example, .NET synchronization uses WaitForMultipleObjects for all synchronization coordination. While WCT won’t report that the thread is deadlocked, it will report that the thread is blocked. Fortunately, the WCT API can report the waiting times so you can manually decide if a particular thread is deadlocked through inspection and comparison using a tool like LockWatcher (part of this column’s code download).

WCT is a typical Win32® API, meaning you have to grab an opaque handle, pass that handle to the method that records the data, and call a form of CloseHandle when done. The Platform SDK documentation shows all that’s required to initialize and use the WCT API. For initialization, there’s one slight bump in that you’ll always want to call the RegisterWaitChainCOMCallback function to register the CoGetCallState and CoGetActivationState COM functions so WCT can report COM ownership information. As these functions are not documented, you’ll need to get them through GetProcAddress from ole32.dll. Personally, I think the WCT API should automatically handle this small chore when you initialize it to look for COM deadlocks.

To initialize WCT, call the OpenThreadWaitChainSession function exported from advapi32.dll:

HWCT WINAPI OpenThreadWaitChainSession(
    DWORD Flags, PWAITCHAINCALLBACK callback
);

Once initialized, you can call GetThreadWaitChain whenever you want to determine the blocked threads in an application:

BOOL WINAPI GetThreadWaitChain(
  HWCT WctHandle, DWORD_PTR Context,
  DWORD Flags, DWORD ThreadId,
  LPDWORD NodeCount, PWAITCHAIN_NODE_INFO NodeInfoArray,
  LPBOOL IsCycle
);

While you can initialize the WCT API to do all operations asynchronously, nearly everyone who uses it will use the synchronous version. In addition to the thread ID, the Flag parameter to GetThreadWaitChain sets the type of data you’re interested in receiving. The Flags parameter defaults to only looking at threads inside the specified process. If you want to get information from other processes, you’ll need to specify the three flags WCT_OUT_OF_PROC_COM_FLAG, WCT_OUT_OF_PROC_CS_FLAG, and WCT_OUT_OF_PROC_FLAG. You can do so easily with the summary flag that ORs these three together, WCTP_GETINFO_ALL_FLAGS.

The key information returned by GetThreadWaitChain is an array of WAITCHAIN_NODE_INFO structures that describes the state of the thread and what it’s currently blocked on. Additionally, GetThreadWaitChain will also return whether the thread is deadlocked based on the analysis engine inside the WCT code. On the whole, the WCT API is relatively simple, and while it won’t tell you all your deadlocks, it’s accurate for what it does report.

When a thread is blocked, the WCT API reports in the array of WAITCHAIN_NODE_INFO structures the thread and the cause of the deadlock. For example, if you have a deadlock where thread A owns a critical section X and wants to acquire critical section Y, and thread B owns critical section Y and wants to acquire X, the deadlock is reported as follows in the array:

Index 1: Thread A’s ID
Index 2: Waiting on a critical section
Index 3: Thread B’s ID
Index 4: Waiting on a critical section
Index 5: Thread A’s ID

Note that the critical section address is not reported. However, for other objects, such as mutexes, if it has a name, the WCT API will report that name.

To get the most out of the WCT API, you’ll always want to name your handles so you can see exactly what’s causing your deadlock. Of course, if you’re naming handles, those handles are now visible outside the process, which can lead to serious synchronization problems. For example, say you give a mutex a name such as MyHappyMutex. You have two processes running, A and B, and both are waiting on the same mutex name. If process A releases the mutex, both process A and process B could grab it. Thus, you could have a serious debugging challenge. If you are going to name your handles and you want to ensure they are unique to your process, you’ll want to append on the name the process ID or some other string guaranteed to be unique to the process.

While naming your handles does help with your debugging, some people might be nervous about security attacks because, once a handle is global, another process can open the handle. For certain scenarios, this can be a valid concern; if you are worried at the prospect, at least consider naming your handles in debug builds.

WAITCHAIN_NODE_INFO Structure Pain

While the WCT API is designed for native code, my plan was to write a tool that uses it from .NET code as I had some ideas for additional analysis. Additionally, .NET applications are easier to write since you don’t have to worry as much about memory so I thought I could implement it faster. Little did I know I was about to run into some real head-scratching problems with interop.

Figure 2 shows the native WAITCHAIN_NODE_INFO structure. If you look at it closely, you’ll see that it has two overlapping unions in it. For most structures P/Invoke is no trouble, but whenever there’s a C++ union floating around you know you’re in for some serious pain. Nearly all structures and APIs from Windows are defined at the pinvoke.net Web site, but a sinking feeling told me that I was on my own with this nasty native structure.

Figure 2 Native WAITCHAIN_NODE_INFO Structure (Close Figure 2)
typedef struct _WAITCHAIN_NODE_INFO
{
    WCT_OBJECT_TYPE ObjectType;
    WCT_OBJECT_STATUS ObjectStatus;

    union {
        struct {
            WCHAR ObjectName[WCT_OBJNAME_LENGTH];
            LARGE_INTEGER Timeout;    // Not implemented in v1
            BOOL Alertable;           // Not implemented in v1
        } LockObject;

        struct {
            DWORD ProcessId;
            DWORD ThreadId;
            DWORD WaitTime;
            DWORD ContextSwitches;
        } ThreadObject;
    };

} WAITCHAIN_NODE_INFO, *PWAITCHAIN_NODE_INFO;

My first hope was that I could possibly get by with defining two separate structures on the .NET side. On other projects, I’ve been able to declare separate DllImportAttribute P/Invoke methods with the different .NET side declarations to ease the work on defining the structure. With the WCT code, I’d be assuming possible internal implementation details, but the .NET structure definition would be trivial. Running the native Wait Chain Traversal sample (msdn2.microsoft.com/ms681418.aspx) on a deadlocking sample quickly showed me that there was enough difference in the returned deadlock information that this first attempt at a solution wasn’t going to be enough.

That meant I needed to turn to the fun of LayoutKind.Explicit to define the individual offsets inside the structure. A quick native sample that returned the size of the WAITCHAIN_NODE_INFO structure showed it is 280 bytes on both 32-bit and 64-bit Windows, so that eliminated the necessity of defining separate structures and programmatically determining which one to use based on the operating system.

My secret trick to easily finding structure offsets is to use WinDBG from the Debugging Tools for Windows package, which you can download from microsoft.com/whdc/devtools/debugging. Since WinDBG is a native-only debugger, you’ll need to debug a native C++ application that’s using the structure in question. To display a type you use the DT command, which conveniently shows all field offsets by default. In the case of the WAITCHAIN_NODE_INFO, I’ll want to pass the –r 1 and –v options to the DT command. The former tells WinDBG to recursively expand the structure one level and the latter dumps all the information about the structure. Figure 3 shows the result of the command on Windows Vista x64.

Figure 3 Result of WinDBG DT Command (Close Figure 3)
0:000> dt -v -r 1 NodeInfoArray[0]
Local var [AddrFlags 90  AddrOff 0000000000000050  Reg/Val rsp (7)] @ 0x12d970
[0] [16] struct _WAITCHAIN_NODE_INFO, 4 elements, 0x118 bytes
   +0x000 ObjectType       : Enum _WCT_OBJECT_TYPE,  11 total enums
8 ( WctThreadType )
   +0x004 ObjectStatus     : Enum _WCT_OBJECT_STATUS,  11 total enums
3 ( WctStatusBlocked )
   +0x008 LockObject       : struct _WAITCHAIN_NODE_INFO::<unnamed-tag>::<unnamed-tag>,
                             3 elements, 0x110 bytes
      +0x000 ObjectName       : [128]  “۰”
      +0x100 Timeout          : union _LARGE_INTEGER, 4 elements, 0x8 bytes
 0x0
         +0x000 LowPart          : 0
         +0x004 HighPart         : 0
         +0x000 u                : struct <unnamed-tag>, 2 elements, 0x8 bytes
         +0x000 QuadPart         : 0
      +0x108 Alertable        : 0
   +0x008 ThreadObject     : struct _WAITCHAIN_NODE_INFO::<unnamed-tag>::<unnamed-tag>,
                             4 elements, 0x10 bytes
      +0x000 ProcessId        : 0x6f0
      +0x004 ThreadId         : 0x6f4
      +0x008 WaitTime         : 0x3005ce
      +0x00c ContextSwitches  : 0x2fb

The first interesting piece of information is 0x118 in the third line of the output in Figure 3, which is the size of the structure. As you would expect, the first two enumerations are at offsets 0x0 and 0x4. The two unions, LockObject and ThreadObject, start at offset 0x8. Taking the LockObject union, the offset for the Unicode character array is 0x8 and runs for 0x100 bytes. That means the Timeout low part starts at 0x108 and the Timeout high part starts at 0x110. Finally, the Alertable field is at 0x114. Once you see the pattern, the offsets for the ThreadObject union are trivial.

After coding my .NET version of the WAITCHAIN_NODE_INFO structure using the FieldOffset attributes, I ran into my first major bump. The LockObject union has a 128 Unicode character array, ObjectName, as the first item. No matter how I tried to define that field, I was getting a TypeLoadException every time my code accessed my defined structure. For type-safety reasons as well as issues related to garbage collection, fields that point to reference types can’t overlap other fields, and as arrays and strings are both reference types, they’re unusable in this situation.

Unsafe

I was about to give up when I was reminded of the unsafe keyword and a new use of the fixed keyword in C# that was introduced with the .NET Framework 2.0. I could define the structure as unsafe and use a fixed char array definition for the ObjectName field; this allows for an unsafe inline array within a structure. It looks like an array in the structure, but the field actually ends up being accessed through a pointer; I could then use the String constructor that takes a character pointer and access the actual value.

After defining my structure and adding the /unsafe compiler switch, I was very happy to see that my test application reported a critical section deadlock exactly as the native test sample. Expanding my test application to have one thread acquire a critical section and call SendMessage to window on another thread waiting on the same critical section worked as well. Adding a deadlock test to block on the Explorer process also showed up correctly.

Thinking everything was going well, I added a test that deadlocked on a named mutex. The Platform SDK WCT sample showed the name of the mutex as:

\Sessions\1\BaseNamedObjects\Deadlock Mutex A

The \Session\1 indicates my interactive login session, and BaseNamedObjects is the kernel object table. In the test application, I’d named the mutex Deadlock Mutex A. However, my P/Invoke masterpiece of a structure said the name was only "\Sessi".

Thinking that I didn’t have the structure size right, I did some checks with Marshal.SizeOf, and the .NET side was reporting that it thought the structure was the proper 280 bytes. It also dawned on me that if I had the structure size wrong, the previous tests on critical sections would not have worked at all because the native GetThreadWaitChain function would have stepped all over the .NET memory for the array. Additionally, when I ran through the array for those previous deadlocks, I would have not seen the correct data in the array in my C# code.

At this point, I was a little confused, as the critical section deadlock test said my structure was the right size, but a named mutex was saying that the structure might be too small. My first thought was that I needed to see exactly what the native GetThreadWaitChain function was writing, so I ran my .NET application under WinDBG on my named mutex deadlock. Setting a breakpoint on ADVAPI32!GetThreadWaitChain, finding the array parameter, and stepping out of the function showed that the native side was copying in the complete name of the mutex into the array. However, that data was certainly not getting back into my C# code!

Unfortunately, I needed to see what .NET was doing with the array in native code. Of course, I’m running the x64 version of Windows Vista, so that meant that mixed debugging—where you can single step both managed and native sides of your application at the same time—is not supported. In the simple case where you have only one binary, set the platform to x86, which forces the application to be 32-bit—then you can do mixed debugging on 64-bit Windows Vista. However, for real-world applications that have many assemblies, it’s far faster to do your mixed debugging on a true 32-bit computer and not change your build. As I was writing this column, I received several questions about mixed debugging on x64 systems and can report it’s somewhat possible. At the end of the column, I’ll discuss some of the tricks I’ve been using to see both the native and managed sides when running full-bore x64 .NET Framework code.

After starting to spelunk through far too much assembly language, I had a brainstorm. Because I was only seeing "\Sessi" in the fixed array, I was wondering if the .NET interop code was only marshaling data when other fields existed at the same location. It almost seemed like the thread union portion, which is 12 bytes long, determined the length of the name union portion. To test my theory, I tossed in an extra Int32 field for the thread union and said that it began at 0x18 in the structure layout. You can imagine my shock when I ran the application and got back a string of "\Session"!

As it turns out, there is a bug with how the marshaler interacts with the code generated by the C# compiler for the fixed keyword when used to create a char array (this bug should be fixed in the Visual Studio "Orcas" release). To work around the issue, I needed to modify the struct so that it is blittable, meaning that the types that make up the struct must all have the same native representation as they do managed. The list of blittable types includes Byte, SByte, Int16, UInt16, Int32, UInt32, Int64, UInt64, IntPtr, UIntPtr, Single, and Double. You’ll notice, however, that Char is conspicuously missing from the list. In fact, Char is not blittable, because it can have multiple representations in native code (as either one or two bytes). To work around my problem, I changed the field type of the fixed array to be ushort rather than char; then in my code that turns this array into a String, I first cast from the resulting ushort* into the char* that the String constructor expects. Figure 4 shows the final structure that works correctly.

Figure 4 Final WAITCHAIN_NODE_INFO Definition (Close Figure 4)
[StructLayout ( LayoutKind.Explicit, Size=280 )]
internal unsafe struct WAITCHAIN_NODE_INFO
{
    [FieldOffset ( 0x0 )]
    public WCT_OBJECT_TYPE ObjectType;
    [FieldOffset ( 0x4 )]
    public WCT_OBJECT_STATUS ObjectStatus;

    // The name union.
    [FieldOffset ( 0x8 )]
    private fixed ushort RealObjectName [ WCT_OBJNAME_LENGTH ];
    [FieldOffset ( 0x108 )]
    public Int32 TimeOutLowPart;
    [FieldOffset ( 0x10C )]
    public Int32 TimeOutHiPart;
    [FieldOffset ( 0x110 )]
    public Int32 Alertable;

    // The thread union.
    [FieldOffset ( 0x8 )]
    public Int32 ProcessId;
    [FieldOffset ( 0xC )]
    public Int32 ThreadId;
    [FieldOffset ( 0x10 )]
    public Int32 WaitTime;
    [FieldOffset ( 0x14 )]
    public Int32 ContextSwitches;

    // Does the work to get the ObjectName field.
    public String ObjectName ( )
    {
        fixed ( WAITCHAIN_NODE_INFO* p = &this )
        {
            return (p->RealObjectName [ 0 ] != ‘\0’) ?
                new String ( (char*)p->RealObjectName ) :
                String.Empty;
        }
    }
}

LockWatcher

Once I got the WAITCHAIN_NODE_INFO structure figured out, I was able to finish the LockWatcher application. Now you can use it to find deadlocks in your applications. To look for deadlocks in a single process or a group of specific processes, just pass the process ID values on the command line. If you don’t specify any process IDs, LockWatcher will report on all processes on the computer. If you’re looking at all processes, you’ll want to start LockWatcher from a command window that has elevated rights so it has full permission to inspect data in other elevated processes.

By default, LockWatcher only shows the threads that are blocked and any blocked items reported through the WCT API. To see the wait time and context switch information, pass the –a command line option. One nice feature of LockWatcher is the –t <seconds> option, where LockWatcher will continually inspect the process for deadlocks on the interval you specify.

When LockWatcher reports a deadlock, you’ll see output such as the following, which is an example of a critical section deadlock:

Process : DEAD.EXE, PID : 5828
TID: 5392
**Following thread is DEADLOCKED!
TID:  424
   CriticalSection Status: Owned
      TID: 5492
         CriticalSection Status: Owned
            TID:  424
**Following thread is DEADLOCKED!
TID: 5492
   CriticalSection Status: Owned
      TID:  424
         CriticalSection Status: Owned
            TID: 5492

For the second thread, you can read the output as thread 5492 is waiting on a critical section owned by thread 424. Thread 424 is waiting on a critical section.

As I discussed earlier, the WCT API does not report deadlocks when using WaitForMultipleObjects. The following is the output of a deadlock where thread 5652 owns the Deadlock Mutex A and has called WaitForMultipleObjects to wait on the two other thread handles.

Process : DEAD.EXE, PID : 5796
TID: 5652
TID: 4776
   Mutex Status: Owned Name: \Sessions\1\BaseNamedObjects\Deadlock Mutex A
      TID: 5652
TID: 4920
   Mutex Status: Owned Name: \Sessions\1\BaseNamedObjects\Deadlock Mutex A
      TID: 5652

By looking at the output, you can piece together part of the deadlock, but not the whole story. My goal for LockWatcher was to build an algorithm on top of the returned data from the WCT API to identify deadlocks if WaitForMultipleObjects is used. Working from the previous example, it’s quite easy to see the work I had to do. My next step was to whip up a .NET sample that deadlocked on two Mutex class instances and had the following output:

Process : DEADDOTNET.EXE, PID : 3644
TID: 4976
TID: 6120
TID: 2152
TID: 5852

As you can see, the WCT API is not reporting anything useful so unless I was going to start doing code injection, stack walking, and parameter deciphering, there’s not much I can do. Even though the WCT API is limited, it will still help find far more deadlocks than if you did not have it. To see how LockWatcher reports different deadlocks, I included the appropriately named DEAD program, which I wrote in native C++, as a test program.

Mixed Debugging on x64

Earlier, I discussed how to recompile your application for an x86 platform to allow mixed managed and native debugging. However, as part of my exploration of the P/Invoke challenges I faced with the WAITCHAIN_NODE_INFO structure, I did a bit of mixed debugging on a .NET application running as an x64 binary. While not a supported scenario from Microsoft, I thought I’d tell you about some of the tricks I used to allow me to see both sides without going insane.

Managed debugging does not go through the standard Windows Debugging API (msdn2.microsoft.com/ms679276.aspx), but uses a scheme reliant on vectored exception handling. If you poke at what’s going on in the CLR Debugging API (msdn2.microsoft.com/ms404520.aspx), you can see they are using the usual single step and breakpoint exceptions as a normal debugger. Because managed debugging does not go through the Windows Debugging API, you won’t get the error you get when you attach a native debugger to an application that’s already running under a debugger. Thus, once you start debugging your managed application in Visual Studio®, you can attach a native debugger to the managed application.

While you can attempt to use a second instance of Visual Studio as the native debugger, it’s nearly impossible to do anything because you’ll be breaking on every single step the managed debugging API triggers. Because there’s no way in Visual Studio to automatically continue on exceptions, the only hope you’ll have is to enjoy the fun of WinDBG.

While WinDBG is harder to use, the lovely Event Filtering will let you ignore the single step exceptions that are triggered constantly by the CLR Debugging API. After you’ve attached WinDBG to the managed application, you’ll immediately stop at the initial, or loader, breakpoint.

The first command you’ll want to use is the following:

sxd –c “gn” sse

The SXD (Set Exception Disabled) command sets the Single Step Exception (SSE) to disabled, which means you’ll not stop when they occur. Since the CLR Debugging API controls everything with the single step, I’ve also set the first chance command to have WinDBG automatically continue and pass them on to the debuggee as not handled. Remember that you’ve set this command when you do stop inside the 64-bit native portion because you won’t be able to single step anything with it on. To turn single stepping back on when you need it, execute the following command:

sxe -c “” sse

Once you’ve got the single-stepping under control, you need to plan for how you’re going to deal with the CLR Debugging API’s breakpoints as well as the ones you want to set in native code. When I was trying to figure out what was going on in my P/Invoke problems, I needed to see the parameters to GetThreadWaitChain, which is in advapi32.dll. Consequently, I needed a breakpoint on that address via the following command:

bu ADVAPI32!GetThreadWaitChain

The big trick I needed to accomplish was ignoring all the breakpoints coming from the CLR Debugging API. Fortunately, WinDBG treats breakpoint exceptions like any other exceptions so I can continue using the SX (Set Exception) family of commands to have WinDBG perform a command whenever a breakpoint exception is hit. Even more fortunate, WinDBG supports rudimentary control flow commands known as debugger command programs. With the .if and .else commands, you can execute conditional logic on your debugger commands.

To ignore all breakpoint exceptions except the location where I set my breakpoint, the following command takes care of everything on an x64 machine:

sxe -c “.if (rip==ADVAPI32!GetThreadWaitChain){.echo STOPPED AT YOUR BU!}.else{gn}” bpe

The SXE (Set Exception Enabled) says for every BPE (Break Point Exception), execute the command on the first chance exception. The command checks to see whether the instruction register, RIP, is executing the first instruction in GetThreadWaitChain. (For a 32-bit machine, substitute EIP as the instruction register.) If it is, the debugger will report the stop by echoing the text to the screen. If it is not, the debugger will execute the GN (Go Not Handled) command to let the debuggee have the exception, thus not confusing the CLR Debugging API.

While a little bit convoluted, these steps make it possible to do a modicum of mixed debugging on an x64 system. Keep in mind that it is very easy to mess up the managed application if you are not super careful as to what you are doing. It’s highly unlikely that the techniques I presented here are supported—or even condoned—by Microsoft. However, when you need to see both the native and managed side of your application on an x64 system, at least you have a hint to get you going.

Wrap Up

What a long, strange trip this column has been! What started out as a seemingly simple piece turned into a long trip through P/Invoke and mixed debugging under x64, and led to the ability to detect some of the deadlocks in your applications. Given the Microsoft track record of API improvements over the years, my hope is that you’ll see WaitForMultipleObjects support in WCT in a future service pack. When we get that support, I’ll update LockWatcher to support it!

Tip 78 If you are doing any ASP.NET development, you know that there are times when you just have to look at the raw HTTP information going across the wire. In those cases, Fiddler (www.fiddler2.com) comes to the rescue. Not only can you see every byte, but you can change and tweak the data with a really cool scripting API. And the newest version, Fiddler 2, supports HTTPS interception, even more reason to make it a mandatory part of your toolkit.

Tip 79 Gregg Miskelly came up with a brilliant debugging trick on his blog. In managed code, you don’t have the address of an object, so setting a per instance breakpoint is almost impossible. However, Gregg points out that if you do the cool Make Object Id trick on the instance, you can set a per instance breakpoint by setting a conditional breakpoint to this == 1#. That’s one I could have used a million times over!

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in both .NET and Windows. His latest book is Debugging Microsoft .NET 2.0 Applications (Microsoft Press, 2006). You can contact John at www.wintellect.com.


Security Briefs: Active Directory Cache Dependencies

Security Briefs
Active Directory Cache Dependencies
Keith Brown

Code download available at: SecurityBriefs2007_07.exe (182KB)
Contents
LDAP and SQL Queries
Caching Properties and Objects
SqlDependency and Smarter Caching
The DirSync Control
System.DirectoryServices.Protocols
ADDependency Usage Model
Drilling Deeper
For More Information

If you are in the process of building an application that can be used anywhere in an enterprise, chances are you could benefit by integrating with Active Directory®. There’s a wealth of information right there for the taking if you’re willing to invest just a little time to learn how to program against one of the many APIs that Active Directory supports.

Suppose, for example, your application already uses integrated Windows® authentication to validate a client’s domain identity. You’ve already taken the first—and arguably the biggest—step toward integrating with Active Directory. You are already making use of the user’s existing Active Directory credentials, rather than creating another private user database. This is a great choice because you are making it easy for your application to run in the enterprise without requiring the user to have special credentials for your application.

You may even be taking advantage of techniques such as role-based security, which uses groups in Active Directory to determine the user’s privilege level. You can test for the presence of a particular group by calling the IsInRole method on the WindowsPrincipal class. This is very common.

Take it one step further, though, and things start to get really interesting. If your application is running under a domain account (even a low-privileged one), you should have no problem making queries to Active Directory to learn more about the user. You may be unaware of this, but even if your application is running as Network Service on a domain machine in the lab, that application’s process is represented by the domain credentials of that machine when you use integrated authentication to make secure calls on the network, including calls to Active Directory.

So, alongside that call to IsInRole, you might consider adding code such as that in Figure 1 to retrieve your client’s e-mail address or phone number in an application. Any managed application, from an ASP.NET application to a Windows Communication Foundation service to a Windows Forms desktop app, can benefit from Active Directory integration.

Figure 1 From WindowsIdentity to E-Mail (Close Figure 1)
string email = LookupEmail((WindowsIdentity)User.Identity)

...

public string LookupEmail(WindowsIdentity id)
{
    string path = string.Format(“LDAP://<SID={0}>”, id.User);
    using (DirectoryEntry user = new DirectoryEntry(
        path, null, null, SECURE_BINDING))
    {
        return (string)user.Properties[“mail”].Value;
    }
}

const AuthenticationTypes SECURE_BINDING =
    AuthenticationTypes.Secure | AuthenticationTypes.Signing |
    AuthenticationTypes.Sealing;

There are many useful attributes, such as mail, that you can use to make your application work more seamlessly inside an enterprise. Want to send a friendly e-mail to someone and have it addressed with the user’s first name? The attribute you want is givenName. Want to get the e-mail address of the user’s personal assistant? Just bind to the assistant object for your user and look up his mail attribute.

LDAP and SQL Queries

My code sample in Figure 1 uses a namespace called System.DirectoryServices, which is one of the easiest ways to start integrating with Active Directory. Under the covers, this managed library uses a COM library called Active Directory Services Interface, or ADSI. Under that is a native C library called the LDAP API, where LDAP stands for Lightweight Directory Access Protocol. One of the reasons it’s called lightweight is because it runs over TCP/IP, but that’s a story for another day. Figure 2 illustrates this stack. (I’ll refer back to this figure later in the column.)

Figure 2 Directory Services Stack
Figure 2 Directory Services Stack

When you communicate with Active Directory, you’re ultimately using LDAP, which is both a network protocol and a programming model with its own query language. Simple queries such as the one shown in Figure 1 are quite trivial to implement.

That’s not necessarily the case when you try to do something such as build an org chart, which is entirely possible since Active Directory allows organizations to track relationships among users. Take for example the manager, directReports, department, and title attributes on a user object in Active Directory. Now, if this sort of data were stored in SQL Server™, it would be possible to generate an org chart with very few round-trips to the database. Using the SQL JOIN operator, you can ask the database to build a report and ship it back to you.

LDAP doesn’t have a JOIN operator. And Active Directory doesn’t have anything akin to stored procedures that you can write to put together custom reports like this on the server side. So when your LDAP queries start to get more complex, you may find yourself doing manual joins on the client to put together the data you need. This can result in many round-trips to the domain controller—if you’re not careful, your application might end up wasting network bandwidth while the user waits impatiently at the console for a response.

Caching Properties and Objects

So what’s the solution? Given that directories are designed for read-often, write-seldom operations, a temporary client-side cache is a logical way to help reduce the overhead of complex LDAP queries. In fact, ADSI already implements a property cache for this reason. When the code in Figure 1 asks for the LDAP attribute called mail, the DirectoryEntry class actually downloads all the attributes on the user object making them available locally (in case you’re wondering, you can control this using the RefreshCache method on DirectoryEntry to select exactly which attributes you care about).

The property cache is useful for caching the attributes of a single object. However, when you’re dealing with many objects, such as in the org chart example, you need to cache those objects, not just their properties. And then the question comes up, "Now that I have this cache of objects, how long can I keep it around?" It seems wasteful to just throw this information away if there is a possibility that you will soon need the data again.

ASP.NET has a caching infrastructure that you can use to help manage a local cache of directory objects. It’s pretty easy to drop something in the cache, specifying a time in the future when the cache should become invalid. If you can’t find what you’re looking for in the cache, you know it has expired and you can then go back to the directory and refresh the cache. Here’s the basic pattern:

List<DirectoryEntry> users = Cache[“users”];
if (null == users)
{
    // refresh cache from Active Directory
    users = loadUserListFromAD();
    Cache.Add(“users”, users, null, DateTime.Now.AddMinutes(5),
        Cache.NoSlidingExpiration, CacheItemPriority.Normal, null);
}
displayOrgChart(users);

If you’re getting a lot of requests for this org chart, a cache such as this can be a lifesaver.

SqlDependency and Smarter Caching

The release of ADO.NET 2.0 and SQL Server 2005 brought a new class to help manage local data caches. SqlDependency, in the System.Data.SqlClient namespace, uses a feature of SQL Server 2005 called query notifications. In managed code, you can create an instance of SqlDependency and give it a SqlCommand with a query. It will then notify you if the data represented by that query has changed. To limit the burden on the database, the query you watch has some limitations; but keep in mind that the query you watch does not necessarily have to be exactly the same as the query you use to retrieve the data. The query you use as a dependency could be a superset of the query you actually execute, and you can still reap the benefits of this feature.

This got me thinking. If only there was a corresponding ADDependency class! I’ve done some research in the past trying to figure out how to get notifications from Active Directory when its data changes because local caching of LDAP data is very common. I never did have much luck finding a solution. But I recently chatted with Joe Kaplan (Microsoft MVP and Directory Services guru) and he mentioned the DirSync control. A day later, I had built and started testing the ADDependency class I had always wanted.

The DirSync Control

LDAP version 3.0 includes controls that allow a directory service to expose extended features. For most programmers who use the Microsoft® .NET Framework, the term LDAP control probably conjures up thoughts of Windows Forms controls or ASP.NET server-side controls. But all an LDAP control really consists of is a way to send extra metadata along with an LDAP command to control how it works.

Active Directory exposes several proprietary LDAP controls. The one I am going to use here is called DirSync and, as its name implies, it’s all about synchronizing other data stores with data in Active Directory.

If you think about the way Active Directory works, it’s naturally designed for synchronizing data. One of the most important features of the directory is its ability to replicate data among domain controllers (DCs) in an efficient manner. Objects and properties in Active Directory are stamped with 64-bit Update Sequence Numbers (USNs). I’m dramatically simplifying the story, but imagine that a DC maintains a current USN and increments it anytime anything changes, stamping the object and attributes that changed with the current USN. Each time a sibling requests replication from another DC, the DC need only send the last USN it received from that partner, and the receiver will be able to find the objects and attributes that have changed since the last replication to that particular partner.

One major goal of the Active Directory implementation is to avoid swamping the network with replication traffic. This is why Active Directory pays so much attention to tracking data version numbers. It’s pretty obvious how it works once you learn a little about Active Directory replication, but you can also find some documentation at msdn2.microsoft.com/ms677626.aspx about polling for changes using DirSync.

You use the DirSync control in conjunction with a search request. When you make your request, you also send along this extra metadata (the DirSync control data), which includes a synchronization cookie. On your very first request to Active Directory with a DirSync control, you send a blank cookie and Active Directory responds with all the results of your search, along with a cookie value, which is a byte array. When you want to find out if any changes have taken place since you made your last query, you just send that same cookie back via the DirSync control metadata in your next search request.

While the cookie is opaque, it’s fairly obvious what’s in it. Among other things, you would imagine there being some sort of version number, perhaps even a USN as used in replication. So what do you get back in your second request? Only the objects and attributes that have changed since your first request! It turns out that by using the DirSync control, you can actually get more effective caching than you get with SqlDependency because the directory will send you the deltas, which you can use to keep your cached version up-to-date with the least possible impact on the network. This is a very good thing if you are synchronizing or, as in my case, caching lots of data.

System.DirectoryServices.Protocols

It turns out that there are actually two mainstream techniques you can use to program Active Directory using the .NET Framework 2.0 or later. You can use code similar to what I used in Figure 1, in which case you’re using System.DirectoryServices and the ADSI stack shown in Figure 2. But version 2.0 of the Framework came along with a new API for talking to LDAP directories. This new interface is entirely managed and completely divorced from ADSI, providing a breath of fresh air in many ways. ADSI adds its own set of abstractions on top of LDAP, such as the property cache and implicit connection management, which, while convenient, can be a source of confusion and sometimes makes it a bit unclear when an actual LDAP network request is going to take place.

The newer System.DirectoryServices.Protocols namespace carries none of this ADSI baggage. It’s a lower-level interface that sits right on top of the native LDAP API in WLDAP32.DLL. What I like about this interface is its explicit nature. You know when a TCP/IP connection for LDAP is open because you’re the one who opened it by calling LdapConnection.Bind. You know exactly when network traffic is going to occur because you call LdapConnection.SendRequest to make all of your requests, simply sending different classes of messages depending on what you want to accomplish. And if you want to go asynchronous, there’s the expected BeginSendRequest and EndSendRequest.

Because there are very few samples out there highlighting this new namespace, I decided to dive in head first and build ADDependency using this model. I was surprised at how easy and natural it was to use. But let me warn you up front: The documentation for this new API is rather scarce as of this writing. If you’re new to Active Directory, you may find it quite a bit easier to get started using the traditional ADSI model through System.DirectoryServices and then checking out this new API once you’ve gotten your head around the basics.

ADDependency Usage Model

I modeled ADDependency after SqlDependency. If you’ve used one, it won’t be terribly difficult to use the other. Figure 3 shows the public interface to ADDependency. You start by constructing an instance and specifying at least a search root and LDAP filter, such as:

Figure 3 Public Interface for ADDependency (Close Figure 3)
public ADDependency(string dnSearchRoot, string ldapFilter,
    params string[] attrsToWatch) :
        this(“localhost”, dnSearchRoot, ldapFilter,
            DirectorySynchronizationOptions.ObjectSecurity, attrsToWatch)
{
}

public ADDependency(string adServer, string dnSearchRoot,
    string ldapFilter, DirectorySynchronizationOptions dirSyncOptions,
    params string[] attrsToWatch)
{
    this.adServer = adServer;
    this.dnSearchRoot = dnSearchRoot;
    this.ldapFilter = ldapFilter;
    this.attrsToWatch = attrsToWatch;
    this.dirSyncOptions = dirSyncOptions;
}

public void Start()
{
    if (null != conn) throw new InvalidOperationException(
        “You may only call Start one time.”);
    conn = new LdapConnection(new LdapDirectoryIdentifier(adServer),
        null, AuthType.Negotiate);
    conn.Bind();

    timer = new Timer(timerCallback, null,
        TimeSpan.FromSeconds(0), pollingInterval);
}

public void Stop()
{
    timer.Dispose();
    conn.Dispose();
}

public event ChangedEventHander Changed;

public delegate void ChangedEventHander(object sender,
    ChangedEventArgs args);

public class ChangedEventArgs : EventArgs
{
    public ChangedEventArgs(SearchResponse response)
    {
        this.Response = response;
    }
    public readonly SearchResponse Response;
}

ADDependency dep = new ADDependency(
    “dc=contoso,dc=com”, “(objectclass=user)”);

Once you have an instance of the dependency, you need to keep it around for the duration of whatever data you’re synchronizing or caching—ADDependency will manage the cookie and an underlying LDAP connection to the server. Now you need to wire up to the Changed event to receive change notifications, like so:

dep.Changed += OnChange;
...
void OnChange(object sender, ChangedEventArgs args)
{
    foreach (SearchResultEntry e in args.Response.Entries)
    {
        Console.WriteLine(e.DistinguishedName);
    }
}

The simple handler shown here just prints out the names of all objects that changed recently. In a real implementation, you’d want to update the data in your cache with the attributes on each object in the collection or invalidate a cache and eventually come around and repopulate it by rereading data from the directory.

When you’re ready to open the connection and start the query, simply call start:

dep.Start();

This will start a System.Threading.Timer instance that will, by default, poll Active Directory every N seconds looking to see if anything has changed. N is a constant that you should adjust to balance polling activity with cache coherency. Once running, as long as nothing has changed, this request isn’t very expensive; it’s just a round-trip to the server to find that nothing has changed. But note that the very first time this fires it’s going to return a list of all the objects so that you can initialize your cache.

When you’re finished and want to close the connection, call the Stop method. The current design doesn’t allow you to restart the instance once it’s been stopped (you’ll see an exception if you attempt to do this).

When comparing ADDependency and SqlDependency, you may have noticed a major difference in how they operate. ADDependency polls the directory every N seconds and receives deltas with each request so the user can incrementally update the local cache. SqlDependency, on the other hand, does no polling—it receives notifications pushed from SQL Server 2005 and the caller must then go back to the directory to retrieve the actual data, as opposed to retrieving deltas.

Is polling a bad thing? It can be if you do too much of it. But given that directory data doesn’t change very much, your polling interval shouldn’t be very frequent. As with many decisions, it’s best to measure and decide if this strategy makes sense for you given the data access pattern your application uses.

Drilling Deeper

If you look closely at Figure 3, you’ll see that there’s more to the constructor on ADDependency than I’ve shown so far. For example, you can (and should) pass a list of LDAP attributes that you care about, such as mail, title, manager, and so on. This will limit the size of the initial data you receive and any subsequent deltas. It will also reduce the load on the DC.

On the second, more interesting, constructor, you must specify a particular DC. (The constructor in Figure 1 assumes you’re running on a DC and connects to localhost, which is fine in a lab setting but isn’t likely in an enterprise.) Specifying a particular DC and sticking with it is important. This is because at any given instance in time, two DCs may not necessarily agree on what the truth is about a particular object. Remember, although updates in Active Directory may be relatively infrequent when compared to a SQL database, they do happen occasionally and replication takes a finite amount of time. Besides, it’s not documented whether a DirSync cookie issued by one DC would even make any sense if given to a different DC! However, you should also make the DC configurable rather than hardcoding its name into your app. This could be done via Group Policy, via an application configuration file, or from a method such as System.DirectoryServices.ActiveDirectory.Domain.GetCurrentDomain, which is used to retrieve current domain information for the user.

Just like the queries used with SqlDependency, there are some limitations on the queries you can perform with ADDependency. The search root must be the root of a partition. Note in my example how I set the search root to dc=contoso,dc=com. This is the root of the application partition for the directory hosted at contoso.com. But if I had instead tried to use cn=users,dc=contoso, dc=com to narrow my search, the Start method would have thrown an exception. This is simply a limitation of the DirSync control. You’ll need to use the ldapFilter and attrsToWatch arguments to throttle the data you watch instead.

The magic that makes all this work comes from the DirSyncRequestControl and DirSyncResponseControl classes (see Figure 4). This is how the synchronization cookie (as well as a set of synchronization options) gets passed back and forth to the server. The most important option here is ObjectSecurity, which is only supported by Windows Server® 2003. Without this, you must be highly privileged to use the DirSync control. With this option, you can use a low-privilege account and you’ll only be able to see changes to objects that you would normally be allowed to read, which is probably what you want.

Figure 4 ADDependency Implementation (Close Figure 4)
private void timerCallback(object o)
{
    beginPollDirectory();
}

private void beginPollDirectory()
{
    SearchRequest request = new SearchRequest(dnSearchRoot,
        ldapFilter, SearchScope.Subtree, attrsToWatch);
    request.Controls.Add(
        new DirSyncRequestControl(cookie, dirSyncOptions));
    conn.BeginSendRequest(request, requestTimeout,
        PartialResultProcessing.NoPartialResultSupport,
        endPollDirectory, null);
}

private void endPollDirectory(IAsyncResult asyncResult)
{
    // harvest results
    SearchResponse response =
        SearchResponse)conn.EndSendRequest(asyncResult);

    // grab the current search cookie
    foreach (DirectoryControl control in response.Controls)
    {
        DirSyncResponseControl dsrc = control as DirSyncResponseControl;
        if (null != dsrc) cookie = dsrc.Cookie;
    }

    // if anything changed, notify any listeners
    if (response.Entries.Count > 0 && null != Changed)
    {
        Changed(this, new ChangedEventArgs(response));
    }
}

Note the sequence of events. First, the timer fires, initiating a call to timerCallback. This method leads to an asynchronous request being sent to the DC via LdapConnection.BeginSendRequest. The result is collected in the endPollDirectory method of ADDependency, which then fires off its Changed event, which packages the search response in its event arguments.

For the full implementation of ADDependency, download the sample code associated with this article. It includes a few examples: a console application that monitors user objects in the directory and prints out when they change, an ASP.NET application that implements a custom CacheDependency called ADCacheDependency, and a console application that acts as a random agent of change, having users randomly trade job titles every N seconds. (Ever worked at a place like that?) ADCacheDependency invalidates an ASP.NET cache item when a directory change is detected, similar to the way SqlCacheDependency works.

Before I finish, I want to mention the DirectoryNotification LDAP control, which is an area that deserves future exploration. I’ve not yet seen a single example that uses it, but it appears to allow you to configure an asynchronous search and receive results when the search would return something interesting (changed). It may turn out to be an even lighter-weight technique for detecting changes if you plan on rereading all cached objects from the directory, which is exactly how SqlCacheDependency works in ASP.NET.

For More Information

If this column has piqued your interest in directory service programming, let me give you a tip: Pick up a copy of The .NET Developer’s Guide to Directory Service Programming, by Joe Kaplan and Ryan Dunn. There aren’t many books out there that cover programming Active Directory from managed code, and this is the one you’ll find on my desk whenever I’m doing this type of work.

Now what are you waiting for? Go write some code to find out what treasures await in your local directory!

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is a cofounder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight’s Applied .NET Security course as well as several books, including The .NET Developer’s Guide to Windows Security, which is available both in print and on the Web. Learn more at www.pluralsight.com/keith.


Foundations: Extending the WPF Animation Classes

Foundations
Extending the WPF Animation Classes
Charles Petzold

Code download available at: Foundations2007_07.exe (200KB)
Contents
Interpolated Values
The Structure of the Classes
Freeze Out
Text Morphing
Keyframe Animations
Animating 3D Points

The support for animation in Microsoft® Windows® Presentation Foundation is mostly collected into the System.Windows.Media.Animation namespace. This is a large namespace that defines 164 classes devoted to animating properties of 22 specific data types. These 22 data types range from the basics like byte, int, and double to more exotic types like Matrix, Point3D, and Quaternion.

The animation facilities in Windows Presentation Foundation are surely an impressive piece of work, but only a non-programmer would consider those 22 data types to be sufficient for every application. I personally kept running into applications where I wanted to animate a whole collection of objects, specifically collections of coordinate points. The existing API gives us classes such as PointAnimation and Point3DAnimation for animating individual 2D or 3D coordinates; I wanted PointCollectionAnimation and Point3DCollectionAnimation classes to animate entire collections of points by interpolating between corresponding members of two collections.

We can probably blame the Microsoft .NET Framework in general and Windows Presentation Foundation in particular for making collections of objects almost as easy to define and use as the objects themselves. Windows Presentation Foundation already defines PointCollection and Point3DCollection classes, and in XAML the use of these collections is trivial. In XAML, the difference between assigning a property of type Point and assigning a property of type PointCollection is the difference between typing "100 50" and "100 50, 40 25, 35 80, 50 100." So why, I thought, can’t animating properties of these types be just as easy?

A hypothetical PointCollectionAnimation class would animate properties of type PointCollection. Five classes built into Windows Presentation Foundation have properties of that type: Polyline and Polygon (both derived from Shape), and PolyLineSegment, PolyBezierSegment, and PolyQuadraticBezierSegment (derived from PathSegment), and in all five classes that property is named Points. Animating that Points property lets you change a graphical figure from one shape to another with a single animation.

As far as I can tell, Point3DCollection only shows up in one class, but that’s the MeshGeometry3D class that sits at the center of the Windows Presentation Foundation 3D graphics system. The ability to animate a Point3DCollection would provide a fairly easy way to deform 3D objects—generally considered a rather advanced 3D programming task.

Visions of shifting, morphing, deforming shapes of 2D and 3D figures on the computer screen are what drove my journey to extend the Windows Presentation Foundation animation classes.

Interpolated Values

In Windows Presentation Foundation, animation is basically interpolation, and very often linear interpolation. What makes the coding of new animation classes complex is not the interpolation itself, but determining the exact values to interpolate between.

Let’s look at a simple example involving the animation of a Point value. You can draw a filled circle in XAML like this:

<Path Fill=”Blue”>
    <Path.Data>
        <EllipseGeometry x:Name=”ellipse”
            RadiusX=”10” RadiusY=”10” />
    </Path.Data>
</Path>

Not specified in this markup is the Center property of the EllipseGeometry, which is a property of type Point. You can animate that Center property with a PointAnimation element like this:

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                From=”10 10” To=”100 100”
                Duration=”0:0:3” />

The Center property is animated from the point (10, 10) to the point (100, 100) over three seconds, which means that the PointAnimation class performs a linear interpolation between those two values based on the elapsed time.

You can also specify a Center property in the EllipseGeometry markup like this:

<EllipseGeometry x:Name=”ellipse” Center=”50 50” ...

This is called a base value because the animation can build on this value. You can make use of this base value by omitting the From or To attributes in the PointAnimation markup, as in this markup:

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                To=”100 100”
                Duration=”0:0:3” />

The animation now begins at the center point (50, 50) specified in the EllipseGeometry markup and ends at the point (100, 100). This syntax is also allowed:

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                From=”100 100”
                Duration=”0:0:3” />

Now the animation begins at the point (100, 100) and ends at the point (50, 50). If either the From or To attributes are missing, and the EllipseGeometry markup specifies no Center attribute, then the animation involves the default value of the Center property, which is (0, 0).

It gets messier. An alternative to the To property is the By property, which is an amount added to the starting point.

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                From=”100 100” By=”125 125”
                Duration=”0:0:3” />

This animation goes now from the point (100, 100) to (225, 225). But the From attribute doesn’t need to be present.

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                By=”125 125”
                Duration=”0:0:3” />

Now the animation begins at the point (50, 50) specified in the EllipseGeometry markup and ends with the point (175, 175).

The base value of an animation is often the value that was originally defined for the property being animated, but it doesn’t have to be. Consider the following two sequential PointAnimation objects:

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                To=”100 100” Duration=”0:0:3” />

<PointAnimation Storyboard.TargetName=”ellipse”
                Storyboard.TargetProperty=”Center”
                BeginTime=”0:0:5” Duration=”0:0:3” />

Over the first three seconds, the initial animation changes the Center property from its base value of (50, 50) to the To value of (100, 100), and there the value stays for two more seconds until the second animation kicks in. This animation includes no From, To, or By attributes! The animation begins at a base value of (100, 100) and ends at the value specified in the EllipseGeometry markup: (50, 50).

As you’ll see, a class such as PointAnimation is provided with two default values for the animation to begin and end, and it uses these defaults if the From, To, and By properties are not defined for the particular animation object. Such classes also define two Boolean properties named IsAdditive (which causes the animated values to be added to the base value) and IsCumulative (which accumulates the animated values if the animation is repeating). Neither of these properties makes the animation logic any simpler.

The Structure of the Classes

The various animation classes are often referred to collectively with names like <Type>AnimationBase, which indicates the 22 classes named DoubleAnimationBase, PointAnimationBase, and so on. But don’t assume from this convenient notation that the animation classes are implemented as generics; they really can’t be because each class needs to perform calculations specific to the particular data type. Figure 1 shows a generalized class hierarchy of the animation classes (although there are some variations for the 22 animatable types).

Figure 1 The Animation Class Hierarchy
Figure 1 The Animation Class Hierarchy(Click the image for a smaller view)
Figure 1 The Animation Class Hierarchy
Figure 1 The Animation Class Hierarchy (Click the image for a larger view)

If you decide you want to write classes that animate a custom type, you’ll probably want to mimic the structure of the existing animation classes. You generally begin by defining an abstract <Type>AnimationBase class, and from that class you’ll probably derive a plain <Type>Animation class to perform animations based on linear interpolations. I want to animate properties of type PointCollection, so I began with a class named PointCollectionAnimationBase. Derive this class from AnimationTimeline and define it as abstract. You’ll need to override the read-only TargetPropertyType and return typeof(PointCollection).

In your new <Type>AnimationBase class, you’ll also want to override the GetCurrentValue method defined by AnimationTimeline:

public override object GetCurrentValue(
    object defaultOriginValue,
    object defaultDestinationValue,
    AnimationClock animationClock)
{
    ...
}

This is the method in which your class must calculate an interpolated value for the animation. The parameter names are those used in the documentation. The first two parameters provide the default base values where the animation is to begin and end. You use these parameters in the absence of explicit From, To, and By settings. The AnimationClock object includes a property of type double named CurrentProgress with values ranging from 0 to 1. This is the value you use for the interpolation.

Because you’ve overridden the TargetPropertyType property, WPF knows the type of the objects you wish to animate. If your class is animating objects of type PointCollection, the defaultOriginValue and defaultDestinationValue parameters will be of type PointCollection, and WPF is expecting this method to return an object of type PointCollection.

If you want to emulate the existing animation classes, then GetCurrentValue can simply cast its parameters to the proper type and call another GetCurrentValue method defined in the class, which in turn calls a method named GetCurrentValueCore. For animating PointCollection objects, the code looks like this:

public PointCollection GetCurrentValue(
    PointCollection defaultOriginValue,
    PointCollection defaultDestinationValue,
            AnimationClock animationClock)
{
    return GetCurrentValueCore(defaultOriginValue,
        defaultDestinationValue,
        animationClock);
}

The GetCurrentValueCore method is then defined as protected and abstract:

protected abstract PointCollection GetCurrentValueCore(
    PointCollection defaultOriginValue,
    PointCollection defaultDestinationValue,
    AnimationClock animationClock);

Now any non-abstract class that derives from PointCollectionAnimationBase is required to override this GetCurrentValueCore method and provide actual code for the interpolation.

The downloadable code for this column consists of a single Visual Studio® solution with five projects. One of these projects is a DLL named Petzold.AnimationExtensions, which contains the classes described in this article to animate objects of type PointCollection and Point3DCollection. The other four projects in this Visual Studio solution are demonstration programs that make use of that DLL. The PointCollectionAnimationBase class is part of the DLL and implemented in the file named PointCollectionAnimationBase.cs. All other classes and interfaces in the DLL are likewise in files named after the class or interface.

Freeze Out

All the <Type>AnimationBase classes inherit from AnimationTimeline, which derives from the Freezable class by way of Animatable and Timeline. In fact, all the classes in the Petzold.AnimationExtensions DLL derive from Freezable one way or another, and that fact has some important consequences.

Classes that derive from Freezable have some very special requirements. Ignore these requirements at your own risk. If you do, you might encounter problems where things don’t work correctly and you don’t know why, or you might get strange exceptions that provide no clues to the real problem.

Every non-abstract class that derives from Freezable must override the CreateInstanceCore method. Generally, all this method needs to do is call the class’s constructor. Freezable defines this method as abstract, so you’ll be reminded by the compiler if you don’t include the method. But the compiler will indicate its absence only if your non-abstract class is the first non-abstract class in the hierarchy from Freezable. If you’re deriving from a non-abstract class, it’s easy to forget CreateInstanceCore.

Also, classes that derive from Freezable should back all their properties with dependency properties. It’s not a requirement, but if you don’t do it, you need to override five other methods and learn all about cloning and freezing.

To perform common linearly interpolated animations, you should create a class of the form <Type>Animation that derives from <Type>AnimationBase. (For this example, that class is PointCollectionAnimation, and it performs an interpolation of corresponding points between two PointCollection objects.) This class must override the CreateInstanceCore method, and it should define properties named From, To, and By backed by dependency properties. You should also define IsAdditive and IsCumulative properties; the dependency properties for these two properties are already defined by AnimationTimeline.

The biggest job for <Type>Animation is the override of GetCurrentValueCore. The last argument is an object of type AnimationClock, and the CurrentProgress property is a number between 0 and 1 that indicates the progress of the animation. This value takes account of properties such as AccelerationRatio, DecelerationRatio, AutoReverse, and SpeedRatio defined by the Timeline class and inherited by <Type>Animation. However, GetCurrentValueCore must first determine what it will be interpolating between. For example, if the From property has been set, but the To or By properties have not been set, then the interpolation occurs between From and the defaultDestinationValue parameter to the method.

For this reason, it’s important for GetCurrentValueCore to know when From, To, and By have been set in code or markup and when they have not been set. Most of the existing Windows Presentation Foundation animations are based on value types such as Double or the Point structure. How then can the DoubleAnimation class determine if the From property is its default value of zero, or if it has been specifically set to zero?

Easy: in all the <Type>Animation classes where <Type> is a value type, the From, To, and By properties are defined as nullable types. (This is something I never noticed before because I never had occasion to do anything except set these properties.) The dependency property definitions set the default values to null, which makes it easy for GetCurrentValueCore to determine if the property has been set.

For a class such as PointCollectionAnimation, the animated object is a reference type, so the From, To, and By properties will also be null if they haven’t been set. However, animating reference types involve other, much messier problems. In particular, the last thing you want is for a memory allocation to occur during each call to GetCurrentValueCore. For that reason, your class should create an object of the animated type as a field and return that object from the GetCurrentValueCore method.

Actually, I discovered when working on PointCollectionAnimation that simply returning the same object during successive calls to GetCurrentValueCore was insufficient. Whoever is on the receiving end of GetCurrentValue wants a different object altogether. For that reason, I created two PointCollection objects as fields (named ptsDst1 and ptsDst2 because they were destinations of the interpolation), and switched back and forth between them in successive calls to GetCurrentValueCore based on a Boolean field named flip. I also created a PointCollection object named ptsTo as a field to store the collection of points the method interpolates to. This separate collection became necessary because of the By property, which causes this collection to be a sum of two collections.

The implementation of GetCurrentValueCore in PointCollectionAnimation is shown in Figure 2.

Figure 2 GetCurrentValueCore in PointCollectionAnimation (Close Figure 2)
protected override PointCollection GetCurrentValueCore(
    PointCollection defaultOriginValue,
    PointCollection defaultDestinationValue,
    AnimationClock animationClock)
{
    // Let’s hope this doesn’t happen too often
    if (animationClock.CurrentProgress == null) return null;

    double progress = animationClock.CurrentProgress.Value;
    int count;

    // Set ptsFrom from From or defaultOriginValue
    PointCollection ptsFrom = From ?? defaultOriginValue;

    // Set ptsTo from To, By, or defaultOriginValue
    ptsTo.Clear();

    if (To != null)
    {
        foreach (Point pt in To) ptsTo.Add(pt);
    }
    else if (By != null)
    {
        count = Math.Min(ptsFrom.Count, By.Count);

        for (int i = 0; i < count; i++)
            ptsTo.Add(new Point(ptsFrom[i].X + By[i].X,
                                ptsFrom[i].Y + By[i].Y));
    }
    else
    {
        foreach (Point pt in defaultDestinationValue) ptsTo.Add(pt);
    }

    // Choose which destination collection to use
    PointCollection ptsDst = flip ? ptsDst1 : ptsDst2;
    flip = !flip;
    ptsDst.Clear();

    // Interpolate the points
    count = Math.Min(ptsFrom.Count, ptsTo.Count);

    for (int i = 0; i < count; i++)
    {
        ptsDst.Add(new Point((1 - progress) * ptsFrom[i].X +
                                  progress  * ptsTo[i].X,
                             (1 - progress) * ptsFrom[i].Y +
                                  progress  * ptsTo[i].Y));
    }

    // If IsAdditive, add the base values to ptsDst
    if (IsAdditive && From != null && (To != null || By != null))
    {
        count = Math.Min(ptsDst.Count, defaultOriginValue.Count);

        for (int i = 0; i < count; i++)
        {
            Point pt = ptsDst[i];
            pt.X += defaultOriginValue[i].X;
            pt.Y += defaultOriginValue[i].Y;
            ptsDst[i] = pt;
        }
    }

    // Take account of IsCumulative
    if (IsCumulative && animationClock.CurrentIteration != null)
    {
        int iter = animationClock.CurrentIteration.Value;

        for (int i = 0; i < ptsDst.Count; i++)
        {
            Point pt = ptsDst[i];
            pt.X += (iter - 1) * (ptsTo[i].X - ptsFrom[i].X);
            pt.Y += (iter - 1) * (ptsTo[i].Y - ptsFrom[i].Y);
            ptsDst[i] = pt;
        }
    }

    // Return the PointCollection
    return ptsDst;
}

For any particular animation, the first couple of calls to GetCurrentValueCore might cause some memory allocations as the PointCollection fields are re-allocated to be of sufficient size to store all the points. However, once they’re the proper size, the animation should continue with no further memory allocations.

It wouldn’t make any sense to use PointCollectionAnimation for simply translating, scaling, or rotating a set of points because these operations can be performed with the normal transform classes. This class is most useful for non-linear transforms that fall outside the norm. For example, the SquaringTheCircle.xaml file in Figure 3 animates a PointCollection so a figure changes from a square to a circle with equal area and back to a square again, thus solving a geometry problem that has been baffling Western civilization since the days of Euclid.

Figure 3 SquaringTheCircle.xaml (Close Figure 3)
<Window xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
        xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
        xmlns:pae=”clr-namespace:Petzold.AnimationExtensions;
                   assembly=Petzold.AnimationExtensions”
        x:Class=”SquaringTheCircle.SquaringTheCircle”
        Title=”Squaring the Circle”>
  <Canvas RenderTransform=”2 0 0 -2 300 300”>
    <Path StrokeThickness=”3” Stroke=”Blue” Fill=”AliceBlue”>
      <Path.Data>
        <PathGeometry>
          <PathFigure x:Name=”fig” IsClosed=”True”>
            <PolyBezierSegment x:Name=”seg” />
          </PathFigure>
          <PathGeometry.Transform>
            <RotateTransform Angle=”45” />
          </PathGeometry.Transform>
        </PathGeometry>
      </Path.Data>
    </Path>
  </Canvas>

  <Window.Triggers>
    <EventTrigger RoutedEvent=”Window.Loaded”>
      <BeginStoryboard>
        <Storyboard RepeatBehavior=”Forever” AutoReverse=”True”>
          <PointAnimation Storyboard.TargetName=”fig”
                          Storyboard.TargetProperty=”StartPoint”
                          From=”0 100” To=”0 125” />

          <pae:PointCollectionAnimation
            Storyboard.TargetName=”seg”
            Storyboard.TargetProperty=”Points”
            From=”  55  100,  100   55,  100    0,
                   100  -55,   55 -100,    0 -100,
                   -55 -100, -100  -55, -100    0,
                  -100   55,  -55  100,    0  100”
            To=” 62.5  62.5,  62.5  62.5,  125    0,
                 62.5 -62.5,  62.5 -62.5,    0 -125,
                -62.5 -62.5, -62.5 -62.5, -125    0,
                -62.5  62.5, -62.5  62.5,    0  125” />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Window.Triggers>
</Window>

A functionally equivalent program in my WPF book actually required 13 individual Point animations. I certainly prefer the elegance of this new version.

Text Morphing

The TextMorphCheckBox project defines a template for a CheckBox control that overlays the CheckBox content (if any) with a Polyline element that references one of two PointCollection objects defined as template resources. When you click the CheckBox, it performs an animation between that PointCollection object and the other one. A good chunk of the ControlTemplate markup is shown in Figure 4.

Figure 4 TextMorphCheckBox.xaml Excerpt (Close Figure 4)
<ControlTemplate x:Key=”templateTextMorph”
                 TargetType=”{x:Type CheckBox}”>
  <ControlTemplate.Resources>
    <PointCollection x:Key=”textYes”>
       5 11,  3 10,  2  8,  2  7,  3  5,  5  4,  6  4,  8  5,
       9  7,  9  9,  8 13,  7 16,  6 20,  6 22,  7 24,  8 25,
      10 25, 12 24, 14 22, 16 19, 17 17, 19 11, 21  4, 19 11,
      16 21, 14 27, 12 32, 10 36,  8 37,  7 36,  7 34,  8 31,
      10 28, 13 26, 17 25, 23 23, 25 22, 26 21, 27 19, 27 17,
      26 16, 25 16, 23 17, 22 19, 22 22, 23 24, 25 25, 27 25,
      29 24, 30 23, 32 20, 34 17, 35 15, 35 17, 37 20, 38 22,
      38 24, 36 25, 32 24, 34 25, 38 25, 40 24, 41 23, 43 20
    </PointCollection>

    <PointCollection x:Key=”textNo”>
       5 11,  3 10,  2  8,  2  7,  3  5,  5  4,  6  4,  8  5,
       9  7,  9  8,  9  9,  8 14,  7 18,  5 25,  7 18, 10 10,
      11  8, 12  6, 13  5, 15  4, 17  4, 19  5, 20  7, 20  8,
      20  9, 19 14, 17 21, 17 24, 18 25, 19 25, 21 24, 22 23,
      24 20, 25 18, 26 17, 28 16, 29 16, 30 16, 29 16, 28 16,
      26 17, 25 18, 24 20, 24 21, 24 22, 25 24, 27 25, 28 25,
      29 25, 31 24, 32 23, 33 21, 33 20, 33 19, 32 17, 30 16,
      29 17, 29 18, 29 19, 30 21, 32 22, 35 22, 37 21, 38 20
    </PointCollection>
  </ControlTemplate.Resources>
  ...
  <!-- This Border displays the text -->
  <Border>
    <Border.Background>
      <VisualBrush Stretch=”Uniform”>
        <VisualBrush.Visual>
          <Polyline Name=”polyline”
                    Stroke=”{TemplateBinding Foreground}”
                    StrokeThickness=”2”
                    StrokeStartLineCap=”Round”
                    StrokeEndLineCap=”Round”
                    StrokeLineJoin=”Round”
                    Points=”{StaticResource textNo}” />
        </VisualBrush.Visual>
      </VisualBrush>
    </Border.Background>
  </Border>
  ...
  <!-- Triggers convert text from No to Yes and back -->
  <ControlTemplate.Triggers>
    <EventTrigger RoutedEvent=”CheckBox.Checked”>
      <BeginStoryboard>
        <Storyboard TargetName=”polyline”
                    TargetProperty=”Points”>
          <pae:PointCollectionAnimation
                    To=”{StaticResource textYes}” />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>

    <EventTrigger RoutedEvent=”CheckBox.Unchecked”>
      <BeginStoryboard>
        <Storyboard TargetName=”polyline”
                    TargetProperty=”Points”>
          <pae:PointCollectionAnimation
                    To=”{StaticResource textNo}” />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  ...
  </ControlTemplate.Triggers>
</ControlTemplate>

The two PointCollection objects used in this animation contain definitions for the words No and Yes in a script font. By animating between these two collections, the text morphs from one word into another over the course of a second, as shown in Figure 5.

Figure 5a The Morphing Text CheckBox
Figure 5a The Morphing Text CheckBox

Figure 5b

Figure 5c

Figure 5d

Figure 5e

Figure 5f

The fact that you can click the CheckBox midway through the morphing text and the animation will reverse itself is a good indication that the PointCollectionAnimation class is handling the parameters passed to the GetCurrentValueCore method correctly.

Let me explain where the two polylines in the TextMorphCheckBox template came from. Some of the geezers who worked with Windows in those far distant days when "Rock Me Amadeus" was a hit might remember the existence of primitive vector fonts in Windows. These fonts are still alive in Windows Vista™ and can be found in the files Modern.fon, Roman.fon, and Script.fon. Characters in these fonts are defined simply by straight line segments and were often used with plotters. There are no curves, no hinting, and no filling as in TrueType fonts. You can select these old vector fonts in Notepad, but Microsoft Word would positively die before inserting one of these hideous things into its gorgeous documents.

If you know a little bit about font file formats, it’s easy to crack these font files and extract the polylines that define each character. After that job, I adjusted the coordinates of the individual characters by hand to create single continuous polylines for the words No and Yes, and then tweaked them some more to ensure that both PointCollection objects had the same number of points. I was mentally prepared to convert the coordinates to a series of quadratic Bezier splines by inserting some control points, but I didn’t need to. With a suitably thick stroking pen, the polylines seemed adequate for this odd purpose.

Keyframe Animations

Programmers who have worked with Windows Presentation Foundation animation know that the <Type>Animation classes are really just an easy alternative to the more versatile <Type>AnimationUsingKeyFrames classes. The keyframe animations let you mix a series of discrete jumps, linear animations, and animations that slow down and speed up based on a Bezier spline segment. Each keyframe consists of a time and a value. The animation interpolates between a base value (which is generally the ending value of the previous keyframe) and the keyframe value.

I expected that implementing a PointCollectionAnimationUsingKeyFrames class would be challenging, and I wasn’t disappointed. The first step was to derive an abstract PointCollectionKeyFrame class that inherits from Freezable and implements the IKeyFrame interface. This requires the definition of KeyTime and Value properties that you’ll want to back by dependency properties. I mimicked the existing keyframe animation classes by defining a public method named InterpolateValue in PointCollectionKeyFrame:

public PointCollection InterpolateValue(
    PointCollection baseValue,
    double keyFrameProgress)
{
    return InterpolateValueCore(baseValue, keyFrameProgress);
}

I also defined a protected and abstract InterpolateValueCore method. Notice that the second parameter is a progress value between 0 and 1, but it’s a progress value for the particular key frame and not for the animation as a whole.

The next step is to derive three classes from PointCollectionKeyFrame named DiscretePointCollectionKeyFrame, LinearPointCollectionKeyFrame, and SplinePointCollectionKeyFrame. These classes override the InterpolateValueCore method to perform an interpolation between the baseValue parameter and the Value property. SplinePointCollectionKeyFrame also defines a property named KeySpline. Once again, I defined two fields named ptsDst1 and ptsDst2 and a Boolean named flip for the result. I realized I could avoid some repetitive code by first deriving an abstract NonDiscretePointCollectionKeyFrame class from PointCollectionKeyFrame that contains the InterpolateValueCore method shown in Figure 6.

Figure 6 InterpolateValueCore Method (Close Figure 6)
protected override PointCollection InterpolateValueCore(
    PointCollection baseValue, double keyFrameProgress)
{
    // Choose which destination collection to use.
    PointCollection ptsDst = flip ? ptsDst1 : ptsDst2;
    flip = !flip;
    ptsDst.Clear();

    int num = Math.Min(baseValue.Count, Value.Count);
    double progress = GetProgress(keyFrameProgress);

    for (int i = 0; i < num; i++)
        ptsDst.Add(new Point(
            (1 - progress) * baseValue[i].X +
                 progress  * Value[i].X,
            (1 - progress) * baseValue[i].Y +
                 progress  * Value[i].Y));
    return ptsDst;
}

Notice that the method calls a method named GetProgress, which NonDiscretePointCollectionKeyFrame defines as abstract. Both LinearPointCollectionKeyFrame and SplinePointCollectionKeyFrame override this method. LinearPointCollectionKeyFrame simply returns the parameter; SplinePointCollectionKeyFrame returns a value conveniently provided by a method defined by the KeySpline structure, which converts a linear progress value into a progress based on the Bezier spline.

Next you’ll need a class of the form <Type>KeyFrameCollection, which is a freezable collection of <Type>KeyFrame objects. In my example, this collection class has a rather confusing name, PointCollectionKeyFrameCollection, and is a collection of PointCollectionKeyFrame objects. You can save yourself a whole lot of work by defining this collection class as inheriting from the generic FreezableCollection<T> class. Don’t forget: when deriving from FreezableCollection<T>, you must override CreateInstanceCore.

public class PointCollectionKeyFrameCollection :
    FreezableCollection<PointCollectionKeyFrame>
{
    // CreateInstanceCore override required when
    // deriving from Freezable
    protected override Freezable CreateInstanceCore()
    {
        return new PointCollectionKeyFrameCollection();
    }
}

The final and most difficult step is the <Type>AnimationUsingKeyFrames class that derives from <Type>AnimationBase and implements the IKeyFrameAnimation interface by defining a KeyFrames property (and dependency property) of type <Type>KeyFrameCollection. The <Type>AnimationUsingKeyFrames class overrides the GetCurrentValueCore method, but must determine which of the objects in its KeyFrames collection should actually perform the interpolation.

While working on this class, I realized that the IsCumulative property was not very useful, but accumulating the animation based on a single point might provide some interesting utility. I defined a new property (and dependency property) named AccumulationPoint, which is a single point that gets added to the collection during each iteration. This let me create an animation I called StickFigureWalking. It is based on five PointCollection objects and five keyframes with an accumulation point that make a stick figure walk across the screen, as shown in Figure 7.

Figure 7 Walking
Figure 7 Walking

Animating 3D Points

When you create your own animation classes, you can mimic the existing classes (as I did with the PointCollection animations), or you can abandon that paradigm and strike out on your own. This is what I did with animating objects of type Point3DCollection, which is the data type of the Positions property of MeshGeometry3D. I defined a normal Point3DCollectionAnimationBase class, but then pursued a different strategy with a class named Point3DCollectionAnimationUsingDeformer. The Point3DCollectionAnimationUsingDeformer defines just one property named Deform of type IDeformer, an interface that defines just one method:

Point3DCollection Deform(Point3DCollection points, double progress);

The Deform method accepts a collection of 3D points and a progress value that ranges from 0 to 1, and it returns a collection of 3D points. The intent is to allow non-linear transforms that can’t be accomplished with the standard classes. Exactly how Deform transforms one set of points to another is limited only by your imagination—and perhaps your math skills.

I provide one example: a class named Twister that twists a 3D object. Twisting is similar to the standard AxisAngleRotation3D transform available to 3D programmers and involves a rotation around an axis, except that the angle of rotation for a particular point varies with the distance of that point from a center point.

The Twister class implements the IDeformer interface and defines several properties on its own. The Axis and Center properties indicate the axis of rotation expressed as a vector and the rotation center. The MinAngle and MaxAngle properties specify the rotation angles when the progress value is 0 and 1, respectively. These angles are per unit of distance along the rotation axis. For example, imagine the Axis vector passing through the Center point. The plane of points that include that Center point and are perpendicular to Axis do not rotate at all. The plane one unit from the Center point rotates between MinAngle and MaxAngle. The plane two units from the Center point rotates between twice MinAngle and twice MaxAngle. On the other side of the Center point, points rotate in the opposite direction.

Figure 8 shows the Deform method in the Twister class. Notice that it flips between two pre-created Point3DCollection classes for the destination collection, and that it also uses a pre-created RotateTransform3D object named xform and an AxisAngleRotation3D object named axisRotate.

Figure 8 The Deform Method in Twister (Close Figure 8)
public Point3DCollection Deform(Point3DCollection pointsSrc,
    double progress)
{
    // Clear destination collection
    Point3DCollection pointsDst = flip ? pointsDst1 : pointsDst2;
    flip = !flip;
    pointsDst.Clear();

    // Prepare RotateTransform3D object using AxisAngleRotation3D
    xform.CenterX = Center.X;
    xform.CenterY = Center.Y;
    xform.CenterZ = Center.Z;

    // Prepare AxisAngleRotation3D object
    axisRotate.Axis = Axis;
    Vector3D axisNormalized = Axis;
    axisNormalized.Normalize();
    double angleAttenuated = MinAngle + progress * (MaxAngle - MinAngle);

    // Transform each point based on its distance from the center
    foreach (Point3D point in pointsSrc)
    {
        axisRotate.Angle = angleAttenuated *
            Vector3D.DotProduct(axisNormalized, point - Center);

        pointsDst.Add(xform.Transform(point));
    }

    // Return the points
    return pointsDst;
}

The real fun comes with finding objects to twist. The TeapotTwister project includes a TeapotMesh.cs file that generates a MeshGeometry3D object for the famous Iowa Teapot used in countless 3D demonstrations. I created this file from data available from the static Mesh.Teapot method in DirectX® 9.0. Figure 9 shows the complete XAML file that references the TeapotMesh class, the Point3DCollectionAnimationUsingDeformer class, and Twister. The twisted teapot is shown in Figure 10.

Figure 9 TeapotTwister.xaml (Close Figure 9)
<Window xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
        xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
        xmlns:pae=”clr-namespace:Petzold.AnimationExtensions;
                   assembly=Petzold.AnimationExtensions”
        xmlns:src=”clr-namespace:TeapotTwister”
        x:Class=”TeapotTwister.TeapotTwister”
        Title=”Teapot Twister (3D Deformation)”>
  <Window.Resources>
    <src:TeapotMesh x:Key=”teapot” />
  </Window.Resources>

  <Viewport3D>
    <ModelVisual3D>
      <ModelVisual3D.Content>
        <Model3DGroup>

          <!-- 3D teapot geometry and materials -->
          <GeometryModel3D x:Name=”geomod”
            Geometry=”{Binding Source={StaticResource teapot},
                       Path=Geometry}”>
            <GeometryModel3D.Material>
              <DiffuseMaterial Brush=”Cyan” />
            </GeometryModel3D.Material>

            <GeometryModel3D.BackMaterial>
              <DiffuseMaterial Brush=”Blue” />
            </GeometryModel3D.BackMaterial>
          </GeometryModel3D>

          <!-- Light sources -->
          <AmbientLight Color=”#404040” />
          <DirectionalLight Color=”#C0C0C0” Direction=”2 -3 1” />
        </Model3DGroup>
      </ModelVisual3D.Content>
    </ModelVisual3D>

    <!-- Camera -->
    <Viewport3D.Camera>
      <PerspectiveCamera Position=”0 0 6”
                         LookDirection=”0 0 -1”
                         UpDirection=”0 1 0”
                         FieldOfView=”45” />
    </Viewport3D.Camera>
  </Viewport3D>

  <!-- Animation using Twister class -->
  <Window.Triggers>
    <EventTrigger RoutedEvent=”Window.Loaded”>
      <BeginStoryboard>
        <Storyboard TargetName=”geomod”
                    TargetProperty=”Geometry.Positions”>
          <pae:Point3DCollectionAnimationUsingDeformer
                    Duration=”0:0:5” AutoReverse=”true”
                    RepeatBehavior=”Forever”>
            <pae:Point3DCollectionAnimationUsingDeformer.Deformer>
              <pae:Twister Axis=”1 0 0” />
            </pae:Point3DCollectionAnimationUsingDeformer.Deformer>
          </pae:Point3DCollectionAnimationUsingDeformer>
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Window.Triggers>
</Window>
Figure 10 A Twisted Teapot
Figure 10 A Twisted Teapot

The XAML file shows the Twister class referenced in a property element of Point3DCollectionAnimationUsingDeformer, but it’s also possible to define the Twister class as a resource:

   <pae:Twister x:Key=”twister” Axis=”1 0 0” />

You then reference it with an attribute in the Point3DCollectionAnimationUsingDeformer tag:

Deformer=”{StaticResource twister}”

While dynamically modifying the Positions collection of a MeshGeometry3D is a powerful technique, it is theoretically not quite adequate. Whatever non-linear transform you make to the Point3D objects in the Positions collection of MeshGeometry3D should also be applied to the Vector3D objects in the Normals collection. Although explicit casts are defined between Point3D and Vector3D, no such cast exists for collections of these objects. This seems to imply that a whole parallel structure of classes needs to be created for animating Vector3DCollection objects.

Even without that enhancement, these new animation classes have satisfied one of my basic motivations in programming: to see visuals on my computer—such as morphing text and twisting teapots—the likes of which I’ve never seen before.

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a Contributing Editor to MSDN Magazine and the author of Applications=Code+Markup: A Guide to the Microsoft Windows Presentation Foundation (Microsoft Press, 2006).


{ End Bracket }: Weaving Your Photos with Photosynth

{ End Bracket }
Weaving Your Photos with Photosynth
Richard Szeliski

The explosion of digital photography in the past several years has led to a dramatic increase in the number of photos that are shared on the Web. The Interactive Visual Media Group at Microsoft Research has responded to this digital photography phenomenon by developing innovative photo-editing and photo-viewing products. Microsoft Research was among the first to produce fully automated image-stitching software and to develop 360-degree video-based walkthroughs as well as generate 3D videos that can be navigated in real time (see research.microsoft.com/IVM).

One of the biggest challenges in this area has been the development of fully automated systems to build photorealistic 3D models from collections of photographs. While recent advances in this field have been dramatic—such as the texture-mapped 3D building models available in Virtual Earth™ 3D (see maps.live.com)—the resulting models did not achieve the full richness and variety inherent in the original photo collections.

To overcome these limitations, I partnered with Noah Snavely and Steve Seitz, both from the University of Washington, in developing a 3D photo-browsing system called Photo Tourism (see phototour.cs.washington.edu). Our system uses computer-vision techniques to reconstruct a partial 3D model of the scene being photographed, along with the 3D position and orientation of each image in a collection. This is accomplished by first extracting distinctive feature points in each image, matching these across the whole collection, and then incrementally reconstructing the 3D camera and scene geometry by solving a large sparse non-linear optimization problem. The user then can navigate from image to image by selecting regions of interest or using intuitive commands such as move left or move right.

To smooth transitions between images and give a sense of 3D motion, our system simulates 3D camera moves while projecting the images onto planar "impostors" (proxies), a technique sometimes used in computer games to model distant geometry. A sketched 3D model of the scene, consisting of a point cloud, line segments, and low-resolution "watercolor washes" is also used. The resulting 3D image browser combines the realism and the beauty of a traditional slideshow with the 3D interactivity of video games.

Once we had the research prototype built, we needed to scale our system to support real-time, multi-resolution streaming of images over the Internet in order to move from the lab environment to general deployment. Microsoft® Live Labs had just acquired a startup called Seadragon, which already had technology that streams images at a variety of resolutions and displays them as animated 3D arrangements. The Seadragon engine converts original images into a set of multi-resolution, overlapping, tiled image fragments, which are streamed on an as-needed basis and combined on the client side to provide visually seamless zooming and progressive image refinement while simultaneously supporting the display of thousands of images.

A small, cross-functional group of designers, program managers, researchers, and developers worked together to design, build, and deploy Photosynth™, a system that combines the 3D image placement and navigation technologies with the underlying Seadragon engine. The initial user interface, which relied on a number of thumbnail panes and icons to control the navigation, underwent several rounds of redesign. In the final design, a glowing quadrilateral indicates the presence of additional images in the vicinity of the user’s mouse. Clicking on the quadrilateral seamlessly transitions the view to the new image. Also added were a 3D overview of the scene and a 2D, scaled thumbnail view of the images as well as the filmstrip control to sequence through the images and to step back through the navigation history (see labs.live.com/Photosynth).

Plans for future versions of Photosynth include improving the 3D reconstruction algorithms to scale to larger data sets and to enable consumer-level authoring, and also to extend the viewer to additional platforms. We are investigating how to integrate Photosynth with regular slideshows and photo-sharing sites, how to achieve smooth circumnavigation of 3D objects, and how to integrate with 3D city models such as those in Virtual Earth. We are also exploring additional applications of Photosynth beyond tourism, such as education (including interactive museum tours and historical time lapses), real estate sales, and indexing large image collections. Our hope is that Photosynth will become an established visual medium combining the beauty and the richness of traditional photography with the interactive exploration inherent in games and 3D worlds.

Richard Szeliski leads the Microsoft Research Interactive Visual Media Group, which invents new ways for people to capture, explore, and share their personal memories.