 MSDN Magazine January 2007

MSDN Magazine January 2007

Least Privilege: Teach Your Apps To Play Nicely With Windows Vista User Account Control

User Account Control in Windows Vista keeps the OS safe from intentional and accidental configuration changes.

Chris Corio Vista and Office: View Data Your Way With Our Managed Preview Handler Framework

Stephen Toub shows you how to write your own Preview handlers for Windows Vista and Outlook 2007.

Stephen Toub SideShow Gadgets: Get Started Writing Gadgets For Windows SideShow Devices

SideShow Gadgets for Windows Vista are cool. Writing your own is even better. Find out how it's done.

Jeffrey Richter Desktop Security: Create Custom Login Experiences With Credential Providers For Windows Vista

Why is a change to the Windows logon plug-in interface so exciting? Because with credential providers you can customize the logon experience for your users.

Dan Griffin Worldly Windows: Extend The Global Reach Of Your Applications With Unicode 5.0

When using global applications, people want to communicate with others on their terms, in their own languages. Unicode 5.0 brings that ideal closer to reality.

Julie D. Allen, Michael S. Kaplan, Cathy Wissink Debug Leaky Apps: Identify And Prevent Memory Leaks In Managed Code

When is the .NET Garbage Collector unable to reclaim memory? The answer might surprise you. Stay tuned.

James Kovacs Editor's Note: A New Year—A New Vista

Josh Trupin revs the engine to get us all ready for Windows Vista.

Joshua Trupin Toolbox: Save Coding Time, Manage Compressed Files, and More

Save Coding Time, Manage Compressed Files, and More

Scott Mitchell CLR Inside Out: Introduction to COM Interop

The CLR allows seamless interactions between Microsoft .NET applications and COM. But how, exactly? The CLR team knows.

Thottam R. Sriram Data Points: Column Expressions, DataRelations, and Computations

This month John Papa fields some of his favorite questions regarding data manipulation with ADO.NET.

John Papa Cutting Edge: The Server Side of ASP.NET Pages

Where are the files generated by ASP.NET stored and how are they used to serve page requests? This month Cutting Edge explains.

Dino Esposito Foundations: Using Templates to Customize WPF Controls

This month we welcome Charles Petzold to MSDN Magazine with his first column on building apps for Windows Vista and the .Microsoft NET Framework 3.0.

Charles Petzold Team System: Team Foundation Server Version Control

In this new column, Brian Randell begins his long look at how to extend and enhance Visual Studio Team System.

Brian A. Randell Extreme ASP.NET: Client-Side Web Service Calls with AJAX Extensions

Microsoft AJAX Library and the ASP.NET 2.0 AJAX Extensions provide a number of compelling features ranging from client-side data binding, to DHTML animations and behaviors. Learn all about them here.

Fritz Onion Security Briefs: Using Protocol Transition—Tips from the Trenches

Now that Windows Server 2003 is widely deployed, Keith Brown addresses questions from readers who are trying to use protocol transition to build secure gateways into their intranets.

Keith Brown { End Bracket }: From Bouncy Balls to Better Estimates

This month End Bracket explores the reasons why predicting the amount of development effort is extremely difficult early in a project.

James Waletzky


Least Privilege: Teach Your Apps To Play Nicely With Windows Vista User Account Control

Least Privilege

Teach Your Apps To Play Nicely With Windows Vista User Account Control

Chris Corio
This article discusses:
  • User Account Control
  • Designing applications to run as standard users
  • Elevating privileges when neccessary
This article uses the following technologies:
Windows Vista
Contents
Active at Logon
Creating a Filtered Token
Designing Apps to Run with the Filtered Token
Designing Apps that Require Administrator Privileges
Privileges During Installation
Marking the Privileges for an MSI
Running Your App with Administrator Privileges
Marking Required Privileges Using an Application Manifest
Marking a COM Object
Alerting Users to Elevated Privileges
Communicating between Processes in Different Security Contexts
Low Rights Internet Explorer
Securing a Windows Service in Windows Vista
Common Application Compatibility Issues
Problems During Installation
Typical Runtime Application Issues
Conclusion

I
have worked with so many developers over the past year that their concerns and their eventual understanding of how User Account Control (UAC) works have all blended into a giant haze. There really is no doubt: it will take time to learn how to write good applications for standard users, but this knowledge will make you a better programmer for Windows®. The goal of this article is to give you the information necessary to make the journey quick and easy.

UAC is the Microsoft answer to reducing the privileges users run with by default in Windows Vista™. Strategically, Microsoft is moving to an environment where users do not have or need privileges that can affect the operating system and machine-wide configuration in order to perform day-to-day tasks. Such an environment also keeps users from affecting each other's state and settings.

There are countless practical reasons for this move. UAC limits the privileges that applications run with by default. In the enterprise, this lets users run in the Users group rather than as Administrators, allowing meaningful corporate security policies to be enforced. This in turn reduces user impact on machines, lowering the total cost of ownership (TCO). In the home, malware cannot affect the system or install a service or driver without the user going through the UAC Consent or Credential prompt. While the risk of social engineering still exists, the consolidation of the path to obtain elevated privileges allows the future implementation of technologies to control which applications can execute with machine-wide privileges without the need to prompt the user.

The primary goal of UAC is to make the default user token that applications are run with mimic that of a member of the Users group. This starts with creating a restricted, or filtered, token during interactive logon for users with elevated privileges. When more privileges are necessary to execute a restricted operation, the user will be prompted for authorization on the secure desktop. UAC also includes other technologies to support application compatibility and to secure the elevated processes.

Active at Logon

UAC starts working when a user logs onto a machine. During an interactive logon, the Local Security Authority (LSA) takes the user's credentials and performs the initial logon, evaluating the user's token to see if it has what are defined as elevated privileges. If the LSA determines that the user has elevated privileges, it will filter this token and then perform a second logon with the filtered token. The most common case in which a user will have both a filtered and fully privileged token is when the user is a member of the Administrators group. The fully privileged token of an administrator can write to machine-wide locations such as Program Files and HKEY_LOCAL_MACHINE (HKLM), thus affecting the entire machine. However, the filtered token is just like a member of the Users group, which doesn't have these privileges, and is used to create the user's interactive desktop session.

The desktop session and explorer.exe will always be created with a token that approximates the token of a member of the Users group. Any process that is initiated from the Start Menu or by a user double-clicking in an Explorer window that doesn't require elevation will simply inherit this filtered token. Therefore, by default, every application will be running with the standard user token. When a process is marked as requiring administrator privileges in the manifest or in the application compatibility settings, UAC will prompt the user for elevation. Figures 1 and 2 show the prompts for consent and for credentials. If the user so authorizes, the Application Information service will create the process with the fully privileged token.

Figure 1 Prompt for Consent
Figure 1 Prompt for Consent(Click the image for a smaller view)
Figure 1 Prompt for Consent
Figure 1 Prompt for Consent (Click the image for a larger view)
Figure 2 Prompt for Credentials
Figure 2 Prompt for Credentials(Click the image for a smaller view)
Figure 2 Prompt for Credentials
Figure 2 Prompt for Credentials (Click the image for a larger view)

Creating a Filtered Token

User Account Control defines the groups listed in Figure 3 as having elevated privileges. Therefore, if the LSA notices that any of those group memberships or privileges are listed in the user's initial token, a filtered token will be created during an interactive logon, using a version of the CreateRestrictedToken API, and the fully privileged token is saved by LSA. These two tokens are linked and the fully privileged token can be obtained from the filtered token using the GetTokenInformation API with the new TokenLinkedToken information type. Note, however, that UAC does not affect service, network, or batch logons.

Figure 3 Elevated Privelages (Close Figure 3)

Built-In Administrators
Power Users
Account Operators
Server Operators
Printer Operators
Backup Operators
RAS Servers Group
Windows NT 4.0 App Compat Group
Network Configuration Operators
Domain Administrators
Domain Controllers
Certificate Publishers
Schema Administrators
Enterprise Administrators
Group Policy Administrators

If the user does not belong to any of the groups listed in Figure 3 but has certain privileges, a filtered token will be created with these privileges removed. The privileges in question are: SeCreateTokenPrivilege, SeTcbPrivilege, SeTakeOwnershipPrivilege, SeBackupPrivilege, SeRestorePrivilege, SeDebugPrivilege, SeImpersonatePrivilege, and SeRelabelPrivilege.

If the user is a member of the Administrators group, the filtered token will have the Administrators group membership set to DENY ONLY, which prohibits AccessCheck from using this group to allow access to a resource. Also, all machine-impacting privileges are removed from the token. (This is the default token used to create explorer.exe and non-elevated processes.) You can look at this token using whoami.exe, which is included in Windows Vista; just run whoami.exe /all in a Command window. Figure 4 and Figure 5 respectively describe group memberships listed in the token and privileges in the token.

Figure 5 Privelages Included in Filtered Token (Close Figure 5)

Privilege | Description
SeShutdownPrivilege | Shut down the system.
SeChangeNotifyPrivilege | Bypass traverse checking.
SeUndockPrivilege | Remove computer from docking station.
SeIncreaseWorkingSetPrivilege | Increase a process working set.
SeTimeZonePrivilege | Change the time zone.
Figure 4 Group Membership Listed in Filtered Token (Close Figure 4)

Group Name | Attributes
Group Name | Attributes
Everyone | Mandatory group, enabled by default, enabled group
BUILTIN\Administrators | Group used for deny only
BUILTIN\Users | Mandatory group, enabled by default, enabled group
Mandatory Label\Medium Mandatory Level | Mandatory group, enabled by default, enabled group

Designing Apps to Run with the Filtered Token

Most applications do not require administrator privileges at run time. If your application doesn't maintain cross-session state while it executes and doesn't do something like modifying the local security policy, it should be just fine running with a standard-user token. Sometimes certain parts of your application will require administrator privileges, and you should separate out those pieces into a separate process. I'll get into that a little later.

The most important step you can take during the development of a standard user application is to test it while running as a standard user. One of the most common reasons an application fails in production is because the developers never tested it as a standard user. Remember to do this.

The next most important step is deciding where to save your application binaries and per-user configuration data. An application that is completely per-user, meaning that both its binaries and configuration are specific to the user, should maintain all of its state in the user profile: %userprofile%. In the registry, you should write all of the state to the user's hive, HKEY_CURRENT_USER (HKCU). In many cases, it makes sense to install the binaries into the %ProgramFiles% directory but this requires administrator privileges and is discussed later in the article.

Sometimes, as when you're storing high scores for a game, you need to maintain system-wide state and allow users to write to it. The recommended place to store this data is in the %allusersprofile% directory. Here you can create a directory specific to your application and allow users to write to it. Just remember, other users can tamper with this file.

Designing Apps that Require Administrator Privileges

My honest advice about writing applications that require administrator privileges is: don't do it! Unless you're absolutely certain that your application requires elevated privileges, just write your application to use the standard user token. There are very few APIs that must be called by administrators.

Of course, there are circumstances where you might need to use administrator privileges, like during installation or when you're maintaining cross-session state, which generally means your application is implemented as a service. UAC doesn't apply to services; there are other security settings available for Windows services that I'll discuss later. If your application implements a security model on the local machine, you'd also need administrator privileges. For example, if you have settings that are deployed with a Group Policy Application Distribution and Management (ADM) or through Systems Management Server (SMS), you will probably have an application on the local machine to update them as well. This would require administrator privileges.

Privileges During Installation

Installing applications typically requires administrator privileges, usually because application binaries are written to the Program Files directory, which is read-only for standard users. This means that any application that writes binaries to this location will need to be marked as requiring administrator privileges. Because Windows Installer has been integrated with UAC, installation packages (MSI files) have become an even more attractive method for installation. They can be specified in two ways: to be per-user or per-machine installations, which would only prompt the user for administrator privileges when necessary.

While installation to the Program Files directory will result in an elevation prompt being shown to users, there are definitely benefits to putting the application's binaries there. Installing in that directory requires the user to be an administrator so a standard user in an enterprise or a Parental Controls user couldn't arbitrarily install the application. Also, your application will share one set of binaries across the machine and they will be protected from tampering by a standard user. This will make it easier to determine the state and version of the application when updating the binaries after a bug fix or to add functionality.

When you design your application, there will likely be some application settings you'll want applied for the user. Perhaps the users specify custom information about how their views are shown or which menus are displayed. A good way to provide this facility is to create a default template under the Program Files directory, then copy and modify it in the user's AppData directory, %localappdata%. This data can be copied at first run of your application and enables per-user state for each user. If, on the other hand, you want machine-wide state that individual users can modify, you would want to put this in the %allusersprofile%, as in the high scores example mentioned earlier.

Another aspect of installation that causes some grief in the UAC world is updating. If you are going to update binaries in the Program Files directory, the process overwriting the binaries must have administrator or system privileges. One benefit of using Windows Installer is the ability to patch using an MSI patch (MSP) file. Since Windows Installer 3.1, which is a required update for Windows XP, an MSP file will be applied to the system if it is signed by a certificate that was included in the original MSI file. For more information, see the documentation on User Account Control Patching on MSDN® (at windowssdk.msdn.microsoft.com/ms710366.aspx).

Other than using MSI files, application developers can either integrate a binary that is specified to run with administrator privileges-which will prohibit updating in an enterprise with standard users-or include a service that will execute any updates required. Shipping a service is strongly discouraged because any security vulnerability in the service could make the ISV an attack vector for malware. MSP files are your best bet.

Marking the Privileges for an MSI

Before Windows Vista, the ALLUSERS property was used to mark whether an MSI file would install an application's shortcuts to the user location or for all users on the machine. These shortcuts included the DesktopFolder, ProgramMenuFolder, StartMenuFolder, and StartupFolder. Because there was no analogous per-user Program Files directory, the application binaries generally were still written to the Program Files directory.

Unfortunately, for app compatibility reasons Windows Installer could not determine whether the user should be prompted for credentials based solely on the ALLUSERS property. Instead, an additional bit was allocated in the MSI file to determine whether to prompt the user. This is bit 3 in the Word Count Summary property. If this bit is set to 1, the package is assumed to be a per-user MSI and the user will not be prompted for an Administrator token.

To specify that a package can only be installed by an administrator into the Public profile, set ALLUSERS="1" or ALLUSERS="2" and set bit 3 of the Word Count Summary property to 0. To specify that a package is a per-user installation that can be installed by a standard user, set ALLUSERS="" or don't define the property, and set bit 3 of the Word Count Summary property to 1.

Running Your App with Administrator Privileges

There are times that you may need administrator privileges for an application. You may write code that directly interacts with a piece of hardware or an app that sets machine-wide settings in HKLM. When possible, you should design your apps to limit the need for admin privileges to narrow sections of code, or communicate to an application started with full administrator privileges.

Aside from MSI-based elevation, there are two ways to create processes with a user's full administrator token. The Application Information Service (AIS) will check during process creation and during the creation of a COM object using the CoCreateAsAdmin moniker to see if the binaries require administrator privilege. It is important to note that the elevation occurs at the time of process creation. The process token never has privileges or group membership added during run time, only when it is created.

One word of caution: the UAC elevation prompt is only presented to the user when ShellExecute is called to create the process. ERROR_ELEVATION_REQUIRED is returned by any call to the CreateProcess family that requires elevation.

Marking Required Privileges Using an Application Manifest

When a new process is created, the AIS will inspect the binary to determine whether it requires elevation. The first thing that gets checked is the application manifest that is embedded into the application's resources. This takes precedence over any other type of application marking including an application compatibility marking or UAC's Installer Detection, which is described later. The manifest defines a run level that tells Windows the privileges needed to run the binary. The three choices for run level are: asInvoker, highestAvailable, and requireAdministrator.

When AIS finds a binary that is marked with the "asInvoker" run level, it takes no action and the process inherits the process token of the parent process that created it. The "requireAdministrator" run level is pretty straightforward as well and defines that the process must be created by a user token that is a member of the administrator group. If the user who attempted to create this process is not an administrator, he will be presented with the Credential dialog to input his credentials.

The highestAvailable run level is a little more complicated. It denotes that if a user has a linked token, then the application should run with the higher privileged token. This is generally used for applications that have a UI designed for the Users and Administrators groups and it ensures that the application gets the user's full privileges. It is important to note that users in the Backup Operators and Network Operators group will have a linked token and will be prompted for their credentials; the credentials dialog will have their user tile and those of the members of the administrators group. Figure 6 is an example of an application manifest.

Figure 6 AdminApp.exe.manifest for AdminAppl.exe (Close Figure 6)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <assemblyIdentity version="1.0.0.0"
     processorArchitecture="X86" name="AdminApp" type="win32"/>
  <description>Description of your application</description>
  <!-- Identify the application security requirements. -->
  <ms_asmv2:trustInfo xmlns:ms_asmv2="urn:schemas-microsoft-com:asm.v2">
    <ms_asmv2:security>
      <ms_asmv2:requestedPrivileges>
        <ms_asmv2:requestedExecutionLevel
          level="requireAdministrator"
          uiAccess="false"/>
        </ms_asmv2:requestedPrivileges>
       </ms_asmv2:security>
  </ms_asmv2:trustInfo>
</assembly>

The application manifest marking is only relevant to EXEs, not DLLs. This is because UAC does not inspect DLLs during the creation of the process. To embed this manifest into your native application, copy the manifest file into the same directory as your source and add the manifest into the resource files to specify that the manifest is to be embedded. The following lines in the .rc file would embed the manifest above if it were saved as AdminApp.exe.manifest:

#define MANIFEST_RESOURCE_ID 1
MANIFEST_RESOURCE_ID RT_MANIFEST "AdminApp.exe.manifest"

Marking a COM Object

When a COM object is being created, you can use the CoCreateAsAdmin moniker to get a COM object running with elevated privileges. This is very useful for creating objects that run elevated and are exposed in your application's UI.

There are a couple additions that must be made to a COM object so it can run with the fully privileged token. A display name to show in the UAC elevation dialog must be specified as well as a marking showing that the object can be created with elevated privileges. The following registry keys must be added to the registration of your object.

HKEY_LOCAL_MACHINE\Software\Classes\CLSID\
 {CLSID}\LocalizedString = <displayname>
HKEY_LOCAL_MACHINE\Software\Classes\CLSID\
 {CLSID}\Elevation\Enabled = 1

If the COM object is not registered in HKLM and both of these pieces of information are not specified, then trying to create the elevated COM object will fail. If it succeeds, the user will be prompted for elevation, assuming the call is coming from a process with a filtered token, and a new process will be created to host the COM object. For more information about creating elevated COM objects, see the MSDN documentation for the CoCreateAsAdmin moniker at msdn2.microsoft.com/ms679687.aspx.

Alerting Users to Elevated Privileges

Another challenge for UAC is communicating to users that may not have any notion of the difference between an application that is running with reduced privileges and one that is running as an administrator. The key to understanding this difference is the standardized workflow when the prompt for elevation occurs. The elevation sequence always begins with the user clicking on something that has a shield image in its icon. If starting an application is going to cause an elevation, you should see a shield overlaying the icon. Anywhere in your application that causes elevation should be marked with a shield.

Many of the common controls have been updated to allow you to add shields to them. This is extremely helpful if you want to expose settings where clicking on a button or link results in the creation of a new process that will require elevation. The code to add the shield glyph is very simple. For example, the standard button control (PUSHBUTTON, DEFPUSHBUTTON) has been enhanced to let you add an icon along with the displayed text, without requiring the BS_ICON or BS_BITMAP styles to be set.

Figure 7 The Shield on a Button
Figure 7 The Shield on a Button

To show the shield icon (which you can see in Figure 7), you should call the following macro (defined in commctrl.h), which sends a SETSHIELD message to the button:

Button_SetElevationRequiredState(hwndButton, true);

The same style of showing the shield can be used for Aero™ Wizard buttons, Syslink and Hyperlink controls, Commandlink and Command buttons, and more.

Communicating between Processes in Different Security Contexts

Windows Vista also includes an authentication technology called Windows Integrity Mechanism (WIM). WIM defines different security levels that are used as labels on objects for further authorization: Low, Medium, High, and System. Because of UAC there may be processes in the same desktop session that have full administrator privileges and others that are running with a standard user token. UAC uses WIM to block Windows messages from being sent between processes of different privilege levels.

The token used to create the desktop will be running at the medium integrity level-see the sample filtered administrator token. This way everything by default is running with an integrity level of medium. When a user is prompted for elevation and gives his consent, the resulting process is created with the fully privileged token and has an integrity level of high. In the past, there have been attacks, called "shatter attacks," where a lower-privileged UI sends messages to a higher-privileged UI to drive the higher privileged process. In order to avoid this, most Windows messages are no longer sent from lower privileged processes to higher privileged processes, and this is based on the integrity level.

There are still ways to communicate between processes of different integrity levels. You can use shared memory or a remote procedure call (RPC). Be sure that when you create your RPC entry point or IPC mechanism, you specify that it can be connected to by a process that is running as a standard user. You will need to specify this in the object's security descriptor.

Low Rights Internet Explorer

One other side note about WIM and UAC is that it enables LowRights Internet Explorer®, a setting that takes advantage of the low integrity level to protect users while they browse the Web. When running at Low, the process can interact with a very narrow set of directories and files on the system. If you have an ActiveX® control implemented as part of your application, which is hosted by Internet Explorer, you will want to test it to make sure it runs properly. For more information, see the whitepaper at msdn.microsoft.com/library/en-us/ietechcol/dnwebgen/protectedmode.asp.

Securing a Windows Service in Windows Vista

There are many new facilities for locking down services in Windows Vista. Developers can specify the privilege level that a service needs, and a service can have a Security Identifier (SID) applied to it, which can be used to secure resources so that only the service can write to them. Services can also be prevented from accessing the network. For more information, see the documentation on Windows services available at microsoft.com/whdc/system/vista/Vista_Services.mspx.

You can still use sc.exe to define these new security parameters in Windows Vista (see Figure 8). Privileges is a string that contains a list of privileges that are separated by a forward slash (/). For example, to specify backup and restore privileges, set Privileges to SeBackupPrivilege/SeRestorePrivilege.

Figure 8 Sc.exe Commands (Close Figure 8)

Command | Syntax | Purpose
Privs | sc <server> privs [service name} [privileges] | Sets the required privileges for a service.
Qprivs | sc <server> qprivs [service name] buffersize | Queries for the required privileges of a service.
Sidtype | sc <server> sidtype [service name] [type] | Applies service SID to process token.
Qsidtype | sc <server> qsidtype [service name] | Retrieves the setting for service's SID.

Sc.exe can also be used to get a specified service's SID. The following command returns the SID for a service that is named service name.

Sc showsid [service name]

To programmatically change this information, use the new API called ChangeServiceConfig2. To change the service privileges specified, call ChangeServiceConfig2 with the following parameter values. (The change in privileges takes effect the next time the service is started.)

The dwInfo parameter should be SERVICE_CONFIG_REQUIRED_PRIVILEGES_INFO and the lpInfo buffer should point to a SERVICE_REQUIRED_PRIVILEGES_INFO structure. This structure contains a single multistring that lists the required privileges.

To enforce the service SID for the service, set the flag programmatically, calling ChangeServiceConfig2 with the following parameter values. (The change takes effect the next time the system is booted.) The dwInfo parameter should be set to SERVICE_CONFIG_SERVICE_SID_INFO, and the lpInfo buffer should be set to point to a SERVICE_SID_INFO structure. This structure contains a single DWORD member that contains the SID type.

Two related public functions are also useful to service owners: LookupAccountName and LookupAccountSID. LookupAccountName takes a SID and returns the associated service name while LookupAccountSID takes a service name and returns the associated SID. To query the values of these settings use the QueryServiceConfig2 API.

Another way to protect your service is to limit its access to the network. To configure firewall restrictions for the service, use the INetFwServiceRestriction interface.

One last note-be wary of exposing RPC interfaces that can be connected to from lower-privileged processes. These interfaces should be carefully designed and tested so that the tasks that can be performed on the user's behalf do not allow an elevation of privilege by the user.

Common Application Compatibility Issues

One of the biggest challenges our team encountered while producing UAC was the incredible impact it had on our application ecosystem. Many applications were not designed to be run by a member of the Users group. Microsoft has invested many hours to understand the common application compatibility issues and has added technologies to Windows Vista to address some of the problems. In the final section of this article, I will describe some of the problems and briefly describe how to fix them.

The first application compatibility technology that is part of UAC is called Installer Detection. Because most installers write binaries to the Program Files directory, they overwhelmingly need administrator privileges. Installer Detection is designed to scan the name and the resources of the EXE to determine whether an application is an installer. For example, an executable would be marked as an installer if the executable name or description contained the strings "install" or "setup". So an application named setup.exe, without an application manifest, would trigger a UAC elevation if launched by a token without administrator privileges.

Another technology under the UAC umbrella designed to help fix file and registry issues at run time is called File and Registry Virtualization. Many applications write their log files or their configuration to the Program Files or Windows directory. These directories do not allow a standard user to write to these locations and hence these writes would immediately fail. In order to fix these issues, File Virtualization takes the files written to certain protected locations in the file system and redirects the writes to the user's profile-it will not perform virtualization if the file being written to is a binary. Registry virtualization does the same thing for the registry by redirecting writes to HKLM\Software to a per-user location in the registry. These technologies have raised the application compatibility for legacy applications immensely.

One important note is that these technologies are disabled if the binary has an application manifest or if the binaries are natively compiled for 64-bit. This is because these technologies are really just crutches for legacy applications that were not designed with the standard user in mind. Unfortunately, these technologies don't solve all of the problems. Next, I'll look at some other common issues.

Problems During Installation

One of the most frequent problems we see is mismarked CustomActions in an MSI file. Many CustomActions do not specify the no impersonate flag msidbCustomActionTypeNoImpersonate, causing the CustomAction to be run as Local System. Because the default token on Windows Vista is now similar to that of a member of the Users group, these MSI files fail because the CustomActions attempt to do something that requires administrator privileges. Usually this can be fixed by merely adding the msidbCustomActionTypeNoImpersonate attribute to the CustomActions.

Another typical problem we've encountered during testing concerns applications that are launched as part of an install. It is very common to start an application at the end of installation. Unfortunately, the application is often started under the wrong user context because the user provided elevated credentials to perform the installation and the application is created with the elevated user token. The recommended guidance for solving this problem is to create a bootstrapper EXE. This bootstrapper may unpack the binaries to a temp location and then start another process that initiates the installation (see Figure 9). When the process that started the installation completes, it returns control to the unpacker that starts the application. This way the application is running with the standard user token.

Figure 9 Using a Bootstrapper to Install and Execute an App
Figure 9 Using a Bootstrapper to Install and Execute an App(Click the image for a smaller view)
Figure 9 Using a Bootstrapper to Install and Execute an App
Figure 9 Using a Bootstrapper to Install and Execute an App (Click the image for a larger view)

The last installation issue we've often come up against is performing operations that require administrator privileges on the first run of the application. These include copying or updating binaries into Program Files, installing a driver, or any other operation that assumes the user is an administrator. The best way to get a handle on these issues is to test your application when installed and run as a standard user.

Many applications have some mechanism for updating the application binaries. Often these updaters fail because they are running with a lower-privileged token and attempt to overwrite the binaries in the Program Files directory. If your application updates its binaries, move the update functionality to a separate process and mark it as requiring administrator privilege. Another option is to employ MSP files to do the elevation based on the signature of the patch. For this, you would need to populate the MsiPatchCertificate table in your original MSI file.

Typical Runtime Application Issues

One of the most common mistakes that occur at run time is an unnecessary check to see if the user is an administrator. In many games, for example, the first thing checked is whether the user is a member of the administrator group. Often the games will report a failure to the user-but they actually don't ever call any APIs that require administrator privileges! Obviously, this should be avoided. However, for resiliency, if you write an application that requires administrator privileges and is manifested, add this check and exit if the user doesn't have administrator privileges.

Many applications will begin execution and immediately attempt to register a class to HKEY_CLASSES_ROOT (HKCR). Because the applications no longer have administrator privileges, it will be impossible for them to write to this location and the operation will fail. Registering these classes should be done during installation or redirected to the user's classes root: HKCU\Software\Classes.

Another potential issue to look out for is that standard users are prevented from creating objects in the Global namespace. For example, a standard user could not create a Named Pipe or Shared Memory within the namespace. Global is intended only for use by services or applications maintaining cross-session state. The Local namespace should be used instead. This is writable by a standard user.

Using MAX_ALLOWED when opening an object is another issue. In Windows XP, it is generally assumed that the users running the applications were administrators. Therefore, when developers used MAX_ALLOWED when opening files and registry keys, there was never a problem in this environment because they were always opening the files with full administrator rights. In Windows Vista, this has changed. Now, when developers write applications they need to be more careful when specifying the privileges a user will need to open an object.

Conclusion

This article is an overview of the new environment you'll be developing under in Windows Vista. The most fundamental change is that applications now run with a lower-privileged token by default and thus cannot impact the machine or other users. For more information about topics related to UAC in Windows Vista, see the UAC whitepapers mentioned earlier on MSDN and TechNet. And remember that UAC will make the lives of your users and systems administrators a lot easier.

Chris Corio is a Program Manager on the Windows Security team at Microsoft. His interests are primarily application security technologies and management technologies for securing Windows. You can reach Chris at chris.corio@microsoft.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Vista and Office: View Data Your Way With Our Managed Preview Handler Framework

Windows Vista and Office

View Data Your Way With Our Managed Preview Handler Framework

Stephen Toub
This article discusses:
  • Writing managed add-ins for Windows and Office
  • Implementing a managed preview handler framework
  • Developing custom preview handlers for any file extension
  • COM interop
This article uses the following technologies:
.NET Framework, Windows Vista, Outlook
Code download available at: PreviewHandlers2007_01.exe (444KB)
Contents
Hosting a Preview Handler
The PreviewHandler Framework
Implementing PreviewHandler
Using the COM Interfaces
Registering Handlers
Debugging
Conclusion

E
very version of Microsoft Windows and Office brings new methods and approaches for improving your ability to see, interact with, and understand data. Likewise, each version also presents new and improved extensibility points for plugging in custom functionality. In Windows Vista™ and in the 2007 Microsoft® Office system, these two areas of advancements have merged, giving you the ability to write custom preview handlers for Windows Vista and for Outlook® 2007.

Outlook 2003 provided a reading pane for e-mail that made it easy to view the contents of a message without having to open it. You simply selected the message in the mail folder's list view, and the message was rendered in a side or bottom window. Outlook 2007 extends this concept by allowing you to view message attachments in that same preview pane, without having to double-click an attachment to open it in the appropriate viewer.

Out of the box, Outlook includes preview handlers for Word documents, PowerPoint® presentations, Excel spreadsheets, font files, video and audio files, and a variety of other file types that are commonly sent as attachments. But that's only the beginning. Windows Vista supports a similar preview pane that is accessible from any folder in the shell. You can enable the preview pane by selecting Organize | Layout | Preview Pane from the folder's menu (see Figure 1).

Figure 1 Enabling Previews in Windows Vista Folders
Figure 1 Enabling Previews in Windows Vista Folders

Both Outlook and Windows Vista subscribe to the same underlying preview mechanism, and they allow developers to implement custom preview handlers for any file type, register them, and instantly gain preview capabilities for those file types in Outlook 2007 and Windows Vista.

In this article, I'll explain what is required to implement a preview handler and discuss how to do so using managed code (the Windows® SDK for Vista includes a sample preview handler written in native C++). The code download for this article includes a framework that makes it a snap to implement your own preview handlers, and it provides several sample previewers (including previewers for PDF, XML, ZIP, MSI, BIN, CSV, XPS, and XAML files).

Hosting a Preview Handler

Those of you who have attempted to implement managed add-ins for the Windows shell in the past might be a bit uneasy about this concept. After all, Microsoft strongly advises that shell add-ins not be implemented in managed code, and such add-ins are not considered a supported scenario. This is because add-ins are loaded in-process into the shell (explorer.exe), only one version of the common language runtime (CLR) can be loaded into a given process, and managed code built against one version of the runtime may not run in a process that is running an earlier version of the runtime. What happens, then, if you have two shell add-ins that are both written in managed code-one targeting the .NET Framework 1.1 and one targeting the .NET Framework 2.0? If the 2.0-targeted add-in is loaded first, you probably won't notice any problems; the 1.1 add-in should load and run successfully against the CLR 2.0. But if the 1.1-targeted add-in loads first, the .NET Framework 1.1 CLR will load into explorer.exe. Assemblies targeting the .NET Framework 2.0 cannot be loaded by the .NET Framework 1.1, and therefore the 2.0-targeted add-in will fail when loaded after the 1.1-targeted add-in.

The same versioning problems exist with Windows Vista, so Microsoft still advises that shell add-ins shouldn't be implemented in managed code-even for new extensibility points within the shell, such as thumbnail providers and property handlers (which are used out-of-process by the Windows Vista search indexer but in-process by the Windows Vista shell).

There is good news, however, regarding preview handlers: they are always loaded out-of-process, at least as far as the shell is concerned. Preview handlers are implemented as COM components, but they're not hosted by the Windows Vista shell. Instead, a preview handler is either hosted by the preview handler surrogate host (prevhost.exe) or implemented as a local COM server. For managed code, implementing the latter is well beyond the scope of this article (for an overview of what's involved, see the "Preview Handlers and Windows XP" sidebar by Ryan Gregg). Besides, using prevhost.exe is the preferred, Microsoft-recommended service model.

The PreviewHandler Framework

To be a valid preview handler, several interfaces must be implemented (all of which are documented at windowssdk.msdn.microsoft.com/aa361576.aspx). This includes IPreviewHandler (shobjidl.h); IInitializeWithFile, IInitializeWithStream, or IInitializeWithItem (propsys.h); IObjectWithSite (ocidl.h); and IOleWindow (oleidl.h). There are also optional interfaces, such as IPreviewHandlerVisuals (shobjidl.h), that a preview handler can implement to provide extended support.

If you were to write a preview handler in native code, you could dive right in, given that all of these interfaces are already defined and ready for inclusion from the header files I just noted parenthetically. However, to write a preview handler in managed code, you must first write or obtain COM interop definitions for each of these interfaces. My definitions are shown in Figure 2.

Figure 2 COM Interfaces Interop (Close Figure 2)
[ComImport]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[Guid("8895b1c6-b41f-4c1c-a562-0d564250836f")]
interface IPreviewHandler
{
    void SetWindow(IntPtr hwnd, ref RECT rect);
    void SetRect(ref RECT rect);
    void DoPreview();
    void Unload();
    void SetFocus();
    void QueryFocus(out IntPtr phwnd);
    [PreserveSig]
    uint TranslateAccelerator(ref MSG pmsg);
}

[ComImport]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[Guid("8327b13c-b63f-4b24-9b8a-d010dcc3f599")]
interface IPreviewHandlerVisuals
{
    void SetBackgroundColor(COLORREF color);
    void SetFont(ref LOGFONT plf);
    void SetTextColor(COLORREF color);
}

[ComImport]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[Guid("b7d14566-0509-4cce-a71f-0a554233bd9b")]
interface IInitializeWithFile
{
    void Initialize([MarshalAs(UnmanagedType.LPWStr)] string pszFilePath,
    uint grfMode);
}

[ComImport]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[Guid("b824b49d-22ac-4161-ac8a-9916e8fa3f7f")]
interface IInitializeWithStream
{
    void Initialize(IStream pstream, uint grfMode);
}

[ComImport]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[Guid("fc4801a3-2ba9-11cf-a229-00aa003d7352")]
public interface IObjectWithSite
{
    void SetSite([In,MarshalAs(UnmanagedType.IUnknown)] object pUnkSite);
    void GetSite(ref Guid riid,
                 [MarshalAs(UnmanagedType.IUnknown)] out object ppvSite);
}

[ComImport]
[Guid("00000114-0000-0000-C000-000000000046")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IOleWindow
{
    void GetWindow(out IntPtr phwnd);
    void ContextSensitiveHelp(
             [MarshalAs(UnmanagedType.Bool)] bool fEnterMode);
}

In my managed framework for implementing preview handlers, most of the functionality for these interfaces is wrapped up into an abstract base class, called PreviewHandler:

public abstract class PreviewHandler :
    IPreviewHandler, IPreviewHandlerVisuals, IOleWindow, IObjectWithSite
{ ... }
This hides all of the gory implementation details (which, as you'll see, aren't really gory at all). Note, however, that neither IInitializeWithFile nor IInitializeWithStream are implemented by PreviewHandler. Those honors are bestowed upon two abstract classes that derive from PreviewHandler:
public abstract class StreamBasedPreviewHandler :
    PreviewHandler, IInitializeWithStream { ... }

public abstract class FileBasedPreviewHandler :
    PreviewHandler, IInitializeWithFile { ... }

To implement a custom preview handler, you derive a new class from StreamBasedPreviewHandler or FileBasedPreviewHandler and override one method. Here's the implementation from my XmlPreviewHandler class:

public sealed class XmlPreviewHandler : FileBasedPreviewHandler
{
    protected override PreviewHandlerControl
        CreatePreviewHandlerControl()
    {
        return new XmlPreviewHandlerControl();
    }
}

The CreatePreviewHandlerControl method returns an instance of a custom type that you write derived from either StreamBasedPreviewHandlerControl or FileBasedPreviewHandlerControl. Both derive from my PreviewHandlerControl abstract base class:

public abstract class FileBasedPreviewHandlerControl :
   PreviewHandlerControl { ... }

public abstract class StreamBasedPreviewHandlerControl :
   PreviewHandlerControl { ... }

public abstract class PreviewHandlerControl : Control
{
    public abstract void Load(FileInfo file);
    public abstract void Load(Stream stream);
    public virtual  void Unload() { ... }
}

As its name implies, the Load method is called when a file or stream should be loaded and previewed. Likewise, the Unload method is called when the current preview should be torn down. A custom PreviewHandlerControl is then responsible for deriving from the appropriate type (either FileBasedPreviewHandlerControl or StreamBasedPreviewHandlerControl), overriding the Load method, and creating whatever Windows Forms controls are needed to render the file or stream. For my XML preview handler, I simply create a WebBrowser control and load the XML document into it, giving Windows Vista shell users the same pretty-printing of XML that is provided by Internet Explorer®:

public class XmlPreviewHandlerControl : FileBasedPreviewHandlerControl
{
    public override void Load(FileInfo file)
    {
        WebBrowser browser = new WebBrowser();
        browser.Dock = DockStyle.Fill;
        browser.Navigate(file.FullName);
        Controls.Add(browser);
    }
}

Figure 4 ZIP Preview in Outlook 2007
Figure 4 ZIP Preview in Outlook 2007(Click the image for a smaller view)
Figure 4 ZIP Preview in Outlook 2007
Figure 4 ZIP Preview in Outlook 2007 (Click the image for a larger view)

The base PreviewHandlerControl's Unload implementation disposes and clears all controls from the Controls collection. If that functionality is appropriate for your control, there's no need to override it.

Aside from applying a few attributes to the derived PreviewHandler class, that's all there is to writing a custom preview handler. Figure 3 shows the complete implementation of a ZIP preview handler, the results of which are shown in Figure 4, running in Outlook 2007 (the download includes a richer ZIP preview handler that sports a tree view of the files and directories within the .zip, file icons, and double-click support for viewing the files contained in the archive). This class uses the Visual J#® 2005 ZIP library (included in the .NET Framework 2.0) to provide a list of all the files contained in the ZIP file. To demonstrate that these preview handlers do work in both Windows Vista and Outlook, Figure 5 shows the XmlPreviewHandler running in the Windows Vista shell.

Figure 3 ZIP Preview Handler (Close Figure 3)
[PreviewHandler("ZIP Preview Handler", ".zip",
                "{c0a64ec6-729b-442d-88ce-d76a9fc69e44}")]
[ProgId("MsdnMag.ZipPreviewHandler")]
[Guid("853f35e3-bd13-417b-b859-1df25be6c834")]
[ClassInterface(ClassInterfaceType.None)]
[ComVisible(true)]
public sealed class ZipPreviewHandler : FileBasedPreviewHandler
{
    protected override PreviewHandlerControl CreatePreviewHandlerControl()
    {
        return new ZipPreviewHandlerControl();
    }

    private sealed class ZipPreviewHandlerControl :
        FileBasedPreviewHandlerControl
    {
        public override void Load(FileInfo file)
        {
            ListView listView = new ListView();
            listView.Dock = DockStyle.Fill;
            listView.BorderStyle = BorderStyle.None;

            listView.FullRowSelect = true;
            listView.HeaderStyle = ColumnHeaderStyle.Nonclickable;
            listView.MultiSelect = false;
            listView.View = View.Details;

            listView.Columns.Add("File Name", -2);
            listView.Columns.Add("Size", -2);
            listView.Columns.Add("Comment", -2);

            ZipFile zip = new ZipFile(file.FullName);
            Enumeration entryEnum = zip.entries();
            while (entryEnum.hasMoreElements())
            {
                ZipEntry entry = (ZipEntry)entryEnum.nextElement();
                if (!entry.isDirectory())
                {
                    listView.Items.Add(new ListViewItem(new string[] {
                        entry.getName(), entry.getSize().ToString(),
                        entry.getComment() }));
                }
            }
            zip.close();

            Controls.Add(listView);
        }
    }
}
Figure 5 XML Preview in Windows Vista
Figure 5 XML Preview in Windows Vista(Click the image for a smaller view)
Figure 5 XML Preview in Windows Vista
Figure 5 XML Preview in Windows Vista (Click the image for a larger view)

Implementing PreviewHandler

Now that you know how to implement a managed preview handler using my framework, let's take a look at the guts of the PreviewHandler base class to better understand how preview handlers work.

At its core, PreviewHandler is simply a container for a Windows Forms control, which is stored in a private member variable, _previewControl. It is initialized to the control returned from the GetPreviewHandlerControl method implemented by the custom preview handler implementation (for example, XmlPreviewHandlerControl or ZipPreviewHandlerControl):

protected PreviewHandler()
{
    _previewControl = CreatePreviewHandlerControl();
    IntPtr forceCreation = _previewControl.Handle;
}

While this is a very small constructor, there are several subtle issues worth pointing out. The first is that I'm violating an important .NET Framework design guideline, which at a high level says that constructors should not call virtual methods (this will be caught by the FxCop rule ConstructorsShouldNotCallBaseClassVirtualMethods). In .NET, unlike in ISO C++, the target of a virtual call will be on the most derived type (the override) rather than on the base type currently being constructed (the virtual). The problem here is that at the time when the base type's constructor is executing, the derived type's constructor has not yet executed. This means the method override will be called on the derived instance before that instance's construction has been completed, hence the rule that constructors should not call virtual methods that haven't been sealed in the same class. Keep this in mind, as it means you shouldn't do anything in your override of CreatePreviewHandlerControl that relies on anything set up in the constructor (that construction code will not have run yet). Based on the framework I've created, however, where CreatePreviewHandlerControl should simply be instantiating and returning the correct control type, this rule shouldn't be difficult to follow.

Of course, this begs the question, if I knew about this guideline in the first place, why did I break it by calling CreatePreviewHandlerControl from the constructor? Because I need to create the Control on the VI thread. This leads to the second issue I want to address. At the end of the constructor shown previously, you'll notice that I've retrieved the value of _previewControl.Handle, but I'm not doing anything with the results. I've done this in order to invoke the get accessor of the Handle property on the control. In other words, I didn't want the result; I simply wanted the accessor to execute. Invoking the Control.Handle get accessor forces the creation of the underlying window for the control. This is important, because the thread that instantiates the preview handler component and calls its constructor is a single-threaded apartment (STA) thread, but the thread that calls into the interface members later on is a multithreaded apartment (MTA) thread. As you may know, Windows Forms controls are meant to run on STA threads and sometimes die a horrible death if an attempt is made to use them from MTA threads. So, the best time to create the Windows Forms preview window is in the preview handler's constructor. Later calls to the other interface methods that require interaction with the preview control will need to be marshaled back to that STA main thread. This is easily accomplished using the control's ISynchronizeInvoke interface. Note that this problem of two threads does not exist when implementing a native preview handler and is most likely an artifact of how the CLR handles COM interop.

(It's worth pointing out that although I am discussing implementations that rely on Windows Forms, there may be times when the easiest way to render a preview is to use an ActiveX control. For a discussion on this, see the "Using ActiveX Controls" sidebar.)

Using the COM Interfaces

Believe it or not, the constructor is the most difficult part of the whole PreviewHandler implementation. The rest of the implementation simply serves to route calls made on the COM interfaces to the Windows Form controls and to handle some basic bookkeeping. I'll start by discussing the easiest interfaces first.

IOleWindow IOleWindow allows a hosting application to get a handle to the window that participates in in-place activation-in other words, a handle to our control (and for preview handlers, ContextSensitiveHelp is never called). This makes the interface extremely easy to implement, as shown here:

void IOleWindow.GetWindow(out IntPtr phwnd)
{
    phwnd = _previewControl.Handle;
}

void IOleWindow.ContextSensitiveHelp(bool fEnterMode)
{
    throw new NotImplementedException();
}

IObjectWithSite IObjectWithSite is just as simple. The interface is used to provide an object with a pointer to the site associated with its container. The site provided to SetSite is actually the IPreviewHandlerFrame that contains the preview handler's window. As such, the SetSite method stores the provided IUnknown interface pointer into a private member (which GetSite can return on request) and casts this to an IPreviewHandlerFrame (which it also stores). Under the covers, this cast results in a QueryInterface call:

private object _unkSite;
private IPreviewHandlerFrame _frame;

void IObjectWithSite.SetSite(object pUnkSite)
{
    _unkSite = pUnkSite;
    _frame = _unkSite as IPreviewHandlerFrame;
}

void IObjectWithSite.GetSite(ref Guid riid, out object ppvSite)
{
    ppvSite = _unkSite;
}

IInitializeWithFile and IInitializeWithStream Moving on, IInitializeWithFile and IInitializeWithStream are implemented in FileBasedPreviewHandler and StreamBasedPreviewHandler, respectively. They are called to provide the preview handler with the full path or IStream to the file to be previewed. Implementing these methods takes only a few lines of code:

// in FileBasedPreviewHandler
private string _filePath;
void IInitializeWithFile.Initialize(string pszFilePath, uint grfMode) {
    _filePath = pszFilePath;
}

// in StreamBasedPreviewHandler
private IStream _stream;
void IInitializeWithStream.Initialize(IStream pstream, uint grfMode) {
    _stream = pstream;
}

The sole method on these interfaces, Initialize, is provided with a path to the file to be previewed or with an IStream representing the file, along with a file mode indicating how the file can be opened. The file path or stream is stored in a member variable so it can be accessed later. Preview handlers should be read-only, so I've ignored the second parameter. (In fact, all preview handlers should ignore the second parameter and open files read-only. And they should allow for subsequent deleting, reading, and writing of files). IInitializeWithFile and IInitializeWithStream are easy to implement, but they do deserve some further discussion.

If you read the MSDN® documentation on implementing preview handlers, you'll find that the documentation strongly advocates implementing IInitializeWithStream instead of IInitializeWithFile. The main advantage IInitializeWithStream has over IInitializeWithFile is that IInitializeWithStream allows a preview handler to preview data that isn't stored in an independent file, such as a text file stored in a ZIP file. (If the shell tries to preview data available only as a stream using a previewer that only implements IInitializeWithFile, the shell will save a copy of the stream to a local file and then preview that file. Clearly that's not as efficient as previewing the stream directly.) Whenever possible, follow this advice. Admittedly, however, this isn't always possible or practical. This is because many previewers are for file types most easily loaded and rendered using APIs that only support file paths or URLs. For example, the MsiOpenDatabase function I use in my sample MsiPreviewHandler to open an MSI file expects a file path to the target MSI file. It doesn't accept an IStream, which is the only data I would have if I implemented IInitializeWithStream instead of IInitializeWithFile.

I've chosen to support this model in my framework using the two PreviewHandler subclasses mentioned earlier: FileBasedPreviewHandler and StreamBasedPreviewHandler. Each implements only one of the interfaces, not both. If PreviewHandler itself implemented both, the shell would always use the IInitializeWithStream implementation, which it prefers for reasons having to do with security and isolation. (Outlook would actually favor the IInitializeWithFile implementation, for legacy reasons). In your derived class, you can choose which interface to use by deriving from the appropriate PreviewHandler subclass.

Another important consideration is that the data from the file or stream should not be loaded in Initialize. Rather, as I've done here, the path to the file or the reference to the stream should be stored, and only when the preview handler is actually asked to render the preview should it load the contents of the file. Additionally, it should not in any way lock the file for exclusive access while the preview is being displayed. A user should be able to look at the preview and open the target file for editing at the same time. Most importantly, a user should still be able to delete the file while it is being previewed.

IPreviewHandler This brings me to the core interface a preview handler must support, which is aptly named IPreviewHandler. (My implementation of this interface is shown in Figure 6.) IPreviewHandler exposes seven methods that must be implemented: SetWindow, SetRect, DoPreview, Unload, SetFocus, QueryFocus, and TranslateAccelerator.

Figure 6 IPreviewHandler Implementation (Close Figure 6)
private void InvokeOnPreviewThread(MethodInvoker d)
{
    _previewControl.Invoke(d);
}

private void UpdateWindowBounds()
{
    if (_showPreview)
    {
        InvokeOnPreviewThread(delegate()
        {
            NativeWin32.SetParent(_previewControl.Handle, _parentHwnd);
            _previewControl.Bounds = _windowBounds;
            _previewControl.Visible = true;
        });
    }
}
void IPreviewHandler.SetWindow(IntPtr hwnd, ref RECT rect)
{
    _parentHwnd = hwnd;
    _windowBounds = rect.ToRectangle();
    UpdateWindowBounds();
}

void IPreviewHandler.SetRect(ref RECT rect)
{
    _windowBounds = rect.ToRectangle();
    UpdateWindowBounds();
}

protected abstract void Load(PreviewHandlerControl c);

void IPreviewHandler.DoPreview()
{
    _showPreview = true;
    InvokeOnPreviewThread(delegate()
    {
        Load(_previewControl);
        UpdateWindowBounds();
    });
}

void IPreviewHandler.Unload()
{
    _showPreview = false;
    InvokeOnPreviewThread(delegate()
    {
        _previewControl.Visible = false;
        _previewControl.Unload();
    });
}

void IPreviewHandler.SetFocus()
{
    InvokeOnPreviewThread(delegate() { _previewControl.Focus(); });
}

[DllImport("user32.dll")]
private static extern IntPtr GetFocus();

void IPreviewHandler.QueryFocus(out IntPtr phwnd)
{
    IntPtr result = IntPtr.Zero;
    InvokeOnPreviewThread(delegate() { result = GetFocus(); });
    phwnd = result;
    if (phwnd == IntPtr.Zero) throw new Win32Exception();
}

uint IPreviewHandler.TranslateAccelerator(ref MSG pmsg)
{
    if (_frame != null) return _frame.TranslateAccelerator(ref pmsg);
    return 1; // S_FALSE
}

When a file is about to be previewed, the host passes information about the file or stream to the preview handler using one of the initialization interfaces discussed previously. The handle for the view window that will contain the preview window is then passed to the preview handler using the SetWindow method-this allows the preview handler to set that view window as the parent of the preview window. The host may then set the size of the preview window appropriately, using the SetRect method. At some point after that, the host calls the DoPreview method to cause the preview to be displayed. While the preview is displayed, the host may call SetRect again, whenever the view window is resized. Finally, when the preview is closed, the host tells the preview handler to tear down the preview by calling the Unload method. This refers to the unloading of any resources loaded from the item being previewed; it doesn't mean the preview handler itself is being unloaded. The same instance of the preview handler can be reused for multiple previews.

To support my implementation, I am using a couple of helper functions. First, my InvokeOnPreviewThread helper accepts a MethodInvoker delegate (a void and parameterless delegate defined in the System.Windows.Forms namespace). This is executed on the main UI thread with the help of the preview Control's ISynchronizeInvoke.Invoke method (remember that the control was created in the preview handler's constructor so as to be instantiated from an STA thread). Second, I use my UpdateWindowBounds helper to set the parent window for the preview control, move it to the correct position, and display it. This functionality is useful in a helper method, as several IPreviewHandler interface methods modify this information and require it to affect the preview.Using ActiveX Controls

The preview handlers shown in this article were implemented with Windows Forms controls, such as ListView. But for some file formats, the easiest way to render a preview is to use an ActiveX® control with which the file format is associated. Through COM interop, Windows Forms allows ActiveX controls to be hosted like any other controls. The easiest way to get a managed wrapper for an ActiveX control is to use the Windows Forms ActiveX Control Importer, aximp.exe (see msdn.microsoft.com/library/en-us/cptools/html/cpgrfWindowsFormsActiveXControlImporterAximpexe.asp for more information). This will generate interop assemblies that you would then need to manually add references to in your project. If you instead add the ActiveX control to the Visual Studio Toolbox and drag-and-drop an instance of the control onto one of your design surfaces, Visual Studio will both generate the assemblies and automatically add the references for you.

An alternate approach is to manually create a wrapper for the control. For simple controls or for controls where you don't need to access much functionality, this technique may be more appealing. This is the approach I used to create a preview handler for the Adobe PDF format.

I receive a lot of PDF attachments in e-mail messages, and it would be very handy to be able to view the PDFs right within the attachment preview pane. While Office 2007 can save documents to the PDF file format (see microsoft.com/downloads/details.aspx?FamilyID=4d951911-3e7e-4ae6-b059-a2e79ed87041 for the download), it doesn't open this format for reading. So I created my PdfPreviewHandler, which allows me to preview PDF files by hosting the ActiveX control included with Adobe Reader. In order to host the control, I wrap it with a class derived from AxHost:

public class PdfAxHost : AxHost {
    public PdfAxHost() :
        base("ca8a9780-280d-11cf-a24d-444553540000") {}

    object _ocx;
    protected override void AttachInterfaces(){_ocx = base.GetOcx(); }

    public void LoadFile(string fileName) {
        _ocx.GetType().InvokeMember(
            "LoadFile", BindingFlags.InvokeMethod, null,
            _ocx, new object[] { fileName });
    }
}

Aximp.exe does this, but it also does a whole lot more (providing high-fidelity wrappers for all methods and events). I've done the bare minimum necessary to wrap the control and to expose the one method (LoadFile) I need to be able to invoke on it. LoadFile accepts as an argument the location of the file to be displayed and renders it in the control. I can then use this control like any other:

public override void Load(FileInfo file)
{
    PdfAxHost viewer = new PdfAxHost();
    Controls.Add(viewer);
    IntPtr forceCreation = viewer.Handle; // creates the OCX
    viewer.Dock = DockStyle.Fill;
    viewer.LoadFile(file.FullName);
}

SetWindow and SetRect are two such methods. The former provides the parent window handle and the new bounds for the preview window. The latter provides just the new bounds. SetFocus simply translates into a call to the preview control's Focus method, and QueryFocus returns the result of calling user32.dll's GetFocus function from the preview window's thread.

That leaves TranslateAccelerator, Unload, and DoPreview. My implementation of PreviewHandler currently doesn't provide much support for accelerator keys. As such, TranslateAccelerator simply delegates to the TranslateAccelerator method on the IPreviewFrameHandler stored in SetSite. If your preview handler displays multiple controls that can be tabbed through, you will want to augment TranslateAccelerator to provide more robust tab handling.

IPreviewHandlerFrame has another method on it, GetWindowContext, that allows the previewer to get a table that can be used to filter down the accelerators that should be forwarded to IPreviewHandlerFrame::TranslateAccelerator. This is meant to enable performance gains. However, even in native code, it doesn't provide huge performance improvements, and considering the complications involved in managing an accelerator table, doing so is probably not worthwhile. Preview handlers don't need to use the method-just forwarding everything to TranslateAccelerator is fine. In short, don't worry about GetWindowContext.

The Unload method hides the preview window and then delegates to the PreviewHandlerControl.Unload virtual method shown earlier in this article. And DoPreview, which is arguably the most important method on the interface, is no harder to implement. DoPreview is called to perform the actual rendering of the preview. My implementation calls to the abstract Load method, passing into it the PreviewHandlerControl. The derived implementation then calls either the FileBasedPreviewHandler's Load method with the path stored from IInitializeWithFile or the StreamBasedPreviewHandlerControl's Load method with the stream provided in IInitializeWithStream:

// in FileBasedPreviewHandler
protected override void Load(PreviewHandlerControl c)
{
    c.Load(new FileInfo(_filePath));
}

// in StreamBasedPreviewHandler
protected override void Load(PreviewHandlerControl c)
{
    c.Load(new ReadOnlyIStreamStream(_stream));
}
(Note that it encloses the COM IStream with a wrapper I derived from System.IO.Stream, allowing the derived class to consume the IStream in a read-only manner just as it would with any other .NET stream.) Once the data has been loaded, the DoPreview method calls UpdateWindowBounds to ensure that the preview is properly displayed.

Registering Handlers

Both the Windows Vista shell and Outlook 2007 look to the Windows registry to determine what preview handlers are available and to see what file types they are associated with.

As with any other COM class, a preview handler needs to be assigned a class ID. Referring back to Figure 2, this is the job of the GuidAttribute applied to the class:

    [Guid("853f35e3-bd13-417b-b859-1df25be6c834")]
You'll notice that each of my preview handlers has a different GUID. Any custom preview handlers you create must also be assigned unique IDs. This ID is used as part of the preview handler's registration. The uuidgen.exe and guidgen.exe utilities included with Windows provide quick ways to generate GUIDs.

Figure 7 List of Registered Preview Handlers
Figure 7 List of Registered Preview Handlers(Click the image for a smaller view)
Figure 7 List of Registered Preview Handlers
Figure 7 List of Registered Preview Handlers (Click the image for a larger view)

First, add a REG_SZ value to the PreviewHandlers key at HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\PreviewHandlers, where the name of the value is the GUID assigned to the preview handler. While it's not actually used, the value of this entry should be the display name of the preview handler. All of the built-in preview handlers do this, making it easy to see what preview handlers are registered by simply scanning this list in the registry (see Figure 7). Later on, after everything has been registered correctly, you can also see a list of all of the registered preview handlers within the Outlook Trust Center (see Figure 8).

Figure 8 List of Preview Handlers in the Outlook Trust Center
Figure 8 List of Preview Handlers in the Outlook Trust Center(Click the image for a smaller view)
Figure 8 List of Preview Handlers in the Outlook Trust Center
Figure 8 List of Preview Handlers in the Outlook Trust Center (Click the image for a larger view)

Next, you need to create a special registry key under the shellex key for the target extension. This special key is named with a specific identifier, "{8895b1c6-b41f-4c1c-a562-0d564250836f}", which tells the system that the data it contains represents a preview handler. The default value for that key is set to the GUID for the preview handler used for that extension:

using (RegistryKey extensionKey =
       Registry.ClassesRoot.CreateSubKey(extension))
using (RegistryKey shellexKey = extensionKey.CreateSubKey("shellex"))
using (RegistryKey previewKey = shellexKey.CreateSubKey(
       "{8895b1c6-b41f-4c1c-a562-0d564250836f}"))
{
    previewKey.SetValue(null, previewerGuid, RegistryValueKind.String);
}

If a specific preview handler can be used with multiple file extensions, you can simply repeat this step for as many extensions as appropriate. My framework looks for a PreviewHandlerAttribute on the preview handler. This attribute specifies the display name of the handler, the extensions the handler supports, and an AppID (which I'll discuss in a moment). I allow the supported extensions to be a simple string, as is the case with the ZipPreviewHandler:

[PreviewHandler("ZIP Preview Handler", ".zip",
                "{c0a64ec6-729b-442d-88ce-d76a9fc69e44}")]

I also allow it to be a semicolon-delimited list of extensions, as is the case with my BinPreviewHandler:

[PreviewHandler("Binary Preview Handler", ".bin;.dat",
                "{e92d3c10-89c8-4543-91b9-7a74305e9df4}")]

In such a scenario, my registration function (which you can see in the download) loops through each of the listed extensions and registers the handler for each of them. (Note that there are some preview handlers in Windows Vista that can handle more file types than they're registered to support. For info on previewing more file types, see the "If You've Got It, Flaunt It" sidebar.)

If You've Got It, Flaunt It

There are a handful of preview handlers built into Windows Vista. However, some of them can actually handle more file types than they're registered to support by default. Take, for instance, the Microsoft Windows TXT Preview Handler. As you can guess, this preview handler renders .txt files. But nothing about this preview handler restricts it to working with just .txt. As a developer, I frequently receive C#, Visual Basic®, and C++ code files as e-mail attachments. And I'd love to be able to preview these attachments within Outlook, rather than having to open Visual Studio® as the default viewer. Well, I can. All I have to do is register .cs, .vb, .cpp, and .h files as extensions to be previewed with the Microsoft Windows TXT Preview Handler. The following .reg file does just that:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\.cs\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]
@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

[HKEY_CLASSES_ROOT\.vb\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]
@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

[HKEY_CLASSES_ROOT\.cpp\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]
@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

[HKEY_CLASSES_ROOT\.h\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]
@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

If you want to be really slick, you can write a simple application or script that enumerates all of the file extensions under HKEY_CLASSES_ROOT. You can check to see if each has a Content Type value set, and if it does, whether that Content Type's value starts with "text/". If it does and if the extension doesn't already have a preview handler configured, you can set it to be previewed with the TXT handler—just as I did in the previous registry script. With only a few minutes work, you can dramatically expand the number of file types that can be previewed.

Of course, the TXT Preview Handler isn't the only handler capable of previewing a variety of file types. Some handlers you write might also exhibit this flexibility. Consider the XmlPreviewHandler described in this article. If you look in the code download, you'll see that the preview handler is actually named InternetExplorerPreviewHandler. I changed its name since it's not really specific to the XML file format—it's more specific to the control used to render the XML (the WebBrowser control). Since the preview handler uses the WebBrowser control to navigate to the file being previewed, this handler can be used with quite a variety of file types. Many of these file types are already covered by other preview handlers, but some aren't. For example, the new XML Paper Specification (XPS) file format delivered with Windows Vista doesn't have a built-in preview handler, but Internet Explorer is capable of rendering XPS documents. Therefore, I augmented the InternetExplorerPreviewHandler by adding .xps to the list of supported file extensions, and with that simple effort, I now have a preview handler for XPS.

In addition to registering on file types, you can register on Prog IDs. For example, the default value in HKEY_CLASSES_ROOT\.xml is "xmlfile", so my XML preview handler is actually registered under HKEY_CLASSES_ROOT\xmlfile\ShellEx\{8895b1c6-b41f-4c1c-a562-0d564250836f}:

[PreviewHandler("XML Preview Handler", "xmlfile",
                 "{88235ab2-bfce-4be8-9ed0-0408cd8da792}")]

That way, if you have multiple file extensions that are really the same type, you can use the same Prog ID for each and register the previewer only once. (For example, you could register on HKEY_CLASSES_ROOT\jpegfile instead of on both HKEY_CLASSES_ROOT\.jpeg and HKEY_CLASSES_ROOT\.jpg). In fact, the Windows shell team recommends that new file types should define new Prog IDs and register previewer handlers on those, rather than on actual extensions, even if there's currently a 1:1 mapping between the Prog ID and the extension.

The next registry change (for managed handlers) involves AppIDs. The reason relates back to my discussion about writing managed add-ins for the Windows Vista shell. As I mentioned, preview handlers are loaded out-of-process from the shell, by default in a surrogate host (prevhost.exe). For efficiency, the system wants as few instances of prevhost.exe running as possible, so multiple types of preview handlers are loaded into the same external process. This means that while the shell won't be attempting to load managed preview handlers targeting different versions of the CLR, prevhost.exe might. And this brings us back to the problem we initially had. The solution is to tell the system that instances of your preview handler must be loaded into a surrogate host dedicated to preview handlers of that same type. This guarantees you won't have multiple preview handlers targeting different versions of the CLR loaded into the same surrogate process. The COM class registration for the preview handler (which I'll get to momentarily) specifies an ID for configuration details about exactly what surrogate host to use, and that ID is referred to as an AppID. When a managed preview handler is registered, I create a new AppID registration under HKCR\AppID that will only be used by that managed preview handler:

using (RegistryKey appIdsKey = Registry.ClassesRoot.OpenSubKey(
       "AppID", true))
using (RegistryKey appIdKey = appIds.CreateSubKey(appID))
{
    appIdKey.SetValue("DllSurrogate",
                   @"%SystemRoot%\system32\prevhost.exe",
                   RegistryValueKind.ExpandString);
}

Please note that this AppID still points to prevhost.exe as the surrogate host.

The only thing left to do is to add the preview handler's COM component registration. Most of this is easily done by using the regasm.exe tool included with the .NET Framework SDK:

regasm /codebase ManagedPreviewHandler.dll

(You can do this from an installer as well, with a bit of code that uses the RegisterAssembly method of the RegistrationServices class in the System.Runtime.InteropServices namespace of mscorlib.)

As mentioned previously, however, I need to modify the component registration to point to the new AppID I've created. And I need to add to it a display name that will be used by hosts like Outlook 2007 to list registered handlers:

using (RegistryKey clsidKey = Registry.ClassesRoot.OpenSubKey("CLSID"))
using (RegistryKey idKey = clsidKey.OpenSubKey(previewerGuid, true))
{
    idKey.SetValue("AppID", appID, RegistryValueKind.String);
    idKey.SetValue("DisplayName", name, RegistryValueKind.String);
}

For a production preview handler, DisplayName should really be set to a REG_SZ value that points to a localized Win32 binary resource string ("@myhandler.dll,-101", for example). Unfortunately, the IDE for Visual C#® 2005 does not provide a built-in way to include a Win32 resource file in a managed assembly. To do that, you need to drop down to the command line, since csc.exe supports the /win32res switch that allows for the incorporation of Win32 resource files. For the purposes of this article and my registration code, I've resorted to a non-localized value.

All of this registration logic is wrapped up into a single method on PreviewHandler:

protected static void RegisterPreviewHandler(
    string name, string extensions, string previewerGuid, string appID)

There is a corresponding unregistration method, as well.

The regasm.exe tool supports ComRegisterFunctionAttribute from the System.Runtime.InteropServices namespace in mscorlib. You can apply this attribute to a static method in an assembly to be registered with regasm, and when types in that assembly are found to be ComVisible, their type info is passed to the method marked with the ComRegisterFunction attribute (after the type has had its COM class registered in the registry). This makes it very easy to write a method that handles all of the previously mentioned registration goo when the assembly is registered for COM interop (see Figure 9). I've included a counterpart unregistration function that is marked with the ComUnregisterFunctionAttribute.

Figure 9 COM Registration (Close Figure 9)
[ComRegisterFunction]
public static void Register(Type t)
{
    if (t != null && t.IsSubclassOf(typeof(PreviewHandler)))
    {
        object[] attrs = (object[])t.GetCustomAttributes(
            typeof(PreviewHandlerAttribute), true);
        if (attrs != null && attrs.Length == 1)
        {
            PreviewHandlerAttribute attr =
                attrs[0] as PreviewHandlerAttribute;
            RegisterPreviewHandler(attr.Name, attr.Extension,
                t.GUID.ToString("B"), attr.AppID);
        }
    }
}

With this in place, installing and registering my preview handlers requires just two lines at the command prompt:

gacutil -i MsdnMagPreviewHandlers.dll
regasm /codebase MsdnMagPreviewHandlers.dll
Similarly, unregistering the installed preview handlers requires just two lines at the command prompt:
regasm /unregister MsdnMagPreviewHandlers.dll
gacutil -u MsdnMagPreviewHandlers

If you write your own custom preview handlers that use this framework, and you write them in the same assembly as mine, you won't have to do anything else to get them to work. If you write them using my framework, but you write them in a separate assembly, you just need to add the following class to your assembly:

internal sealed class PreviewHandlerRegistration
{
    [ComRegisterFunction]
    internal static void Register(Type t) {
        PreviewHandler.Register(t);
    }
    [ComUnregisterFunction]
    internal static void Unregister(Type t) {
        PreviewHandler.Unregister(t);
    }
}

First install and register MsdnMagPreviewHandlers.dll, as I showed previously. Then install and register your assembly. When regasm sees the ComRegisterFunction in the PreviewHandlerRegistration class in your assembly, it will call the ComRegisterFunction for each of the preview handlers in your assembly. The function will delegate to all of the previously described functionality included in MsdnMagPreviewHandlers.dll.

Figure 10 XamlPreviewHandler in Outlook 2007
Figure 10 XamlPreviewHandler in Outlook 2007(Click the image for a smaller view)
Figure 10 XamlPreviewHandler in Outlook 2007
Figure 10 XamlPreviewHandler in Outlook 2007 (Click the image for a larger view)

To exemplify how to do this, I've included a preview handler for XAML files in the download for this article. It's implemented in its own assembly, which contains my XamlPreviewHandler class and the PreviewHandlerRegistration class. Figure 10 is a screenshot of the handler in action in Outlook 2007 (the XAML file being previewed, which is from the Windows Presentation Foundation SDK team blog, is available at blogs.msdn.com/wpfsdk/archive/2006/05/23/Animating_XAML_Clip_Art.aspx). Note that XamlPreviewHandler also derives from StreamBasedPreviewHandler rather than FileBasedPreviewHandler. The XamlReader class provided with the Windows Presentation Foundation supports loading from a Stream, and thus it makes sense to implement IInitializeWithStream rather than IInitializeWithFile:

public override void Load(Stream stream)
{
    Frame f = new Frame();

    XamlReader reader = new XamlReader();
    f.Content = reader.LoadAsync(stream);

    ElementHost xamlHost = new ElementHost();
    xamlHost.Child = f;
    xamlHost.Dock = DockStyle.Fill;
    Controls.Add(xamlHost);
}

Debugging

Since in-process preview handlers run under a surrogate hosting process (prevhost.exe by default), to debug a preview handler, you need to attach the debugger to the host process. There are a few ways to do this. The first is to wait for the host process to start and then use the debugger's ability to attach to an existing process. However, there may be multiple instances of prevhost.exe, especially if you follow my suggestion of creating a new AppID for each of your managed preview handlers. When there are multiple instances of prevhost.exe running, you need to know which instance to connect to. You can use a tool like Process Explorer (www.sysinternals.com/Utilities/ProcessExplorer.html) from Sysinternals (recently acquired by Microsoft) to examine the command-line arguments used when the appropriate instance of prevhost.exe was launched. As a surrogate host executable, prevhost.exe needs to be told at startup what COM component to host, and thus it will be passed the GUID for your preview handler on the command line:

prevhost.exe {8FD75842-96AE-4AC9-A029-B57F7EF961A8} -Embedding

You can look at the command-line arguments for each instance of prevhost.exe to find the one with the GUID that matches the preview handler you want to debug. You can then use the process ID to attach the debugger to the correct process.

If the process is already running, the preview handler will have already been constructed and some of its code executed. To debug its startup, you need a different solution. One approach is to place a call to System.Diagnostics.Debug.Break at the beginning of the constructor. This causes the debugger to launch and attach to the process at that moment, allowing you to debug your entire component.

But attaching a debugger isn't always the best solution. Sometimes you just want classic "printf-style" debugging, where you output a bunch of information that you can watch as the preview handler runs. In this case, it's best to use the System.Diagnostics.Trace class. By default, the TraceListenerCollection that Trace outputs to contains a TraceListener that writes the traced data to OutputDebugString, enabling any listening debuggers to display the output text. The Windows SDK includes Debug Monitor (DbMon.exe) which makes it easy to view these traces.Preview Handlers and Windows XP

By Ryan Gregg

With the framework provided in this article, writing an in-process preview handler is a straightforward exercise in managed code. However, creating a preview handler that works with Office Outlook 2007 on both Windows XP and Windows Vista is a more complicated matter. The proxy application (called prevhost.exe) that hosts in-process preview handlers on Windows Vista is not available on Windows XP. Preview handlers for Windows XP need to be written as local COM servers, without relying on another application to do the hosting.

Unfortunately, the .NET Framework 2.0 doesn't provide most of the glue necessary to register a .NET application as a local COM server. As a developer of a managed preview handler, you need to write code that looks and acts a lot like a native code implementation of a preview handler. Here's a very high-level view of what it is you need to write.

A typical local COM server is compiled as a Windows Application, instead of as a Class Library. When COM requests a new instance of the local server, it launches the local server application with a special set of flags to indicate that the application should start up in embedding mode. This causes the application to launch without a UI and to register class factories with COM. This same server likely has two additional logical components over the in-process preview handler implementation: a COM server implementation and a class factory.

The COM server implementation provides the running application framework for the executable launched by COM when the preview handler is called. Inside the Main method for the application, you need to parse the command-line parameters, initiate each class factory's registration with COM, monitor when it is safe to shutdown the process, and keep the message loop pumping so that the application hosting the preview handler won't hang. Most local COM servers also support command-line switches to register and unregister the server in the Windows registry. Since regasm.exe does not provide a method to register as a local server, you also need to write methods to create the proper registration keys for local COM server registration. These keys are well documented in the COM documentation.

The class factory component implements the IClassFactory interface provided by COM. Additionally, this component is responsible for registering class objects with COM and creating instances of class objects for COM callers. In particular, the class factory should make calls to CoRegisterClassObject and CoRevokeClassObject to indicate to the system that a class factory is available (or revoked) for the particular GUID used to identify the preview handler. When requested, the class factory creates a new instance of the preview handler class and marshals it back to the COM caller. These instances need to be tracked and garbage collection may need to be forced so the server application can determine when all external references have been released allowing it to shutdown properly.

Writing a managed preview handler that works with Outlook 2007 on both Windows XP and Windows Vista is a considerably involved effort. While this sidebar covers the required pieces, I haven't gotten into the important details. Anyone developing a local COM server needs to understand the way COM works, the difference between threading apartment styles, and how to work with Win32® APIs using managed code. Without this knowledge, frustration will likely follow.

While writing a local COM server preview handler in managed code may not be as simple as writing an in-process handler, this is a detail you shouldn't overlook if you want to ensure your preview handler will work for every Office 2007 user, regardless of the OS they are running on. For more details and working examples on how these pieces fit together, be sure to check out these helpful resources:

  • COM Fundamentals (windowssdk.msdn.microsoft.com/ms694505.aspx) is MSDN documentation that describes the Microsoft Component Object Model.
  • PInvoke.net (www.pinvoke.net) is an excellent site dedicated to helping users define Win32 API calls in managed code.
  • Building COM Servers in .NET (www.codeproject.com/useritems/BuildCOMServersInDotNet.asp) provides a useful example of a local COM server implementation.

Ryan Gregg works for Microsoft as a Program Manager on Office Outlook, primarily on the extensibility platform and custom forms. He blogs semi-regularly about Outlook extensibility at blogs.msdn.com/rgregg.

For testing purposes, I recommend that you add a trace statement to the beginning of each of the interface methods I discussed earlier. Alternatively, you can use Debugger.Break in concert with the new tracepoint feature in Visual Studio 2005. By doing this, you'll gain a much clearer understanding of how the shell and Outlook 2007 interact with your preview handler.

Conclusion

Preview handlers are an awesome addition to both the Windows Vista shell and Outlook 2007. Throw in the productivity advantages of managed code (using my existing framework, it took me only seven minutes to create the XAML preview handler and get it up and running) and you've got yourself a very powerful platform for displaying existing file types as well as your own custom file types. I believe preview handlers have the potential to make us more productive in our daily routines-and the more supported file types there are, the more productive we'll all be. So get out there and start coding a preview handler for your favorite extension.

Stephen Toub is the Technical Editor for MSDN Magazine.


SideShow Gadgets: Get Started Writing Gadgets For Windows SideShow Devices

SideShow Gadgets

Get Started Writing Gadgets For Windows SideShow Devices

Jeffrey Richter
This article discusses:
  • SideShow and its associated APIs
  • Developing, installing, and configuring gadgets
  • Device cache management
  • Gadget events and notifications
This article uses the following technologies:
Windows Vista
Code download available at: SideShow2007_01.exe (169KB)
Contents
What Are SideShow Devices?
Use and Configuration
SideShow Gadget and Device Architecture
Gadget Installation
Device Cache Management
Gadget Lifetime
Content
Capabilities, Events, and Notifications
Conclusion

T
oday's applications have lots of information to share with end users. However, screen real estate is limited and organizing all of the possible information is taxing for software developers as well as end users. Furthermore, people prioritize information in different ways and at different times. For example, when a meeting is imminent, I need to know the time and place right now. When I'm driving, I need to know how soon to make the next turn. In general, I might want to always be able to check newly received e-mail messages. Likewise, you can imagine scenarios where I might want information on traffic, flights, weather, financial transactions, and so on. Trying to get the right information to the user at the right time is a difficult problem.

The Microsoft gadgets initiative is committed to alleviating this problem. A gadget is an application (or part of an application) that can present users with the data they want when and where they want it. Currently, Microsoft offers three different kinds of gadgets: gadgets for Windows® Sidebar present a user's information on the user's desktop or in the Windows Vista™ Sidebar; gadgets for Windows Live™ present a user's information on any computer via a Web browser; and gadgets for Windows SideShow™ present a user's information on an auxiliary hardware device via its SideShow-enabled device driver. In this article, I will focus specifically on SideShow hardware devices and how to write a SideShow gadget to communicate with these devices using the managed API.

What Are SideShow Devices?

Windows Vista supports the ability to communicate with one or more auxiliary display devices. These auxiliary display devices can come in all kinds of shapes and sizes with varying capabilities. Many of these devices have not yet been manufactured, but should be available soon.

Some units will be powered when the computer is turned on while others can be powered whether the computer is on or off. Some units may even have their own separate rechargeable batteries. Some units will be in color and some will be grayscale or monochrome. Some will allow bitmaps to be displayed while some may only allow text. Some units will offer varying amounts of on-board memory for caching data while others may have no on-board memory at all. Some will offer buttons allowing navigation of the cached data. Some may offer touch-sensitive displays.

Some may offer a built-in wireless technology (such as Bluetooth) so that the computer can communicate with the device without it being physically connected to the computer. Some may offer a headphone jack allowing you to listen to music or video that has been downloaded into its memory.

Figure 1 shows a prototype notebook computer with a SideShow device mounted in its cover and a close-up picture of the SideShow device itself. This particular SideShow device has a 220×176 resolution display with each pixel offering 16 bits of color depth. The unit also offers Menu, Left, Right, Up, Down, and Enter buttons. This SideShow device is self-powered, meaning that it is powered using the computer's battery; however, the computer doesn't have to be turned on for the user to interact with the device. When on, this particular device consumes about 4 milliwatts of power, compared to the 75 or so watts of power consumed by the whole computer when it is turned on.

Figure 1 Prototype Notebook with a Mounted SideShow Device
Figure 1 Prototype Notebook with a Mounted SideShow Device(Click the image for a smaller view)
Figure 1 Prototype Notebook with a Mounted SideShow Device
Figure 1 Prototype Notebook with a Mounted SideShow Device (Click the image for a larger view)

Figure 1b (Click the image for a smaller view)

Figure 1b (Click the image for a larger view)

So, with the computer's lid closed and the computer's power off, the user can still read e-mail, examine his calendar, look at some digital photos, get mapping directions, see the current date/time, and so on. And of course, you can modify your own application to export its data to the SideShow device too. I'll show you how to do this later in this article.Figure 2 shows a prototype for a remote control device for controlling Windows Media® Center. The auxiliary display allows the user to select a song and have Media Center play it. What's cool about this is that the user's television or monitor doesn't have to be turned on in order to listen to a song. Also, while watching a movie, program information can be sent from the Media Center PC to the Windows SideShow device, allowing you to read it without affecting the playback of the movie on the monitor. Furthermore, you can use the remote control in order to view the TV guide data and schedule recordings.

Figure 2 (Click the image for a smaller view)

Figure 2 (Click the image for a larger view)

Other examples of SideShow devices include keyboards with LCD displays, digital picture frames, and even mobile phones. A single computer can have multiple SideShow devices connected to it simultaneously. For example, the prototype notebook computer shown in Figure 1 could have both a keyboard and a remote control talking to it. Each of these displays could show similar or different kinds of information.

Use and Configuration

End users configure their SideShow devices with the Windows SideShow Control Panel applet. When the user opens this applet, she is greeted by a window similar to the one in Figure 3.

Figure 3 SideShow Applet Controls Devices and Gadgets
Figure 3 SideShow Applet Controls Devices and Gadgets(Click the image for a smaller view)
Figure 3 SideShow Applet Controls Devices and Gadgets
Figure 3 SideShow Applet Controls Devices and Gadgets (Click the image for a larger view)

The leftmost panel shows the SideShow hardware devices currently installed, sorted by the date and time of their installation. In the main panel is a grid. The first column shows the installed applications that know how to communicate with SideShow devices; these are the SideShow gadgets. By default, Windows Vista ships with two gadgets: Inbox—Windows Mail and Windows Media Player. Figure 3 also shows another gadget that I've installed (Office Outlook® 2007 Calendar, which ships with Microsoft® Office Outlook 2007) and a gadget that I've written (called Jeff's SideShow Gadget). Each additional grid column represents a SideShow device attached to the computer. Figure 3 shows two SideShow devices: Windows SideShow Simulator—0 and Logitech G-15 Keyboard.

By selecting the checkboxes in the grid, the user indicates which SideShow devices a gadget can talk to. The user can also configure a specific SideShow device by selecting it at the top of the grid. For example, if I select the Windows SideShow Simulator—0 item, I get a window as shown in Figure 4. Here you can configure screen appearance and behavior (theme, language, font size, brightness, backlight, and so on), configure security and privacy settings (use a PIN to unlock the device, delete all data when the user logs off the computer), configure gadgets (turn gadgets on or off, change order of appearance), and uninstall the device.

Figure 4 Configuring a SideShow Device
Figure 4 Configuring a SideShow Device(Click the image for a smaller view)
Figure 4 Configuring a SideShow Device
Figure 4 Configuring a SideShow Device (Click the image for a larger view)

Many SideShow devices will have their own on-board memory that is used to cache gadget data. A gadget must be running in order to send data to a SideShow device, meaning that the computer must be powered on. So, Windows Vista allows the user to configure when and how often the computer should wake itself up and run the gadget. The user configures this by selecting the "Set my computer to wake automatically" option shown in Figure 3. When selecting this option, the window shown in Figure 5 appears. After being awakened, the computer will stay awake until the machine has been idle for the amount of time specified using the machine's Power Options Control Panel applet. Note that some computers do not support auto-wake and some applications can prevent a machine from going back to sleep.

Figure 5 Configuring How a Gadget Wakes Up The Device
Figure 5 Configuring How a Gadget Wakes Up The Device(Click the image for a smaller view)
Figure 5 Configuring How a Gadget Wakes Up The Device
Figure 5 Configuring How a Gadget Wakes Up The Device (Click the image for a larger view)

SideShow Gadget and Device Architecture

Figure 6 shows an architectural diagram of how SideShow gadget applications communicate with SideShow hardware devices. Each hardware device must come with a user-mode device driver that will have to be installed into Windows Vista. When Windows communicates with the device, the operating system creates a WUDFHost.exe process and loads the device driver into it; there is an instance of WUDFHost.exe for each device driver. There are many benefits to user-mode device drivers, including easier development, easier debugging, greater system stability, and improved security. For more info about the Windows User-Mode Device Driver Framework, see "User-Mode Driver Framework (UMDF)"

Figure 6 SideShow Gadget Communication Architecture
Figure 6 SideShow Gadget Communication Architecture(Click the image for a smaller view)
Figure 6 SideShow Gadget Communication Architecture
Figure 6 SideShow Gadget Communication Architecture (Click the image for a larger view)

A SideShow gadget application can be written in native code or in managed code. A native application will control the devices that the user has connected it to by using COM components and methods implemented in AuxiliaryDisplayApi.dll. (The original name for SideShow was Auxiliary Display.) This DLL ships with Windows Vista and is guaranteed to be available on every Windows Vista machine; however, it does not function on the Starter and Home Basic versions of Windows Vista. The main goal of the COM components is to abstract away from the developer the fact that different devices are or are not connected and that the devices connected support different capabilities (screen size, color, and so on).

A managed SideShow gadget will control the devices connected to it using types defined in the Microsoft.SideShow namespace. These types are implemented in the Microsoft.SideShow.dll assembly file. (I designed and implemented this for the Windows Vista team and it is downloadable via Microsoft Download Center.) Internally these types talk directly to the COM components in AuxiliaryDisplayApi.dll. Unfortunately, due to some timing issues with the release, the assembly is not guaranteed to exist on all installations of Windows Vista. If you write a managed SideShow gadget using this assembly, you will have to ensure that this DLL gets on to the end user's machine.

To help developers test their gadget code, the Windows Vista SDK ships with a SideShow Simulator application. This application runs on Windows Vista and emulates a SideShow hardware device. To get the simulator working, you must first run the following from a command prompt:

WindowsSideShowVirtualDevice.exe /regserver

This registers the COM components required by the simulator. You only need to register the COM components once on a machine. Then, whenever you want to test your gadget, you run the VirtualSideShow.exe app that also comes with the SDK.

The Microsoft.SideShow.dll assembly defines a number of types in the Microsoft.SideShow and the Microsoft.SideShow.SimpleContentFormat namespaces, making it easy to build SideShow gadgets or to integrate SideShow functionality into an already existing application. The most important and most-used classes are described in Figure 7.

Figure 7 Important SideShow-Related Classes (Close Figure 7)

Class | Description
GadgetRegistration | A static class to aid in setting up the gadget in the registry.
SideShowGadget | An abstract base class representing a connection between application code and one or more SideShow hardware devices that the user has connected. This class is derived from Component, allowing it to be dropped on a Windows Form.
ScfSideShowGadget | Derived from SideShowGadget, this class allows the application to send Simple Content Format (SCF) XML strings and images down to a SideShow device.
Scf | A static class offering static methods, making it easy to create SCF XML strings in source code.
DeviceCapabilities | An object of this class represents a hardware device's capabilities. This can be used by a gadget application to fine-tune the data it chooses to send to a device.

Gadget Installation

When developing a SideShow gadget, you must first create a GUID for it; this becomes the gadget's ID and Windows Vista uses this ID to interact with the gadget. When you install a gadget, your setup program should create a new registry subkey based on the GUID you've created for your gadget:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\SideShow\Gadgets\{<GUID>}

If you want to install the gadget for all users on the machine, use HKEY_LOCAL_MACHINE instead of HKEY_CURRENT_USER. Under this registry subkey you then need to add a few registry values, as described in Figure 8.

Figure 8 Registry Values for Gadget Installation (Close Figure 8)

Registry Value | Description
Icon | The path of the file (and optional Win32® icon resource ID number) that refers to an icon that should appear in Control Panel and on the device itself. If not specified, a default icon is shown. It is recommended that you supply 16x16, 32x32, and 48x48 pixel images using 32-bit color (with alpha channel).
FriendlyName | A string, or a pathname and Win32 string resource ID (for a localized string), that should appear as the name of the gadget in Control Panel. If you don't provide this, no string will be shown to the user.
StartCommand | The path of the executable file containing the gadget's code. Windows Vista uses this to invoke the application automatically when a user logs on, during fast user switches, when the computer resumes from sleep, or when the user enables a gadget for a device. (There is very little intelligence here; Windows Vista invokes gadget applications fairly frequently,so gadget applications should be kept small. Furthermore, a gadget application should terminate itself if another instance of it is already running or when no devices are connected to it.)
Endpoints | The list of supported endpoints, which can be either iCal format or SCF. iCal (see www.ietf.org/rfc/rfc2445.txt) is a text representation designed for publishing calendar information. The Office Outlook 2007 Calendar gadget uses this endpoint to send iCal data to a SideShow device, allowing the device to render the user's calendar data. Few gadgets and devices support the iCal endpoint. In contrast, SCF is a series of XML content pages that reference each other. Since most applications will use the SCF format, it is the focus of this article.
CacheAlgorithm | This value gives a hint to the device as to how the gadget would like its content managed (devices tend to have very limited cache memory and many gadgets could be connected to a single device). The gadget can tell the device to keep newest items (0, the default), keep oldest items (1), keep frequently accessed items (2), or keep recently accessed items (4).
OnlineOnly | If the value is 1, the user can only work with the gadget on the device if Windows is running and if the gadget application is also running.
PropertyPage | A gadget can offer a configuration settings dialog box to its users that can be launched via the Windows Control Panel. To do this, the gadget must define a COM component that implements the IPropertyPage interface. Then, during setup, the gadget would add this registry value indicating the GUID of the COM component. If specified, Control Panel will underline the gadget application, allowing the user to click it to invoke the gadget's configuration dialog box.

Your gadget's setup program should create its own registry subkey and set any of the registry values it desires. Typically, an installation program will place these registry additions in a .reg file and invoke this file upon application setup. When setup is near completion, you should execute the following command, which will cause Windows Vista to display a tooltip alerting the user that your gadget application has been installed:

SchTasks.exe /run /tn Microsoft\Windows\SideShow\GadgetManager

The GadgetManager task will not show the tooltip if no devices are installed, if it detects that no new gadgets have been installed since it ran last, or if the user turned it off by unchecking the "Notify me when new gadgets are installed" checkbox (see Figure 3).

Once the gadget is installed, users can open Control Panel to configure the gadget and connect it to whatever hardware devices they desire. For security reasons, however, by default a newly installed gadget is not allowed to communicate with any devices.

When a gadget is running, it usually sends data to its connected devices periodically. This way, when the user turns off the computer, the devices have cached some data, making it available whenever the user wants it without having to power up the computer. While a gadget is running, it will receive notifications from Windows indicating when the user has added or removed a device from the gadget via the control panel.

A gadget can also be marked as online-only, meaning that the user can only interact with the gadget on the device if both the computer and the gadget on Windows are running. If the computer is sleeping or in hibernation when the user enters this gadget on the device, the device asks the user whether the computer should be turned on. If the user responds yes, the computer powers up. When the computer is running and the user is logged in, the device enters the gadget. If the computer is in full shutdown, the device tells the user to manually power up the computer and log in, which allows the gadget to be entered on the device.

Device Cache Management

Since devices will have limited amounts of cache memory and users can have several gadgets connected to a single device, devices need to manage memory efficiently. Different devices handle this in various ways, but I'll explain how it works for devices using the Microsoft firmware.

The current Microsoft firmware divides the memory on the device equally among all the gadgets connected to the device. So if three gadgets are connected to a device, then each gadget is guaranteed to get a third of the memory. Individual gadgets are allowed to use more than their portion if other gadgets are using less than their portion. If a gadget that is under its minimum wants more memory, it can take that memory from any gadget that is over its minimum.

Similarly, if a gadget that is over its minimum wants more memory, it can take memory away from the lowest-priority gadget that is over its minimum. In Figure 3, you'll notice that there is an option allowing the user to change the order in which gadgets appear. Gadgets that the user puts at the top are high priority and are likely to get more memory; gadgets towards the bottom are lower priority and have memory taken away from them more quickly. When a gadget has memory taken away from it, the device uses the cache policy set by the gadget via its CacheAlgorithm registry value.

For gadget developers, the important thing to realize is that there is no guarantee that any content pushed down to the device will actually remain on it. If the user wants to examine content that has been purged, the device must notify the gadget on Windows so that it can push down the discarded content again. If the gadget does not send the missing content within an acceptable timeframe (which varies by device—on the simulator it is three seconds), then the device will notify the user that the content is missing. Unlike the native SideShow API, the managed SideShow library caches all content on Windows and will automatically send it back to the device as long as the gadget process stays up and running on Windows Vista.

Gadget Lifetime

Windows must be running a gadget in order for the gadget to download information to a SideShow device. There are basically two kinds of gadgets: UI gadgets and non-UI gadgets. A UI gadget is an application the user runs and interacts with. For example, imagine a Windows Forms application that the user invokes via Windows Explorer. As the user manipulates the application, it can download data to the device. If the user terminates the application, the information remains on the device and can be accessed when the computer is turned off. The lifetime of this application is determined completely by the user. The Microsoft Office Outlook 2007 gadget is an example of a UI gadget.

A non-UI gadget doesn't show any user interface at all. It runs in the background, perhaps monitoring something, and periodically sends data to the device, allowing the data to be viewed when the computer is turned off. To allow the user to remain unaware of non-UI gadgets, Windows offers a mechanism that can automatically invoke non-UI gadgets without the user being involved. Whenever the user logs onto his account, resumes the computer from a sleep state, or uses the SideShow control panel applet to connect or disconnect a device from a gadget, Windows automatically invokes all gadgets present in the registry by executing their StartCommand registry values.

Windows doesn't employ any sophisticated algorithm here; it simply invokes all the registered gadgets. This means that a gadget application could already be running and if the user connects or disconnects a device, Windows will simply run the application again. For this reason, non-UI gadgets should immediately check whether another instance is currently running and, if so, additional instances should just kill themselves. Also, a non-UI gadget would normally terminate itself when the user has disconnected all devices from it. See the Events section later in this article for more about this.

UI gadgets may also want to ensure that only one instance is executing to prevent each application from clobbering data sent to the device by another instance. Furthermore, you typically do not want Windows to invoke UI gadgets automatically (as the UI would appear) and therefore, most UI gadgets should not have a StartCommand registry value at all.

Content

All gadgets push content down to a device. The device has a home display that shows all the gadgets that are connected to the device. In Figure 9, the upper-left image shows a sample home screen. As you can see, each gadget appears with an icon, a name, and optionally some text. The icon and name come from the Icon and FriendlyName registry values. The text shown here is called the glance content and is sent from the gadget to the device by calling SideShowGadget's AddGlanceContent method when the gadget is running.

Navigating through Gadget Screens

Figure 9a Device Home Page
Figure 9a Device Home Page(Click the image for a smaller view)
Figure 9a Device Home Page
Figure 9a Device Home Page (Click the image for a larger view)
Figure 9b Gadget Home Page (Content ID 1)
Figure 9b Gadget Home Page (Content ID 1)(Click the image for a smaller view)
Figure 9b Gadget Home Page (Content ID 1)
Figure 9b Gadget Home Page (Content ID 1) (Click the image for a larger view)
Figure 9c Text Page (Content ID 2)
Figure 9c Text Page (Content ID 2)(Click the image for a smaller view)
Figure 9c Text Page (Content ID 2)
Figure 9c Text Page (Content ID 2) (Click the image for a larger view)
Figure 9d Image Page (Content ID 3 Showing Image (Content ID 1000)
Figure 9d Image Page (Content ID 3 Showing Image (Content ID 1000)(Click the image for a smaller view)
Figure 9d Image Page (Content ID 3 Showing Image (Content ID 1000)
Figure 9d Image Page (Content ID 3 Showing Image (Content ID 1000) (Click the image for a larger view)

When the user enters the gadget on the device, the device renders the content previously downloaded from the gadget while it was running on Windows. Currently, SideShow supports two different kinds of content: Simple Content Format (SCF) and iCal; the latter is not discussed in this article. SCF content is a set of XML strings sent from the gadget application to the device. The device parses the XML strings in order to render the content on the device.

The new SCF format allows the gadget application to define three types of content: text and images, menus and context menus consisting of items that initiate navigation to other content, and dialogs that present the user with questions resulting in navigation to other content. Details about the SCF XML schema can be found at windowssdk.msdn.microsoft.com/ms744133.aspx.

The managed SideShow assembly (Microsoft.SideShow.dll) includes a public, static Scf class that contains a bunch of static methods allowing you to create the XML string in code easily. The DownloadDataToDevices method, shown in Figure 10, demonstrates how to open an ScfSideShowGadget and send glance string content. The sample also shows how to call ScfSideShowGadget's AddContent method to send SCF menu, SCF text, and SCF image content down to a device. As you can see, each piece of content is identified by a content ID number. The first page of the gadget (the lower-left image in Figure 9) is its home page which always has a content ID of 1 (represented by the ScfSideShowGadget's HomePageContentId constant). When the user selects the See Text menu item, the device will navigate to the content page with an ID of 2 and when the user selects the "See Image" menu item, the device will navigate to the content page with an ID of 3. Page 3, the Image page, contains XML describing how the image should be shown (centered and auto fitting). This page also refers to the actual image, which I've assigned a content ID of 1000. In my code, I've extracted the image from an assembly resource.

Figure 10 DownloadDataToDevices Method (Close Figure 10)
private static void DownloadDataToDevices() {
   // Construct a SCF SideShow gadget for our gadget's Guid
   using (ScfSideShowGadget gadget = new ScfSideShowGadget(s_gadgetId))
   {
      // Add Glance content showing date/time of last update
      gadget.AddGlanceContent(
         String.Format("Content updated on {0}{1:D}.",
            Environment.NewLine, DateTime.Now));

      // The home page is a menu with an item that goes to some
      // text and another item that goes to an image
      gadget.AddContent(
         Scf.Menu(ScfSideShowGadget.HomePageContentId,
            "Simple Gadget Menu", ScfSelectAction.Target,
            Scf.Item(2, "See text"), Scf.Item(3, "See image")));

      // This is the text page
      gadget.AddContent(
         Scf.Content(2, "This is the Text Page",
            Scf.Txt(ScfAlign.Left, true, Color.Red,
               "This is some text")));

      // This is the image page
      gadget.AddContent(
         Scf.Content(3, "This is the Image Page",
            Scf.Img(1000, ScfAlign.Center, ScfImageFit.Auto,
               "alt picture text")));

      // This is the image itself (that appears on the image page)
      Stream imageStream = Assembly.GetEntryAssembly().
         GetManifestResourceStream(
            "SimpleSideShowGadget.Aidan And Jeff.jpg");
      gadget.AddContent(1000, ImageContentTransforms.ReduceColorDepth |
         ImageContentTransforms.StretchToFit,
         Image.FromStream(imageStream));
   }
}

Capabilities, Events, and Notifications

A gadget application can query Windows, asking it how many devices are connected to it and what the capabilities are of these devices. Examples of capabilities include DeviceId, ScreenType (Bitmap or Text), ScreenWidth, ScreenHeight, ClientAreaWidth, ClientAreaHeight, ColorDepth, and SupportedImageFormats.

A gadget application could use the device capabilities when deciding what content to push down to a device. The sample application associated with this article demonstrates how a gadget application can query the number of devices connected to it and can also query the capabilities of these devices.

When running, a gadget application can receive events (described in Figure 11) from a device by registering with several events. The SideShowGadget type (the base class of the all gadget types) offers events that apply to all gadgets regardless of endpoint. The ScfSideShowGadget type (derived from SideShowGadget) offers additional events specific to the SCF gadget endpoint. An application can respond to these events in order to add or remove content. A sophisticated gadget may require that it always be executing in order to properly respond to these events. If this is required, the gadget should set its OnlineOnly registry value to 1.

Figure 11 Device Events to which Your Gadget Can Respond (Close Figure 11)

Event | Description
DeviceAdded | Raised when the user connects a device to the gadget application. The receiving method is passed the device capabilities of the device.
DeviceRemoved | Raised when the user disconnects a device from the gadget application. The receiving method is passed the device capabilities of the device.
AllDevicesRemoved | Raised when the user has disconnected all devices from a gadget. Usually a non-UI gadget will terminate itself when it receives this event.
ContentMissing | Raised when the user attempts to navigate to content on the device that is missing. The receiving method is passed the ID of the missing content. As mentioned previously, the managed wrapper saves all content in a collection. If a device discards content due to the gadget's caching policy (as set by its CacheAlgorithm registry value), then the managed wrapper automatically sends the content back to the device and your code will not receive this event. This is one of the big benefits of using the managed API over the native API.
GadgetEnter | Raised when the user enters the gadget on the device.
GadgetExit | Raised when the user exits the gadget on the device.
ContentNavigate | For SCF gadgets only. Raised when the user navigates from one content page to another. The receiving method is passed the previous page ID, the target page ID, and the button that caused the navigation to occur.
MenuSelect | For SCF gadgets only. Raised when the user selects an item from a menu. The receiving method is passed the previous page ID, the target page ID, the menu item ID, and the button that caused the menu selection to occur.
ContextMenuSelect | For SCF gadgets only. Raised when the user selects an item from a context menu. The receiving method is passed the previous page ID, the target page ID, the previous item ID, the menu page ID, and the menu item ID.

When running, a gadget can send a notification to a SideShow device by calling SideShowGadget's ShowNotification method. A gadget application would show a notification for things such as meeting reminders, e-mail alerts, an incoming instant message, or system alerts. A notification is similar to a message box except that it appears on the hardware device. A notification consists of an ID number, an icon, a string caption, a string message, and an expiration time. The device will then show the notification until it is dismissed by the user or until the expiration time has elapsed. An application can also revoke a notification by calling SideShowGadget's RevokeNotification method passing it the notification's ID number.

Conclusion

In this article, I have covered the goals of the Windows SideShow feature and explained in depth how an end user interacts and configures the gadget. I've also shown how developers can integrate SideShow support into their new or existing managed applications. By using these managed wrappers, you should easily be able to integrate SideShow functionality into your own applications.

Personally, I find the new Windows SideShow feature very exciting as it offers new ways for users to access and interact with their data. I expect a slew of new hardware devices that support SideShow to become available in the next few years. In particular, I can't wait until Media Center remote controls and mobile phones support SideShow. It shouldn't take long for these incredibly useful devices to be as commonplace as a PDA or a cell phone.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), a training and consulting firm. He is the author of several books, including CLR via C# (Microsoft Press, 2006). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


Desktop Security: Create Custom Login Experiences With Credential Providers For Windows Vista

New information has been added to this article since publication.
Refer to the Editor's Update below.

Desktop Security

Create Custom Login Experiences With Credential Providers For Windows Vista

Dan Griffin
This article discusses:
  • The new Credential Provider architecture
  • Why GINA-based authentication was dropped
  • Multi-factor authentication
  • Developing and debugging a credential provider
This article uses the following technologies:
Windows Vista, C++
Code download available at: CredentialProviders2007_01.exe (241KB)
Contents
Comparing the Old and the New
A Hybrid Credential Provider
The Requirements
The Design
The Hybrid Credential Provider
The Hybrid Implementation
Opportunities for Improvement
Testing and Debugging
Smart Cards and Initialization

W
indows Vista offers developers many new opportunities for integrating with the platform. The new Credential Provider model represents one of the most dramatic changes, making it much easier to implement new user authentication scenarios that are supported by the OS. This has replaced the GINA (Graphical Identification and Authentication) model-a model that, put bluntly, is known for being difficult for developers to understand and implement as well as being expensive for Microsoft to support.

So why is a change to the Windows® logon plug-in interface so exciting? The logon screen is the first thing users see when they turn on the computer. Now that the experience is driven by credential providers, it's much easier to customize the logon experience and integrate the authentication methods that best meet an organization's needs. Simply put, credential providers offer an easier way to develop and implement better, more robust security.

Comparing the Old and the New

I don't want to go into too much detail about the GINA-based logon architecture. However, it is worth taking a little time to compare the two architectures to help you better understand the new architecture and the changes it introduces.

In a pre-Windows Vista™ environment, every session has an instance of winlogon, which is responsible for driving the interactive logon sequence for that session. (Figure 1 illustrates the old logon architecture under Windows XP and Windows Server® 2003.) On a freshly booted system, an interactive logon at the console is always performed in session zero. Session zero hosts system services and other critical processes, including the Local Security Authority process. (In other words, there are lots of processes running in session zero that Figure 1 doesn't show.)

Figure 1 GINA Logon Architecture
Figure 1 GINA Logon Architecture

The registered GINA on the machine is loaded into the winlogon process space. (A configuration known as "GINA chaining" is also possible, but such a complex configuration is difficult to test and support.) Finally, GINA makes calls to LogonUser and related authentication APIs.

In Windows Vista, session zero is never used for interactive logon (see Figure 2). This is good for security- there is now a session boundary that separates all per-machine processes from per-user processes. Additionally, the kernel Global namespace is now more tightly controlled, since objects created by user applications are kept out of it by default.

Figure 2 New Logon Architecture
Figure 2 New Logon Architecture

There's still an instance of winlogon in every session other than session zero. The figure illustrates that several credential providers have been registered on the system and loaded by the new LogonUI process.

There has also been an important change regarding which component renders the graphical aspect of the logon. Previously, this was handled by GINA and thus a third-party component could have been doing the rendering. In the new architecture, LogonUI, which is a built-in piece of the operating system, is responsible for this.

So how is per-provider user prompting behavior achieved in the new model? The Credential Provider architecture requires each provider to enumerate its UI elements. For example, in a given scenario, a provider might indicate to LogonUI that it requires two edit boxes, two captions, a checkbox, and a bitmap. In turn, LogonUI renders those controls on behalf of the credential provider. This goes a long way toward achieving the goal discussed previously-a consistent look and approach that supports a broad set of evolving authentication scenarios.

The Microsoft development team responsible for Credential Provider thought that external developers would be more comfortable with a COM-based plug-in model. However, early in the Windows Vista development cycle, the first internal design for the new interface was based (like GINA) purely on LoadLibrary and function pointers. The lessons learned from that first attempt were rolled into the subsequent COM-based redesign, and the resulting interface is cleaner and easier to use. Now let's turn to the sample code to help guide us as we drill into the credprov interface.

A Hybrid Credential Provider

The timing of this new plug-in model couldn't be better (well, OK, perhaps it's overdue). Now, developers can more easily meet the demand for multi-factor authentication scenarios while providing an experience consistent with what Microsoft provides out of the box.

That said, the new interface is rather abstract. An equally abstract description of it would be really boring! A more interesting way to get to know it is by walking through the design, development, and testing of a new credential provider. Also, this will better complement the documentation that Microsoft has already provided-see the "Additional Resources" sidebar for pointers.

I've created a sample, the "hybrid credential provider," which demonstrates some neat new features. The hybrid credential provider allows a user name, password, and domain name to be stored on a smart card. Upon insertion of a card, the user is automatically logged on. (The sample code can be downloaded from the MSDN®Magazine Web site.) I didn't write the code from scratch. Instead, I combined code from three sources:

  • The sample password-based Credential Provider available in the Microsoft® Windows SDK.
  • The old PropCert sample, also from the SDK. At its core is a Win32® thread that reads certificate-based smart card credentials.
  • The sample code that is included with my article in the November 2006 issue of MSDN Magazine. The article discussed interfacing with the Windows smart card subsystem via managed code.

Further clarification is needed regarding the sample code provided with my November 2006 article. The credential provider architecture and its host support only native code. While my first article focused on managed code, it included a native helper DLL that conveniently exposes the new Smart Card Module interface. The hybrid credential provider is built on that helper DLL. If you want the full source code for that DLL, it's also available via the download accompanying the November 2006 article.

In summary, a high percentage of the hybrid credential provider code base is not new. The net result is minimal time spent testing and debugging. In fact, the core debugging phase took less than a day, which speaks to the ease-of-use of the new interface.

Let's discuss in more detail what I set out to accomplish with the sample credential provider.

The Requirements

When planning the hybrid credential provider, I had these requirements in mind:

  • Make it smart card-based
  • Maximize code reuse
  • Minimize additional config and infrastructure requirements

This led me to my hybrid approach, in other words password (for security) plus smart card (for convenience). Since the hybrid provider concept is user name and password-based, I started with the stripped-down sample password provider from the Platform SDK. Then I added the PropCert sample from the SDK; this includes the logic to enumerate smart card readers, cards, and digital certificates. I figured all I had to do was replace the certificate-based logic in PropCert with some new code for reading my own credential data, and then I'd simply connect the two samples to each other!

Since we'll be reading password logon information from a smart card, this implies another requirement: a tool to initialize a smart card with that credential. I'll save the discussion of the initialization tool for the end.

With those requirements in mind, let's look at the design of the credential provider architecture and how it drove the design of my sample code.

The Design

Let's first discuss the design of the credential provider architecture, from the perspective of a credential provider at run time.

Although I have yet to discuss the hybrid sample in detail, I'm using it as the basis for analyzing the new credential provider architecture in action. To facilitate this discussion, my sample code includes debug tracing. The tracing consists of a call to OutputDebugString from each implemented credential provider routine. In those trace calls, I use two abbreviations. Calls to the new ICredentialProvider interface (excerpted in Figure 3) are prefaced with "Provider::". Calls to the ICredentialProviderCredential interface (see Figure 4) are prefaced with "Credential::". Note that all credential provider-related interfaces are defined in the new public header, credentialprovider.h.

Figure 4 ICredentialProviderCredential Interface (Close Figure 4)
ICredentialProviderCredential : public IUnknown
{
    HRESULT STDMETHODCALLTYPE Advise(
        /* [in] */ ICredentialProviderCredentialEvents *pcpce);

    HRESULT STDMETHODCALLTYPE UnAdvise( void);

    HRESULT STDMETHODCALLTYPE SetSelected(
        /* [out] */ BOOL *pbAutoLogon);

    HRESULT STDMETHODCALLTYPE SetDeselected( void);

    HRESULT STDMETHODCALLTYPE GetFieldState(
        /* [in] */ DWORD dwFieldID,
        /* [out] */ CREDENTIAL_PROVIDER_FIELD_STATE *pcpfs,
        /* [out] */ CREDENTIAL_PROVIDER_FIELD_INTERACTIVE_STATE *pcpfis);

    HRESULT STDMETHODCALLTYPE GetStringValue(
        /* [in] */ DWORD dwFieldID,
        /* [string][out] */ LPWSTR *ppsz);

    HRESULT STDMETHODCALLTYPE GetBitmapValue(
        /* [in] */ DWORD dwFieldID,
        /* [out] */ HBITMAP *phbmp);

    HRESULT STDMETHODCALLTYPE GetCheckboxValue(
        /* [in] */ DWORD dwFieldID,
        /* [out] */ BOOL *pbChecked,
        /* [string][out] */ LPWSTR *ppszLabel);

    HRESULT STDMETHODCALLTYPE GetSubmitButtonValue(
        /* [in] */ DWORD dwFieldID,
        /* [out] */ DWORD *pdwAdjacentTo);

    HRESULT STDMETHODCALLTYPE GetComboBoxValueCount(
        /* [in] */ DWORD dwFieldID,
        /* [out] */ DWORD *pcItems,
        /* [out] */ DWORD *pdwSelectedItem);

    HRESULT STDMETHODCALLTYPE GetComboBoxValueAt(
        /* [in] */ DWORD dwFieldID,
        DWORD dwItem,
        /* [string][out] */ LPWSTR *ppszItem);

    HRESULT STDMETHODCALLTYPE SetStringValue(
        /* [in] */ DWORD dwFieldID,
        /* [string][in] */ LPCWSTR psz);

    HRESULT STDMETHODCALLTYPE SetCheckboxValue(
        /* [in] */ DWORD dwFieldID,
        /* [in] */ BOOL bChecked);

    HRESULT STDMETHODCALLTYPE SetComboBoxSelectedValue(
        /* [in] */ DWORD dwFieldID,
        /* [in] */ DWORD dwSelectedItem);

    HRESULT STDMETHODCALLTYPE CommandLinkClicked(
        /* [in] */ DWORD dwFieldID);

    HRESULT STDMETHODCALLTYPE GetSerialization(
        /* [out] */ CREDENTIAL_PROVIDER_GET_SERIALIZATION_RESPONSE
                    *pcpgsr,
        /* [out] */ CREDENTIAL_PROVIDER_CREDENTIAL_SERIALIZATION *pcpcs,
        /* [out] */ LPWSTR *ppszOptionalStatusText,
        /* [out] */ CREDENTIAL_PROVIDER_STATUS_ICON
                    *pcpsiOptionalStatusIcon);

    HRESULT STDMETHODCALLTYPE ReportResult(
        /* [in] */ NTSTATUS ntsStatus,
        /* [in] */ NTSTATUS ntsSubstatus,
        /* [out] */ LPWSTR *ppszOptionalStatusText,
        /* [out] */ CREDENTIAL_PROVIDER_STATUS_ICON *pcpsiOptionalStatusIcon);
};
Figure 3 ICredentialProvider Interface (Close Figure 3)
ICredentialProvider : public IUnknown
{
    HRESULT STDMETHODCALLTYPE SetUsageScenario(
        /* [in] */ CREDENTIAL_PROVIDER_USAGE_SCENARIO cpus,
        /* [in] */ DWORD dwFlags);

    HRESULT STDMETHODCALLTYPE SetSerialization(
        /* [in] */ const CREDENTIAL_PROVIDER_CREDENTIAL_SERIALIZATION
            *pcpcs);

    HRESULT STDMETHODCALLTYPE Advise(
        /* [in] */ ICredentialProviderEvents *pcpe,
        /* [in] */ UINT_PTR upAdviseContext);

    HRESULT STDMETHODCALLTYPE UnAdvise( void);

    HRESULT STDMETHODCALLTYPE GetFieldDescriptorCount(
        /* [out] */ DWORD *pdwCount);

    HRESULT STDMETHODCALLTYPE GetFieldDescriptorAt(
        /* [in] */ DWORD dwIndex,
        /* [out] */ CREDENTIAL_PROVIDER_FIELD_DESCRIPTOR **ppcpfd);

    HRESULT STDMETHODCALLTYPE GetCredentialCount(
        /* [out] */ DWORD *pdwCount,
        /* [out] */ DWORD *pdwDefault,
        /* [out] */ BOOL *pbAutoLogonWithDefault);

    HRESULT STDMETHODCALLTYPE GetCredentialAt(
        /* [in] */ DWORD dwIndex,
        /* [out] */ ICredentialProviderCredential **ppcpc);
};

With that in mind, see Figure 5 for the list of the debugging events that occur during a sample scenario (I'll describe most of the events in detail). The scenario for generating the call sequence is simple. Start with a Windows Vista workstation joined to a domain. Configure a smart card with your user name, password, and domain name. Insert the smart card in a reader attached to the machine. Then reboot the system.

Figure 5 Hybrid Credential Provider Calling Sequence (Close Figure 5)
1. [The system boots]

2. [LogonUI.exe process is created]

3. [Credential provider DLLs are loaded]

4. Provider::CreateInstance

5. [User presses Ctrl+Alt+Del]

6. Provider::SetUsageScenario (CPUS_LOGON)

7. Credential::Initialize

8. Provider::Advise

9. Provider::GetCredentialCount

10. Provider::GetCredentialAt (dwIndex = 0)

11. Provider::GetFieldDescriptorCount

12. Provider::GetFieldDescriptorAt (dwIndex = 0)

13. Provider::GetFieldDescriptorAt (dwIndex = 1)

14. Provider::GetFieldDescriptorAt (dwIndex = 2)

15. Provider::GetFieldDescriptorAt (dwIndex = 3)

16. Provider::GetFieldDescriptorAt (dwIndex = 4)

17. Credential::GetBitmapValue (dwFieldID = 0; tile image)

18. Credential::GetStringValue (dwFieldID = 1; user name field)

19. Credential::GetFieldState (dwFieldID = 1; user name field)

20. Credential::GetStringValue (dwFieldID = 2; password field)

21. Credential::GetFieldState (dwFieldID = 2; password field)

22. Credential::GetSubmitButtonValue (dwFieldID = 3; submit button)

23. Credential::GetFieldState (dwFieldID = 3; submit button)

24. Credential::GetStringValue (dwFieldID = 4; domain name field)

25. Credential::GetFieldState (dwFieldID = 4; domain name field)

26. Credential::Advise

27. Credential::GetSerialization

28. Credential::UnAdvise

29. Provider::UnAdvise

30. [The WinLogon process calls LogonUser]

31. Credential::Advise

32. Credential::ReportResult (ntsStatus = 0)

33. Credential::UnAdvise

First, the console session LogonUI process is started by winlogon. Upon creation, LogonUI enumerates all of the credential providers registered under HKLM\Software\Microsoft\Windows\CurrentVersion\Authentication\Credential Providers. Each provider DLL is loaded and receives a Provider::CreateInstance call. For the hybrid credential provider, this results in the creation of a CHybridProvider. (See steps 1 through 4 in Figure 5.)

The user now sees the logon screen. Assuming the user presses Ctrl+Alt+Delete, and each provider receives Provider::SetUsageScenario CPUS_LOGON notification. This indicates to the provider that the user wants to perform an interactive logon. Now, the hybrid credential provider attempts to read a credential from any inserted smart card. If it finds one, a CHybridCredential is instantiated and associated with the current CHybridProvider. There will then be a call to Credential::Initialize. (See steps 5 through 7 in Figure 5.)

LogonUI then calls Provider::Advise for each loaded provider. The purpose of Advise is to give the providers a mechanism for notifying LogonUI asynchronously of any desired change to the visible UI elements (of which there are none yet). The built-in smart card provider gives a good example of how this is used. Any time after initialization, card insertion can increase the number of available credentials and card removal can decrease it. When that happens, LogonUI is notified via this mechanism:

ICredentialProviderEvents : public IUnknown
{
    HRESULT STDMETHODCALLTYPE CredentialsChanged(
       /* [in] */ UINT_PTR upAdviseContext);
};

For the sake of simplicity, the hybrid credential provider doesn't dynamically handle card insertion and removal. Therefore, it doesn't keep track of the ICredentialProviderEvents interface passed to it via Advise.

The next interface call by LogonUI is to Provider::GetCredentialCount, which is step 9 in Figure 5. In the case that a hybrid credential was created (due to an inserted smart card), the hybrid credential provider will take several actions. It will first set the GetCredentialCount *pdwCount output parameter to one. This refers to the number of credential tiles that the provider wants to enumerate. (The hybrid credential provider can only handle one.) When you first install Windows Vista and join a domain, you can infer what pdwCount value the Microsoft password credential provider returned to LogonUI based on the number of tiles rendered.

The hybrid then sets the GetCredentialCount *pdwDefault output parameter to zero. This value refers to a zero-based index into an array of credentials that each provider is assumed to maintain. The actual implementation of how a provider tracks its credentials is up to the implementer, as long as the indices are maintained for the lifetime of a given set of credential objects.

It's entirely possible that multiple providers will enumerate a default credential. For example, in the current scenario, you can expect the built-in password credential provider to enumerate a default credential of its own. How does LogonUI prompt the user to select from multiple default and non-default credentials without causing confusion? In general, the user is shown a tile for each credential, with focus set to the tile that represents the default credential. In the presence of multiple defaults, the true default is selected through a series of precedence rules as each of the default credentials is enumerated. For each credential, if there's already a default without auto-logon, and if this credential will do an auto-logon, this credential becomes the default. If this credential is from the last-logged-on (LLO) provider, and if there isn't already a default with auto-logon, this credential becomes the default. And finally, if there's no default yet, this credential becomes the default. All that said, the auto-logon semantics of my hybrid credential provider make this discussion moot. As long as the enumerated hybrid credential contains valid logon information, the user never sees any tiles. I'll explain this in a bit.

I mentioned the last-logged-on provider with regards to precedence rules, but it should be pointed out that the meaning of LLO changes based on whether the user is logging in or whether it's a post-logon scenario, such as a desktop lock or a password change. At logon, the LLO provider is the last provider that was used for the last logon to the console. Post-logon, the LLO provider is the one that was used for logon to that session only. The idea is that if you always log on with your smart card, your smart card credential provider default tile will win across reboots. But if you lose your smart card and log on with your password, the password credential provider's tile will win for that session when you unlock.

The hybrid credential provider always sets the *pbAutoLogonWithDefault output parameter to TRUE. This serves notice to LogonUI that it should immediately query this provider's default credential for logon information and that there's no need to prompt the user first. Note that the built-in password credential provider has the same capability via the optional password auto-logon information that can be stored in the registry. In fact, this is the default behavior if Windows Vista detects that there's only one user on the machine with no password yet. If multiple credential providers set *pbAutoLogonWithDefault to TRUE, the behavior of LogonUI is undefined.

After GetCredentialCount, LogonUI calls Provider::GetCredentialAt. For the hybrid credential provider, this routine is called at most once, reflecting the maximum credential count for this provider. In response, the provider returns an ICredentialProviderCredential pointer for the credential instance that corresponds to the requested index.

Next, LogonUI calls Provider::GetFieldDescriptorCount, via which the provider returns the maximum number of UI elements that may be found in its credentials. For example, the sample password credential provider has five fields: a bitmap, a username input field, a password input field, a submit button, and a domain name input field. You can see that these same elements are preserved in the hybrid credential provider, even though they're never actually rendered. This completes step 11 in Figure 5.

LogonUI then calls Provider::GetFieldDescriptorAt one time for each UI element in order to retrieve its type. For example, in response to the call corresponding to the index of the bitmap, the sample returns the CREDENTIAL_PROVIDER_FIELD_TYPE CPFT_TILE_IMAGE. One feature not used in the hybrid credential provider is writeable versus read-only text fields. If the hybrid credential provider were modified to prompt the user for a smart card PIN, that would be accomplished with CPFT_PASSWORD_TEXT. The user name read from the smart card can be displayed to provide some context for that prompt. But, technically, the user name should be considered read-only since it's bound to the password also stored on the card. Therefore, the CPFT_LARGE_TEXT field type (as opposed to CPFT_EDIT_TEXT) might be used. (For the full list of options, see credentialprovider.h.)

Following the enumeration of field descriptors, LogonUI makes a sequence of calls into the credential provider based on the type of each credential field. For the CPFT_TILE_IMAGE field type, for example, LogonUI follows up with a call to Credential::GetBitmapValue. For text values such as the CPFT_LARGE_TEXT used for the user name edit box, there are subsequent calls to Credential::GetStringValue and Credential::GetFieldState.

Since all of the required logon information (user name, password, and domain name) for my hybrid credential provider has already been read from the smart card, the strings corresponding to each text field are available at this time; and are returned via the ppwz output parameter of GetStringValue. Other providers are likely to return a NULL string value in response to GetStringValue at this point, since the user hasn't had a chance to type in anything yet. Note this one potentially confusing point: the name of the text field is retrieved via GetFieldDescriptorAt while the current text value in the field is retrieved via GetStringValue. (The name or label of the field will be displayed as the cue text in an empty edit control.)

After the various UI elements have been fully described, LogonUI calls Credential::Advise. (See step 26 in Figure 5.) This serves a similar purpose to the Provider::Advise interface that was called earlier; each credential can asynchronously notify LogonUI of relevant changes affecting the state of its UI elements. As an example, the sample password credential provider uses this mechanism when one of its credential tiles is deselected. In that case, ICredentialProviderCredentialEvents SetFieldString (see Figure 6) is used by the credential object to clear out the password field. This is analogous to what happens when you type in only part of your password at the logon screen in Windows XP and then pause. Eventually, the logon dialog times out and the text is cleared.

Figure 6 ICredentialProviderEvents Interface (Close Figure 6)
ICredentialProviderCredentialEvents : public IUnknown
{
    HRESULT STDMETHODCALLTYPE SetFieldState(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ CREDENTIAL_PROVIDER_FIELD_STATE cpfs);

    HRESULT STDMETHODCALLTYPE SetFieldInteractiveState(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ CREDENTIAL_PROVIDER_FIELD_INTERACTIVE_STATE cpfis);

    HRESULT STDMETHODCALLTYPE SetFieldString(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [unique][string][in] */ LPCWSTR psz);

    HRESULT STDMETHODCALLTYPE SetFieldCheckbox(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ BOOL bChecked,
        /* [in] */ LPCWSTR pszLabel);

    HRESULT STDMETHODCALLTYPE SetFieldBitmap(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ HBITMAP hbmp);

    HRESULT STDMETHODCALLTYPE SetFieldComboBoxSelectedItem(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ DWORD dwSelectedItem);

    HRESULT STDMETHODCALLTYPE DeleteFieldComboBoxItem(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ DWORD dwItem);

    HRESULT STDMETHODCALLTYPE AppendFieldComboBoxItem(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [string][in] */ LPCWSTR pszItem);

    HRESULT STDMETHODCALLTYPE SetFieldSubmitButton(
        /* [in] */ ICredentialProviderCredential *pcpc,
        /* [in] */ DWORD dwFieldID,
        /* [in] */ DWORD dwAdjacentTo);

    HRESULT STDMETHODCALLTYPE OnCreatingWindow(
        /* [out] */ HWND *phwndOwner);
};

In terms of completing a user authentication, the next call is the most interesting. As a result of the *pbAutoLogonWithDefault parameter of GetCredentialCount having been set to TRUE, LogonUI knows that the default credential should already contain sufficient data for authenticating the user (even though no UI elements have yet been rendered and, in turn, no user input gathered). In this case, the Credential::GetSerialization routine is called to retrieve the user name, password, and optional domain name. The credential provider prepares the return value for that routine by marshaling the three items into the format expected by Kerberos. Once the serialized credential has been prepared, the credential provider informs LogonUI via the CREDENTIAL_PROVIDER_GET_SERIALIZATION_RESPONSE type output parameter that a complete credential is being returned. The value CPGSR_RETURN_CREDENTIAL_FINISHED makes this distinction. Again, see credentialprovider.h, as well as the implementation of GetSerialization in the sample code. This completes step 27 in Figure 5.

After GetSerialization, LogonUI passes the marshaled credential to winlogon, which in turn passes it to the Local Security Authority (LSA) by calling LogonUser. Prior to this, LogonUI calls Credential::UnAdvise and Provider::UnAdvise to notify both entities that notifications on their respective Events interfaces are not being accepted. UI changes would be pointless with a logon attempt pending (ideally, the next thing the user sees is his desktop).Additional Resources

  • Credential Provider Samples
  • Smart Storage: Protect Your Data via Managed Code and the Windows Vista Smart Card APIs
  • Download Windows Symbol Packages
  • Windows Smart Card API Documentation
  • Card Module API Documentation

After winlogon gets the result of LogonUser, the result is passed back to LogonUI and then the credential instance (the one that still has focus from GetSerialization) is notified. But before the credential receives the status code returned from LogonUser, it is again given a callback interface for making UI element changes. (See step 31 in Figure 5.)

The result of the authentication attempt is returned to the credential via the Credential::ReportResult routine. Why does a provider (or its credential objects) care about the result of the authentication attempt and why would it start making UI changes at this point?

Many of the interesting ReportResult scenarios stem from authentication failure. One of the canonical examples is user password expiration. If the user's password is near expiration, the authentication sub-status, returned via the ReportResult ntsSubstatus parameter, indicates this. In response, the built-in password credential provider prompts the user to (optionally) initiate a password change. This prompt, as well as the password change dialog itself, requires different UI elements. Thus, the password credential provider utilizes the ICredentialProviderCredentialEvents interface pointer to drive the required changes to its prompting fields.

There are interesting actions a credential provider can take in response to authentication success. For example, the built-in smart card credential provider uses this success notification as a cue to start monitoring for removal of the card that was used for authentication. It does this in order to enforce the optional session lock-on-removal policy. Once the credential-handling sequence is complete, LogonUI notifies the credential provider via Credential::UnAdvise to decrement its reference to the ICredentialProviderCredentialEvents interface pointer.

The Hybrid Credential Provider

Now that I've discussed the new credential provider architecture and how it's used, let's look at the design of the sample hybrid credential provider in more detail. Recall the high-level layout of the Windows Vista interactive logon architecture shown in Figure 2. Figure 7 augments that diagram to include the Windows Smart Card API stack and to focus on the new credential provider.

The main point to take from Figure 7 is that the hybrid credential provider interfaces with the Windows Smart Card API both directly and indirectly. The direct interface is via public routines, such as SCardEstablishContext and SCardListReaders, which allow the detection of a smart card. The indirect interface is via the Card Module API, which allows the credential provider to read a user credential file from the card in a convenient way, without having to resort to low-level card-specific commands. For the sample, almost all of the smart card-related logic is abstracted by a helper library called ScHelp.lib. (I discuss this later in the Implementation section.) The hybrid credential provider's auto-logon behavior offers an interesting view into the capabilities and subtleties of the Credential Provider architecture as a whole.

Figure 7 Hybrid Credential Provider
Figure 7 Hybrid Credential Provider

First, what do I mean by auto-logon? This means that if a hybrid credential is available then the user isn't prompted at all, as we've already seen. Instead, in that case, an authentication attempt is made automatically as soon as the user presses Ctrl+Alt+Delete.

The auto-logon behavior as implemented might be confusing to some users. For example, it's different from the built-in password provider, which, by default, always renders tiles, even in the non-domain-joined scenario in which the user isn't prompted for a password.

Changing the hybrid credential provider to require the user to first click a tile in all scenarios would be easy. Please note that the current implementation of GetCredentialCount is to set *pbAutoLogonWithDefault = TRUE. Instead, you should set it to FALSE. Now, the provider is guaranteed the opportunity to show at least one tile (unless another provider overrides it with a no-tile auto-logon).

When the user clicks the tile, LogonUI calls the provider's ICredentialProviderCredential::SetSelected method. In response, the credential class will set *pbAutoLogon = TRUE, triggering a call to ICredentialProviderCredential::GetSerialization by LogonUI, and a subsequent authentication attempt, without first rendering any UI element changes. In other words, if the authentication is successful, the next thing the user will see is his desktop.

The Hybrid Implementation

The changes required to transform the password credential provider into my hybrid implementation were minimal. Use a graphical comparison tool (such as windiff.exe) to compare the SDK CSampleProvider.cpp to my CHybridProvider.cpp; likewise, compare CSampleCredential.cpp to CHybridProvider.cpp. Most of the modified lines of code are from globally replacing "Sample" with "Hybrid"!

The most substantial change to CHybridProvider.cpp is in its handling of SetUsageScenario. In response to this call, the provider attempts to read a credential from a smart card. This is done via the ScHelpInit routine in the ScHelp library, which abstracts most of the smart card logic. ScHelpInit connects to the smart card subsystem, finds the first inserted card, parses the credential if it finds one, and returns the strings contained therein.

The primary change to CHybridCredential.cpp is to handle the optional domain name string to be read as part of the credential file on the inserted smart card. In GetSerialization, if a domain name is read from the card, it is used in the serialized authentication data to be passed to Kerberos. Otherwise, the result of calling the public GetComputerName is used.

The changes made to convert the PropCert sample into the ScHelp library were more extensive. A few aspects of this helper library are worth noting.

First, the main PropCert thread routine is now being called synchronously; a separate thread is not used. However, card-related operations should be performed asynchronously and that would be a relatively simple change to make. In fact, this change would be critical if the hybrid credential provider were extended to allow the user to select from multiple credential tiles. In that case, you would want the provider to immediately enumerate a tile while reading smart card data in the background, since I/O operations on some older cards can be quite slow. To round out the implementation of this change, note that you would need to establish some sort of notification mechanism to allow the smart card thread to notify the provider of credential availability changes. The provider could then notify LogonUI of the same via CredentialsChanged.

The remaining logic in ScHelp.lib includes the routines _ReadCreds, _Connect, _UnpackCred (see Figure 8), and ScHelpPackCred. The latter two deserialize and serialize the password credential file stored on a smart card. The first two implement the logic I've described briefly: enumerate smart card readers and cards, obtain a read lock on the first enumerated card, bind to the card module corresponding to that card, and read the credential file (if one exists) from the card.

Figure 8 UnpackedCred Function (Close Figure 8)
// Break down the credential byte array
DWORD WINAPI UnpackCred(
    __in_bcount(cbCred)     PBYTE pbCred,
    __in                    DWORD cbCred,
    __out                   LPWSTR *ppwszUserName,
    __out                   LPWSTR *ppwszPassword,
    __out                   LPWSTR *ppwszDomainName)
{
    DWORD status = ERROR_SUCCESS;
    DWORD cbUserName = 0;
    DWORD cbPassword = 0;
    DWORD cbDomainName = 0;
    DWORD cbCurrent = 0;

    *ppwszUserName = NULL;
    *ppwszPassword = NULL;
    *ppwszDomainName = NULL;

    try
    {
        // Read the user name
        cbUserName = (DWORD) sizeof(WCHAR) * (1 + wcslen(
            (LPWSTR) (pbCred + cbCurrent)));
        if (cbUserName > cbCred - cbCurrent)
        {
            status = ERROR_INVALID_PARAMETER;
            leave;
        }

        if (NULL == (*ppwszUserName = (LPWSTR) Alloc(cbUserName)))
        {
            status = ERROR_NOT_ENOUGH_MEMORY;
            leave;
        }

        if (FAILED(StringCbCopy(
            *ppwszUserName, cbUserName, (LPWSTR) (pbCred + cbCurrent))))
        {
            status = ERROR_INSUFFICIENT_BUFFER;
            leave;
        }

        cbCurrent += cbUserName;

        // Read the password
        cbPassword = (DWORD) sizeof(WCHAR) * (1 + wcslen(
            (LPWSTR) (pbCred + cbCurrent)));
        if (cbPassword > cbCred - cbCurrent)
        {
            status = ERROR_INVALID_PARAMETER;
            leave;
        }

        if (NULL == (*ppwszPassword = (LPWSTR) Alloc(cbPassword)))
        {
            status = ERROR_NOT_ENOUGH_MEMORY;
            leave;
        }

        if (FAILED(StringCbCopy(
            *ppwszPassword, cbPassword, (LPWSTR) (pbCred + cbCurrent))))
        {
            status = ERROR_INSUFFICIENT_BUFFER;
            leave;
        }

        cbCurrent += cbPassword;

        // Read the domain name (if any)
        cbDomainName = (DWORD) sizeof(WCHAR) * (1 + wcslen(
            (LPWSTR) (pbCred + cbCurrent)));
        if (sizeof(WCHAR) == cbDomainName)leave;
        else if (cbDomainName > cbCred - cbCurrent)
        {
            status = ERROR_INVALID_PARAMETER;
            leave;
        }

        if (NULL == (*ppwszDomainName = (LPWSTR) Alloc(cbDomainName)))
        {
            status = ERROR_NOT_ENOUGH_MEMORY;
            leave;
        }

        if (FAILED(StringCbCopy(*ppwszDomainName, cbDomainName,
            (LPWSTR) (pbCred + cbCurrent))))
        {
            status = ERROR_INSUFFICIENT_BUFFER;
            leave;
        }
    }
    finally {}

    return status;
}
The Death of GINA

Why did GINA have to go? This question is more complex than it may seem. Microsoft has a respectable track record for supporting third-party developers, and the decision to drop support for a public interface is rarely taken lightly. Nonetheless, there were strong arguments for dropping GINA and ultimately Microsoft determined that that was the right decision.

First, multi-factor authentication is in much higher demand now than it was in the early days of Windows NT®. Smart cards, biometrics, and one-time password solutions are being widely deployed in enterprises. Each evolution in authentication technology places a greater burden on the abstraction layer between the core Windows credentialing engine and the GUI that must prompt the user in a certain way. (For example, the pictorial cues and screen real estate required to prompt the user to type in a password are different than those required to prompt the user to place a finger on a fingerprint reader.) At the same time, Microsoft needs to provide as consistent an experience as possible so that users don't get confused.

In addition, the Windows logon process (winlogon.exe) has been completely rearchitected in Windows Vista. One core requirement of that effort was to move plug-ins out of the winlogon process space to the fullest extent possible. That requirement was born of reliability concerns. If, for example, a poorly written GINA is loaded into the winlogon.exe instance running in session zero on a server, a software fault could kill that critical process and in turn the machine itself. And even if GINA could have been adapted to run out of process, there would still be the issue that it wasn't designed to provide a consistent, controlled experience across arbitrarily complex, interactive credential-gathering scenarios.

Opportunities for Improvement

While I believe that my design for the hybrid credential provider, in tandem with the significant level of code reuse I achieved in the implementation, meets the requirements I laid out earlier, there are aspects of the implementation that prevent the provider from being deployment-ready in its current state. I've already discussed one such shortcoming-data should be read from the smart card asynchronously. Now I'll discuss the remaining opportunities I see for improvement in decreasing order of severity.

First, the user credential is not being stored securely on the smart card. Ideally, reading the card-based file that stores the user password should only be possible after supplying the correct PIN. However, a limitation in the current card module interface makes that difficult to implement. Namely, the set of predefined card file access conditions does not include such a "user-only read" option. (I suppose it is poetic justice that, having been involved in the card module design decisions that led to this limitation, I'm now inconvenienced by it!) I hope the product team will extend the card module interface in a subsequent version.

In the meantime, the smart card password file ought to be encrypted in such a way that, if the card is stolen, the PIN is required in order to decrypt the password. This can be achieved easily via Crypto API, as well as via the new Windows Vista CNG ("Next Generation" Crypto API). An RSA key pair, created and stored on the card, would suffice, but rather than using the RSA public key to encrypt the password file directly, cryptographic best-practice advises you to use a symmetric key and algorithm, such as Advanced Encryption Standard (AES). The RSA key would instead encrypt the symmetric key.

To round out the design for an encrypted password file, you could modify the existing password file format to include the associated encrypted key. If you go this route, be sure to consider the versioning challenges that come with committing to a crypto algorithm and key size. Assume that the algorithm you choose will get hacked some day. Also, the design as discussed thus far doesn't include a cryptographically secure data integrity check. That might seem like a minor point, since an attacker theoretically needs to know the PIN in order to modify anything on the card. But I would consider such a feature to be a necessary aspect of defense-in-depth.

The second limitation of the current implementation is that it only supports a single credential per card. Take a look at the ScHelp.cpp!_UnpackCred routine from the sample code in Figure 8. This performs a simple deserialization of the password file, which was presumably read from the card. The credential-parsing logic as a whole will only handle a single credential per card. However, some users may require multiple distinct domain credentials to get their work done. Would you extend the provider to support that, or would you issue those users multiple cards? The former increases implementation complexity while the latter increases deployment management complexity.

As an aside, I've attempted to demonstrate secure buffer parsing techniques in _UnpackCred. (Again, refer to Figure 8.) Assume that an attacker can create an evil smart card and insert it into workstations on your network. At the application level, the primary mitigation against this threat is to ensure that no assumptions are made about the validity of data read from the card-don't assume that the embedded character counts are correct and don't assume that the strings are well-formed. Simply check that the element count at the beginning of every string doesn't exceed the length of the unparsed portion of the credential, and check that the true length of any string doesn't exceed the buffer allocated for it.

The last limitation I want to discuss in the current hybrid credential provider is that only one smart card reader per machine is supported. For example, if I boot a system with two smart card readers attached, each with a smart card inserted, and each card initialized with a different credential, which card takes precedence will depend on the order in which the readers are enumerated by the smart card subsystem. The fix entails changing the semantics of the SCHELP_CONTEXT struct, defined in ScHelp.h:

typedef struct _SCHELP_CONTEXT
{
    LPWSTR wszUserName;
    LPWSTR wszPassword;
    LPWSTR wszDomainName;
} SCHELP_CONTEXT, *PSCHELP_CONTEXT;

The SCHELP_CONTEXT struct defines the data exchange between the credential provider code and the ScHelp code. It's clear that the struct only supports a single credential; a simple array or singly linked list could be introduced to enhance it. If you want this feature, don't forget to modify the handling of the _rgpCredentials member of CSampleProvider, as well, since it is currently hardcoded to support only a single credential per provider instance.

[ Editor's Update - 6/22/2007: There is a problem with the implementation of the Hybrid Credential Provider sample for this article. As with the credential provider sample included with the RTM release of the Windows SDK for Windows Vista, the Credential::GetSerialization method returns a KERB_INTERACTIVE_LOGON structure. Unfortunately, this doesn't support unlock scenarios. Instead, a KERB_INTERACTIVE_UNLOCK_LOGON structure should be used to support both unlock and logon scenarios. You can see working examples of this in the Windows Vista Credential Provider Samples available for download at http://www.microsoft.com/downloads/details.aspx?FamilyID=b1b3cbd1-2d3a-4fac-982f-289f4f4b9300. ]
Testing and Debugging

As I mentioned, testing my credential provider was a relatively painless process. Regarding my testing strategy, I knew that I wanted to be able to attach a user-mode debugger to LogonUI in order to have maximum flexibility, both for live debugging of my sample code and for generating the tracing information I discussed earlier. I also knew that since LogonUI runs as system and that the interactive logon scenario I was targeting is accessible only from the secure desktop, writing a simple self-contained test program to exercise the various credential provider COM interfaces would be a wise use of time. Nevertheless, because of my background with the technology, I decided to skip writing a test program and instead proceeded with live debugging. But I don't recommend this approach.

Since I didn't write a test program, I had placed additional pressure on myself to get a robust debugging environment set up. Unless you frequently do kernel-mode development, getting this kernel debugger-based test environment properly configured can be frustrating. At a high-level, here's the preferred way of doing this.

First, set up two machines in a standard kernel debugging configuration. One should be a reliable development system (the debugger), the other the Windows Vista test system (the debuggee). They should be connected via serial cable.

Don't neglect to configure the debuggee with a safe boot partition with Windows XP loaded. Configuration (specifically, getting the two machines to talk to each other via serial cable) can take some trial and error. What's the best way to test the serial connection? Boot both machines to Windows XP and run HyperTerminal (by selecting All Programs | Accessories | Communications | HyperTerminal). On both machines, point the program to the serial port you're using and select the data rate you'll be passing to the debugger. If the characters you type into one machine's HyperTerminal window echo on the other machine, you're finished. If not, try another serial port, connection speed, or cable.

To describe the second reason for configuring a safe boot, I need to skip forward slightly. If you find that the credential provider under test causes the host LogonUI process to die or deadlock, then you won't be able to log into Windows Vista any more. This happened to me with my initial build configuration for HybridCredProv.dll. I was using the redistributable msvcr80.dll as my runtime library. My first mistake was that I forgot to copy that binary into the system32 directory of the debuggee. However, that only prevented my credential provider from being loaded.

The next reboot back into Windows Vista left me confused. This time, I saw LogonUI start and I saw my credential provider get loaded, but I never saw any UI appear. In the debugger, I found that the msvcr80.dll startup code was deadlocked on the process loader lock with another thread. Rather than drill into this, I modified the build configuration of HybridCredProv.dll to use a statically linked runtime library.

In summary, a safe boot partition provides a useful option for fixing configuration problems during credential provider testing.

Back to the debugging configuration. The next step is to install the latest debugger package from Microsoft (the one that includes ntsd.exe and i386kd.exe) on both systems. Install the public debug symbols for Windows Vista. (A full local copy of the symbols is recommended for the debuggee.) Some people might consider this step unnecessary, but there are few things more annoying than finding yourself in the middle of live debugging with insufficient data due to missing symbols. I offer the following points based on my own experiences:

  • I find it's important to always get a clean stack trace.
  • Some operating system symbol files (such as ntdll.pdb) may be required in order to get even an approximate stack trace.
  • Certain system-level debugging scenarios can inhibit network access in unforeseen ways, hence the need for local symbols.

Using Image File Execution Options in the system registry, configure LogonUI.exe to start within the user-mode debugger (ntsd.exe). That debugger will, in turn, redirect its output to the kernel debugger. That's why you need the latter-otherwise, since the application is only visible from the secure desktop, you have no reliable way to interact with both it and an attached debugger from the console of the test machine.

You should note that you can specify the path for loading symbols to ntsd.exe in two ways: via the -y command-line option or by the _NT_SYMBOL_PATH environment variable. The former is what's recommended when configuring Image File Execution Options. The latter, however, is the way I prefer, since it allows me to set it once, systemwide, on my test machines.

Finally, note that if your credential provider becomes unusable and you need to regain access to your machine, boot into safe mode and Windows Vista will load only the password provider, as well as the smart card credential provider if you boot into safe mode plus network. (There is a policy to turn off this fallback behavior for the security hyper-conscious.) Once booted into safe mode, you can edit the registry as appropriate.

Smart Cards and Initialization

Of course, any significant testing of the hybrid credential provider requires a credential. That is, a smart card must be prepared with the appropriately formatted credential file containing the user name, password, and optional domain name. The sample code download includes a test utility called WriteCred.exe to accomplish this. To initialize a smart card inserted in the default reader with your credential information, use the following command-line options:

WriteCred.exe -p <PIN> -u <UserName> -d <DomainName> -w <PassWord>

Unlike the behavior of the hybrid credential provider, the current implementation of WriteCred assumes that the domain name parameter is non-optional. Also, note that the password parameter should be the Windows logon password corresponding to the user name, whereas the PIN parameter is required in order to write the credential data file to the smart card.

Finally, regarding testing, the WriteCred tool, as well as the hybrid credential provider itself, only support smart cards compatible with the Microsoft Base Smart Card Crypto Provider. Your best bet for an up-to-date list of compatible smart cards is to do a Web search for "card module smart cards".

Dan Griffin is a software security consultant in Seattle, WA. He previously spent seven years at Microsoft on the Windows Security development team. Dan can be reached via www.jwsecure.com. He'd like to thank Brian McNeill and Eric Perlin at Microsoft for their feedback on this article.


Worldly Windows: Extend The Global Reach Of Your Applications With Unicode 5.0

Worldly Windows

Extend The Global Reach Of Your Applications With Unicode 5.0

Julie D. Allen and Michael S. Kaplan and Cathy Wissink
This article discusses:
  • The importance of Unicode support in an OS
  • What's new in Unicode 5.0
  • Unicode 5.0 support in Windows Vista
  • Unicode security considerations
This article uses the following technologies:
Unicode 5.0, Windows Vista
Contents
Why Unicode Matters
What's New in Unicode 5.0
Why the Change?
Conclusion

M
ost major software vendors provide a number of localized or translated products, as well as products that are capable of handling non-Latin writing systems. But there's still a long way to go. Ultimately, the goal of software internationalization is to allow the greatest number of people to communicate with each other, independent of language, writing system, or location. Simply put, people want to communicate with others, on their terms, in their own languages.

As members of the Unicode Consortium, Microsoft and other companies help the industry move closer to that goal of comprehensive language and regional support. In this article we will discuss the role Microsoft plays in this particular standards effort and, more importantly, describe the new features of Unicode 5.0 and how they are implemented in Windows Vista™.

Why Unicode Matters

Microsoft has actively participated in the Unicode Consortium for a number of reasons. First of all, consistency, stability, and interoperability of data, independent of writing system, location, and platform, are some of the key tenets of global software development. The industry needs internationalization standards, especially for encoding, that work consistently regardless of the platform they run on. These standards must also be representative of the key linguistic and cultural stakeholders they support. The Unicode Consortium develops standards with ongoing input from key stakeholders: industry, government bodies, language authorities, and related standards bodies. The existence of a single worldwide system to store, transmit, and manipulate the world's data streamlines and considerably accelerates the development and deployment of international software.

The benefits Unicode delivered to Microsoft were made clear when adoption of a single-source codebase supporting Unicode reduced the development time. For Windows® 2000, it took months to prepare the localized versions after shipping the English language product. This decreased to weeks in Windows XP.

The Unicode Consortium is also a crucial player in ensuring that the languages and cultures of the world are capable of being represented in technology, most notably through their cooperation with the Script Encoding Initiative (see unicode.org/pending/about-sei.html). This initiative is supported by Microsoft and other software vendors via their various emerging market programs. The result from Microsoft has been the Language Interface Packs (LIPs) and Enabling Language Kits (ELKs) programs, developer tools such as the Microsoft® Locale Builder and the Microsoft Keyboard Layout Creator. Also see the sidebar "ELKs and LIPs," as well as Figures 1 and 2, for a glimpse of Unicode in action.

Figure 1 Inuktitut Characters
Figure 1 Inuktitut Characters(Click the image for a smaller view)
Figure 1 Inuktitut Characters
Figure 1 Inuktitut Characters (Click the image for a larger view)
Figure 2 Malayalam Characters
Figure 2 Malayalam Characters(Click the image for a smaller view)
Figure 2 Malayalam Characters
Figure 2 Malayalam Characters (Click the image for a larger view)

What's New in Unicode 5.0

Like earlier major versions, the Unicode Standard, Version 5.0, has been released as a book (Addison-Wesley, 2006) with associated data files. The book has been updated substantially and contains pages of new and clarified text, new and improved tables and illustrations, and more. For the first time, the Unicode Standard Annexes, which contain information about topics such as the Bidirectional Algorithm, identifiers, line-breaking properties, and text boundaries, are printed in the book. The annexes have been carefully edited and rewritten to provide improved guidance to developers and implementers.

The conformance clauses and definitions in the book have been renumbered in Version 5.0, eliminating the confusion of numbers such as C12b and D30e used in Version 4.0. (Appendix D includes a handy table showing the numbers used in earlier versions.)

Unicode 5.0 has more explicit guidelines for rendering Indic scripts. For CJK characters, the book now contains the IICore radical-stroke index, focusing on the subset of roughly 10,000 CJK characters most important for the East Asian markets. This new index makes it much easier to look up important characters in the standard; the full CJK radical-stroke index is available online for looking up rare characters. Character properties have been extended and improved.

Information about security considerations and reliability, topics of great importance to implementers, has been vastly extended. Version 5.0 is synchronized with coordinated updates to other important standards, including the Unicode Collation Algorithm, the Common Locale Data Repository, and the International Standard ISO/IEC 10646. (See the sidebar "Phishing and Unicode Security.")Phishing and Unicode Security

Although the RFCs related to International Domain Names made great efforts to point out the potential problems relating to visual spoofing that could occur with every character in Unicode being used for domain names, most early adopters paid no heed to the warnings. It took the visual spoofing of Unicode characters to get the attention of the software community, even though it had been called out as a potential security leak long before in various IETF documents.

This kind of spoofing can happen when, for example, a bank Web site has the letter l in its name and a spoofer replaces it with the number l. The characters look the same, and the casual user would never know that he is typing his bank user name and password into the Web site of a criminal. And when the characters allowed in a URL include many languages/writing systems that have characters that look the same as but are different from those in other languages, the opportunity to spoof a URL increases dramatically.

Unicode Technical Report #36, Unicode Security Considerations (unicode.org/reports/tr36) and Unicode Technical Standard #39, Unicode Security Mechanisms (unicode.org/reports/tr39) by Michel Suignard of Microsoft both explain how an application or platform supporting Unicode can avoid these issues.

Version 5.0 adds new characters for Cyrillic, Greek, Hebrew, Kannada, Latin, mathematics, phonetic extensions, and symbols. It also adds several minority and historical scripts, such as Tifinagh and Phoenician. The Tifinagh script is used by more than 20 million people in Morocco and elsewhere who speak varieties of languages commonly called Berber or Amazigh. (Incidentally, the word "Tifinagh" means the Phoenician letters.) Phoenician is the script that sailed the ancient Mediterranean and the forerunner to Arabic, Greek, Hebrew, and other scripts in modern use. The book Unicode Standard, Version 5.0 is a treasure trove of intriguing details about scripts, as well as useful implementation information. It is an indispensable reference. Figures 3 and 4 show a few more character sets.

Figure 3 Khmer Characters
Figure 3 Khmer Characters(Click the image for a smaller view)
Figure 3 Khmer Characters
Figure 3 Khmer Characters (Click the image for a larger view)
Figure 4 Lao Characters
Figure 4 Lao Characters(Click the image for a smaller view)
Figure 4 Lao Characters
Figure 4 Lao Characters (Click the image for a larger view)

Why the Change?

In prior versions of Windows, Microsoft products lagged a few versions behind Unicode; for example, data that came from a number of Unicode versions (anywhere from Unicode 1.1 to 3.1) would ship in Windows XP, so that whichever feature was leveraging the data, whether it was collation, character properties, line breaking, the bidirectional algorithm, or UTF-8 conversion, the necessary Unicode version was available. This mixed approach, while expedient for a shortened development cycle, led to various problems, including:

  • Inadequate algorithmic support for operations such as UTF-8 conversions
  • Insufficient coverage for languages for which there later turned out to be a business need to support
  • Lack of good coverage of security-sensitive issues clarified in later versions

Such problems became more transparent when Microsoft started shipping the ELKs in order to support LIPs beginning with Windows XP Service Pack 2 (SP2). Several of the languages on the list of potential ELKs and LIPs had to be removed from consideration because core support for Unicode properties, casing, and collation did not exist for languages such as Mongolian and Yi. Furthermore, Microsoft learned the hard way that trying to limit the supported ranges of Unicode to those considered strategic business cases for the company often proved ineffective, since yesterday's weak business cases often become tomorrow's strategic interests.

Besides, updating to the most recent version of the Unicode standard comprehensively across Windows and the Microsoft .NET Framework will help fill some gaps in terms of expected international support in Microsoft products, in the standard itself, and with related linguistic community concerns.

The customer requirement for Unicode 5.0 support on Windows Vista spanned the core services that ISVs could not extend themselves, such as casing, property, and character type values; default collation weights; and normalization support. (We did not try to ship font and rendering support of every character in Unicode 5.0, as that would be an incredibly time-consuming and difficult task, especially given the difference in targets between Unicode and Windows. The goal was to support languages as well as they could be supported with regard to the aforementioned core operations, while leaving other extensible details like typography and keyboard support to font authors and tools like the Microsoft Keyboard Layout Creator.)

Given that version support in Windows and the .NET Framework lay somewhere between Unicode 1.1 and Unicode 3.1, describing what was updated in Unicode 5.0 for Microsoft is significantly complex. The support of new languages and scripts is perhaps the easiest part for people to understand in terms of updates to the standard. However, there are many more additional must-have features in the new version of the standard that customers request in terms of Unicode support. Specifically on Windows, the following Unicode-related data has been updated and these character additions have been included:

  • Enhanced security features in Unicode.
  • Updates for normalization and international domain names.
  • Latest version support for simple casing and other common operations.
  • Support in collation beyond the simple code point ordering for Extension B that was added to Windows XP, needed for both Japanese/JIS X 213 and Hong Kong /HKSCS, due to many new characters. This particular item is not a feature of Unicode as much as a feature of collation efforts by Microsoft; it now follows the example of the Unicode Collation Algorithm and adds the characters in the latest version of Unicode to its default table.
  • Support for mathematics in Unicode.
  • Stronger model for scripts in South Asia and Southeast Asia, improved due to implementers' input in recent years.
  • A great deal of work on the conformance model for the standard that makes Unicode as a whole more stable and consistent due to better validation of the data and properties provided by the standard.

Because extensibility of language and culture support, as well as consistent Unicode implementation, were important customer requirements for Windows Vista, it was crucial to develop an internationalization standards strategy and implementation plan that provided integration of the most recent version of the Unicode Standard across the platform.

While Microsoft is one of the early adopters of Unicode 5.0, other industry players are also already using this release or plan to upgrade in the near future. Adoption of this latest version of the standard benefits a wide spectrum of people, including developers, ISVs, and end users. All of the new Unicode features beneficial to Microsoft products apply just as significantly to others in the industry. Some obvious technical benefits include more software that leverages compatible UTF-8 conversions and more browsers that add anti-phishing capabilities.ELK and LIPs

Enabling Language Kits (ELKs) and Language Interface Packs (LIPs) were developed in response to the length of time it was taking to enable new languages for Windows.

ELKs allow installation on a per-locale basis of all the components that a locale needs to be supported, including new locale information, new keyboards, new sorts, new fonts, and new shaping engines. And all of this work is done without waiting for a major system release.

LIPs have a very different goal. They represent a localization framework to help mitigate the expense and time required to localize Windows into another language. Each LIP is a lightweight localization that translates the most commonly seen strings, enforcing the 80/20 rule that localizes the 20 percent of strings seen 80 percent of the time.

Conclusion

For Microsoft, the business case for upgrading to Unicode 5.0 was clear and compelling. Other organizations interested in supporting the Unicode effort should consider the many benefits of joining the Unicode Consortium and being actively involved in important projects. Upgrading to Unicode 5.0 provides an even more comprehensive set of characters for languages, but perhaps even more significantly, it makes for a safer and more functional Internet for everyone around the world, including those who never venture far beyond the world of ASCII.

Julie D. Allen is Senior Editor and Project Manager with the Unicode Consortium. She has been involved with the Unicode Standard since Version 2.1. She has a Ph.D. in Germanics from the University of Washington, and honed her writing skills with an earlier degree in English literature.

Michael S. Kaplan is a Technical Lead at Microsoft, centering on Collation, Keyboards, and Locales. He was the principal developer for both the Microsoft Layer for Unicode on Windows 9x and the Microsoft Keyboard Layout Creator. He is the author of Internationalization with Visual Basic (Sams Publishing, 2000). His blog gets new posts about internationalization at blogs.msdn.com/michkap.

Cathy Wissink is a Group Program Manager at Microsoft, working in the Global Platform and Technology Services group. Cathy was a linguist intern on the Windows NT 3.1 team in 1991. Many Windows projects later, she joined Microsoft in 2000. She holds an MA and a PhC (Candidate of Philosophy Certificate) from the University of Washington in Germanic Linguistics, and an MEd in Adult Education and Training from Seattle University.


Debug Leaky Apps: Identify And Prevent Memory Leaks In Managed Code

Debug Leaky Apps

Identify And Prevent Memory Leaks In Managed Code

James Kovacs
This article discusses:
  • Understanding memory leaks in managed apps
  • Unmanaged memory used in .NET apps
  • Helping the .NET garbage collector do its job
This article uses the following technologies:
.NET Framework
Code download available at: MemoryLeaks2007_01.exe (163KB)
Contents
Memory in .NET Applications
Checking for Leaks
Leaking Stack Memory
Leaking Unmanaged Heap Memory
"Leaking" Managed Heap Memory
Conclusion

T
he first reaction many developers have to the idea of memory leaks in managed code is that it's not possible. After all, the garbage collector (GC) takes care of all memory management, right? The garbage collector only handles managed memory, though. There are a number of places where unmanaged memory is used in Microsoft® .NET Framework-based applications, either by the common language runtime (CLR) itself, or explicitly by the programmer when interoperating with unmanaged code. There are also occasions where the GC seems to be shirking its duties and not efficiently handling managed memory. Usually this is caused by subtle (or not so subtle) programming errors that hinder the GC from performing its job. As good memory citizens, we still have to profile our applications to ensure they are leak-free and make efficient use of the memory they require.

Memory in .NET Applications

As you probably know, .NET applications make use of several types of memory: the stack, the unmanaged heap, and the managed heap. Here's a little refresher.

The Stack The stack is where local variables, method parameters, return values, and other temporary values are stored during the execution of an application. A stack is allocated on a per-thread basis and serves as a scratch area for the thread to perform its work. The GC is not responsible for cleaning up the stack because the space on the stack reserved for a method call is automatically cleaned up when a method returns. Note, however, that the GC is aware of references to objects stored on the stack. When an object is instantiated in a method, its reference (a 32-bit or 64-bit integer depending on the platform) is kept on the stack, but the object itself is stored on the managed heap and is collected by the garbage collector once the variable has gone out of scope.

The Unmanaged Heap The unmanaged heap is used for runtime data structures, method tables, Microsoft intermediate language (MSIL), JITed code, and so forth. Unmanaged code will allocate objects on the unmanaged heap or stack depending on how the object is instantiated. Managed code can allocate unmanaged heap memory directly by calling into unmanaged Win32® APIs or by instantiating COM objects. The CLR itself uses the unmanaged heap extensively for its data structures and code.

The Managed Heap The managed heap is where managed objects are allocated and it is the domain of the garbage collector. The CLR uses a generational, compacting GC. The GC is generational in that it ages objects as they survive garbage collections; this is a performance enhancement. All versions of the .NET Framework have used three generations, Gen0, Gen1, and Gen2 (from youngest to oldest). The GC is compacting in that it relocates objects on the managed heap to eliminate holes and keep free memory contiguous. Moving large objects is expensive and therefore the GC allocates them on a separate Large Object Heap, which does not compact. For more information on the managed heap and GC, see Jeffrey Richter's two part series, "Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework" and "Garbage Collection-Part 2: Automatic Memory Management in the Microsoft .NET Framework". Although the articles were written based on the .NET Framework 1.0, the core concepts have not changed in versions 1.1 or 2.0, although the .NET GC has improved since then.

Checking for Leaks

There are a number of telltale signs that an application is leaking memory. Maybe it's throwing an OutOfMemoryException. Maybe its responsiveness is growing very sluggish because it started swapping virtual memory to disk. Maybe memory use is gradually (or not so gradually) increasing in Task Manager. When a memory leak is suspected, you must first determine what kind of memory is leaking, as that will allow you to focus your debugging efforts in the correct area. Use PerfMon to examine the following performance counters for the application: Process/Private Bytes, .NET CLR Memory/# Bytes in All Heaps, and .NET CLR LocksAndThreads/# of current logical Threads. The Process/Private Bytes counter reports all memory that is exclusively allocated for a process and can't be shared with other processes on the system. The .NET CLR Memory/# Bytes in All Heaps counter reports the combined total size of the Gen0, Gen1, Gen2, and large object heaps. The .NET CLR LocksAndThreads/# of current logical Threads counter reports the number of logical threads in an AppDomain. If an application's logical thread count is increasing unexpectedly, thread stacks are leaking. If Private Bytes is increasing, but # Bytes in All Heaps remains stable, unmanaged memory is leaking. If both counters are increasing, memory in the managed heaps is building up.

Leaking Stack Memory

Although it is possible to run out of stack space, which results in a StackOverflowException in the managed world, any stack space used during a method call is reclaimed once that method returns. Therefore, there are only two real ways to leak stack space. The first is to have a method call that consumes significant stack resources and that never returns, thereby never releasing the associated stack frame. The other is by leaking a thread, and thus that thread's entire stack. If an application creates worker threads for performing background work, but neglects to terminate them properly, thread stacks can be leaked. By default, the stack size on modern desktop and server versions of Windows® is 1MB. So if an application's Process/Private Bytes is periodically jumping in 1MB increments with a corresponding increase in .NET CLR LocksAndThreads/# of current logical Threads, a thread stack leak is very likely the culprit. Figure 1 shows one example of improper thread cleanup caused by (purposely bad) multithreaded logic.

Figure 1 Buggy Thread Cleanup (Close Figure 1)
using System;
using System.Threading;

namespace MsdnMag.ThreadForker {
  class Program {
    static void Main() {
      while(true) {
        Console.WriteLine(
          "Press <ENTER> to fork another thread...");
        Console.ReadLine();
        Thread t = new Thread(new ThreadStart(ThreadProc));
        t.Start();
      }
    }

    static void ThreadProc() {
      Console.WriteLine("Thread #{0} started...",
        Thread.CurrentThread.ManagedThreadId);
      // Block until current thread terminates - i.e. wait forever
      Thread.CurrentThread.Join();
    }
  }
}

A thread is launched, which displays its thread ID and then tries to Join on itself. Join causes the calling thread to block waiting on the other thread to terminate. So the thread is caught in a chicken-or-egg scenario-the thread is waiting for itself to terminate. Watch this program under Task Manager to see its memory usage increase by 1MB, the size of a thread stack, every time <Enter> is pressed.

The reference to the Thread object is being dropped every time through the loop, but the GC does not reclaim the memory allocated for the thread stack. A managed thread's lifetime is independent of the Thread object that creates it, a very good thing given that you wouldn't want the GC to terminate a thread that was still doing work simply because you lost all references to the associated Thread object. So the GC is collecting the Thread object, but not the actual managed thread. The managed thread does not exit (and the memory for its thread stack is not released) until its ThreadProc returns or it is explicitly killed. So if a managed thread is not properly terminated, the memory allocated to its thread stack will leak.

Leaking Unmanaged Heap Memory

If total memory use is increasing, but logical thread count and managed heap memory is not increasing, there is a leak in the unmanaged heap. We will examine some common causes for leaks in the unmanaged heap, including interoperating with unmanaged code, aborted finalizers, and assembly leaks.

Interoperating with Unmanaged Code One source of memory leaks involves interoperating with unmanaged code, such as when C-style DLLs are used through P/Invoke and COM objects through COM interop. The GC is unaware of unmanaged memory, and thus a leak here is due to a programming error in the managed code using the unmanaged memory. If an app is interoperating with unmanaged code, step through the code and examine memory usage before and after the unmanaged call to verify that memory is being reclaimed properly. If it isn't, look for the leak in the unmanaged component using traditional debugging techniques.

Aborted Finalizers A very insidious leak occurs when an object's finalizer does not get called, and it contains code to clean up unmanaged memory allocated by the object. Under normal conditions, finalizers will get called, but the CLR does not make any guarantees. While this may change in the future, current versions of the CLR use only one finalizer thread. Consider a misbehaving finalizer trying to log information to a database that is offline. If that misbehaving finalizer erroneously tries over and over again to access the database, never returning, the "well-behaved" finalizer will never get a chance to run. This problem can manifest itself very sporadically because it depends on the order of finalizers on the finalization queue as well as the behavior of other finalizers.

When an AppDomain is torn down, the CLR will attempt to clear the finalizer queue by running all finalizers. A stalled finalizer can prevent the CLR from completing the AppDomain tear down. To account for this, the CLR implements a timeout on this process, after which it will stop the finalization process. Typically, this isn't the end of the world, as most applications only have one AppDomain, and its teardown is due to the process being shut down. When an OS process is shut down, its resources will be recovered by the operating system. Unfortunately, in a hosting situation such as ASP.NET or SQL Server™, the teardown of the AppDomain doesn't mean the teardown of the hosting process. Another AppDomain can be spun up in the same process. Any unmanaged memory that was leaked by a component because its finalizer didn't run will still be sitting around unreferenced, unreachable, and taking up space. This can be disastrous as more and more memory is leaked over time.

In .NET 1.x, the only solution was to tear down the process and start again. The .NET Framework 2.0 introduces critical finalizers, which indicate that a finalizer will be cleaning up unmanaged resources and must be given a chance to run during AppDomain teardown. See Stephen Toub's article, "Keep Your Code Running with the Reliability Features of the .NET Framework" for more information.

Assembly Leaks Assembly leaks are relatively common and are caused by the fact that once an assembly is loaded, it can't be unloaded until the AppDomain is unloaded. In most cases, this is not a problem unless assemblies are being dynamically generated and loaded. Let's now look at dynamic code generation leaks, and specifically XmlSerializer leaks, in more detail.

Dynamic Code Generation Leaks Sometimes code needs to be generated dynamically. Maybe the application has a macro scripting interface for extensibility similar to Microsoft Office. Maybe a bond-pricing engine needs to load the pricing rules dynamically so end users can create their own bond types. Maybe the application is a dynamic language runtime/compiler for Python. In many cases, it is desirable to compile the macros, pricing rules, or code to MSIL for performance reasons. System.CodeDom can be used to generate MSIL on the fly.

The code in Figure 2 dynamically generates an assembly in memory. It can be called repeatedly without a problem. Unfortunately if the macro, pricing rule, or code changes, the dynamic assembly must be regenerated. The old assembly will no longer be used, but there is no way to evict it from memory, short of unloading the AppDomain in which the assembly was loaded. The unmanaged heap memory, which is used for its code, JITed methods, and other runtime data structures, has been leaked. (Managed memory has also been leaked in the form of any static fields on the dynamically generated classes.) There is no magic formula to detect this problem. If you're dynamically generating MSIL using System.CodeDom, check whether you regenerate code. If you do, you're leaking unmanaged heap memory.

Figure 2 Dynamically Generating an Assembly in Memory (Close Figure 2)
CodeCompileUnit program = new CodeCompileUnit();
CodeNamespace ns = new
  CodeNamespace("MsdnMag.MemoryLeaks.CodeGen.CodeDomGenerated");
ns.Imports.Add(new CodeNamespaceImport("System"));
program.Namespaces.Add(ns);

CodeTypeDeclaration class1 = new CodeTypeDeclaration("CodeDomHello");
ns.Types.Add(class1);
CodeEntryPointMethod start = new CodeEntryPointMethod();
start.ReturnType = new CodeTypeReference(typeof(void));
CodeMethodInvokeExpression cs1 = new CodeMethodInvokeExpression(
  new CodeTypeReferenceExpression("System.Console"), "WriteLine",
    new CodePrimitiveExpression("Hello, World!"));
start.Statements.Add(cs1);
class1.Members.Add(start);

CSharpCodeProvider provider = new CSharpCodeProvider();
CompilerResults results = provider.CompileAssemblyFromDom(
  new CompilerParameters(), program);

There are two main techniques for solving this problem. The first is to load the dynamically generated MSIL into a child AppDomain. The child AppDomain can be unloaded when the generated code changes and a new one spun up to host the updated MSIL. This technique works on all versions of the .NET Framework.

Another technique introduced in .NET Framework 2.0 is lightweight code generation, also known as dynamic methods. Using a DynamicMethod, MSIL op codes are explicitly emitted to define the method body, and then the DynamicMethod is invoked either directly via DynamicMethod.Invoke or via a suitable delegate.

DynamicMethod dm = new DynamicMethod("tempMethod" +
  Guid.NewGuid().ToString(), null, null, this.GetType());
ILGenerator il = dm.GetILGenerator();

il.Emit(OpCodes.Ldstr, "Hello, World!");
MethodInfo cw = typeof(Console).GetMethod("WriteLine",
  new Type[] { typeof(string) });
il.Emit(OpCodes.Call, cw);

dm.Invoke(null, null);

The main advantage of dynamic methods is that the MSIL and all related code generation data structures are allocated on the managed heap. This means that the GC can reclaim the memory once the last reference to the DynamicMethod goes out of scope.

XmlSerializer Leaks Portions of the .NET Framework, such as the XmlSerializer, use dynamic code generation internally. Consider the following typical XmlSerializer code:

XmlSerializer serializer = new XmlSerializer(typeof(Person));
serializer.Serialize(outputStream, person);

The XmlSerializer constructor will generate a pair of classes derived from XmlSerializationReader and XmlSerializationWriter by analyzing the Person class using reflection. It will create temporary C# files, compile the resulting files into a temporary assembly, and finally load that assembly into the process. Code gen like this is also relatively expensive. So the XmlSerializer caches the temporary assemblies on a per-type basis. This means that the next time an XmlSerializer for the Person class is created, the cached assembly is used rather than a new one generated.

By default, the XmlElement name used by the XmlSerializer is the name of the class. Thus, Person would be serialized as:

<?xml version="1.0" encoding="utf-8"?>
<Person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <Id>5d49c002-089d-4445-ac4a-acb8519e62c9</Id>
 <FirstName>John</FirstName>
 <LastName>Doe</LastName>
</Person>

Sometimes it is necessary to change the root element name without changing the class name. (The root element name might be required for compatibility with an existing schema.) So Person may have to be serialized as <PersonInstance>. Conveniently, there is an overload of the XmlSerializer constructor that takes the root element name as its second parameter, like this:

XmlSerializer serializer = new XmlSerializer(typeof(Person),
  new XmlRootAttribute("PersonInstance"));

When the app starts serializing/deserializing Person objects, everything works until an OutOfMemoryException is thrown. This overload of the XmlSerializer constructor does not cache the dynamically generated assembly, but generates a new temporary assembly every time you instantiate a new XmlSerializer! The app is leaking unmanaged memory in the form of temporary assemblies.

To fix the leak, use the XmlRootAttribute on the class to change the root element name of the serialized type:

[XmlRoot("PersonInstance")]
public class Person {
  // code
}

If the attribute is applied directly to the type, the XmlSerializer caches the generated assemblies for the type and there is no leak. If root element names need to be dynamically switched, the application can perform the caching of the XmlSerializer instances itself by using a factory to retrieve them:

XmlSerializer serializer = XmlSerializerFactory.Create(
  typeof(Person), "PersonInstance");

XmlSerializerFactory is a class I created that checks whether a Dictionary<TKey, TValue> contains an XmlSerializer for Person using the PersonInstance root element name. If it does, the instance is returned. If not, a new one is created, stored in the hash table, and returned to the caller.

"Leaking" Managed Heap Memory

Now let's turn our attention to "leaking" managed memory. When dealing with managed memory, the GC takes care of most of the work for us. We do need to provide the GC with the information it needs to do its job. However, there are a number of scenarios that prevent the GC from doing its job efficiently and result in higher managed memory use than would otherwise be required. These situations include large object heap fragmentation, unneeded rooted references, and a midlife crisis.

Large Object Heap Fragmentation If an object is 85,000 bytes or larger, it is allocated on the large object heap. Note that this is the size of the object itself and not any children. Take the following class as an example:

public class Foo {
  private byte[] m_buffer = new byte[90000]; // large object heap
}

Foo instances would be allocated on the normal generational managed heap as it only contains a 4-byte (32-bit Framework) or 8-byte (64-bit Framework) reference to the buffer, plus some other housekeeping data used by the .NET Framework. The buffer would be allocated on the large object heap.

Unlike the rest of the managed heap, the Large Object Heap is not compacted due to the cost of moving the large objects. So as large objects are allocated, freed, and cleaned up, gaps will appear. Depending on usage patterns, the gaps in the large object heap can result in significantly more memory usage than is required by the currently allocated large objects. The LOHFragmentation application that is included in this month's download demonstrates this by randomly allocating and freeing byte arrays in the Large Object Heap. Some runs of the application result in the newly created byte arrays fitting nicely into the gaps left by freed byte arrays. On other runs of the application, this is not the case and the memory required is much larger than the memory required for the currently allocated byte arrays. To visualize fragmentation of the large object heap, use a memory profiler, such as the CLRProfiler. The red regions in Figure 3 are allocated byte arrays whereas white regions are unallocated space.

Figure 3 The Large Object Heap in CLRProfiler
Figure 3 The Large Object Heap in CLRProfiler(Click the image for a smaller view)
Figure 3 The Large Object Heap in CLRProfiler
Figure 3 The Large Object Heap in CLRProfiler (Click the image for a larger view)

There is no single solution for avoiding Large Object Heap fragmentation. Examine how the application uses memory and specifically the types of objects that are on the large object heap using tools like the CLRProfiler. If the fragmentation is due to re-allocating buffers, maintain a fixed set of buffers that are reused. If the fragmentation is being caused by concatenation of large numbers of strings, examine whether the System.Text.StringBuilder class can reduce the number of temporary strings created. The basic strategy is to determine how to reduce the application's reliance on temporary large objects, which are causing the gaps in the large object heap.

Unneeded Rooted References Let's consider how the GC determines when it can reclaim memory. When the CLR attempts to allocate memory and has insufficient memory in reserve, it performs a garbage collection. The GC enumerates all rooted references, including static fields and in-scope local variables on any thread's call stack. It marks these references as reachable and follows any references these objects contain, marking them as reachable as well. It continues this process until it has visited all reachable references. Any unmarked objects are not reachable and hence are garbage. The GC compacts the managed heap, tidies up references to point to their new location in the heap, and returns control to the CLR. If sufficient memory has been freed, the allocation proceeds using this freed memory. If not, additional memory is requested from the operating system.

If we forget to null out rooted references, the GC is prevented from efficiently freeing memory as quickly as possible, resulting in a larger memory footprint for the application. The problem can be subtle, such as a method that creates a large graph of temporary objects before making a remote call like a database query or call to a Web service. If a garbage collection happens during the remote call, the entire graph is marked reachable and is not collected. This becomes even more costly because objects surviving a collection are promoted to the next generation, which can lead to a midlife crisis.

Midlife Crisis A midlife crisis does not cause an application to go out and buy a Porsche. It can, however, cause an overuse of managed heap memory and excessive amounts of processor time spent in the GC. As mentioned previously, the GC uses a generational algorithm, which is predicated on the heuristic that if an object has lived a while, it will probably live a while longer. For example, in a Windows Forms application, the main form is created when the application starts and the application exits when the main form closes. It is wasteful for the GC to continually verify that the main form is being referenced. When the system requires memory to satisfy an allocation request, it first performs a Gen0 collection. If sufficient memory is not available, a Gen1 collection is performed. If the allocation request still can't be satisfied, a Gen2 collection is performed, which involves an expensive sweep of the entire managed heap. Gen0 collections are relatively inexpensive because only recently allocated objects are considered for collection.

A midlife crisis occurs when objects tend to live until Gen1 (or worse, Gen2), but die shortly thereafter. This has the effect of turning cheap Gen0 collections into much more expensive Gen1 (or Gen2) collections. How can this occur? Take a look at the following code:

class Foo {
  ~Foo() { }
}

This object will always be reclaimed in a Gen 1 collection! The finalizer, ~Foo(), allows us to implement cleanup code for our objects that, barring a rude AppDomain abort, will run before the object's memory is freed. The GC's job is to free up as much managed memory as possible as quickly as possible. Finalizers are user-written code and can do absolutely anything. Although not recommended, a finalizer could do something silly such as logging to a database or calling Thread.Sleep(int.MaxValue). So when the GC finds an unreferenced object with a finalizer, it places the object on the finalization queue and moves on. The object has survived a garbage collection and hence is promoted a generation. There is even a performance counter for this: .NET CLR Memory-Finalization Survivors, which is the number of objects during the last garbage collection that survived due to a finalizer. Eventually the finalizer thread will run the object's finalizer and it can subsequently be collected. But you have turned a cheap Gen0 collection into a Gen1 collection, all by simply adding a finalizer!

In most cases, finalizers are not necessary when writing managed code. They are only needed when a managed object holds a reference to an unmanaged resource that needs cleanup, and even then you should use a SafeHandle-derived type to wrap the unmanaged resource rather than implementing a finalizer. Additionally, if you're using unmanaged resources or other managed types that implement IDisposable, implement the Dispose pattern to allow users of the object to aggressively clean up the resources and avoid any related finalization.

If an object only holds references to other managed objects, the GC will clean up unreferenced objects. This is in stark contrast to C++, where delete must be called on child objects. If a finalizer is empty or simply nulling out references to child objects, remove it. It is hurting performance by needlessly promoting the object to an older generation, making them more expensive to clean up.

There are other ways to get into a midlife crisis, such as holding onto objects before making a blocking call like querying a database, blocking on another thread, or calling a Web service. During the call, one or more collections can occur and result in cheap Gen0 objects being promoted to a later generation, again resulting in much higher memory usage and collection costs.

There is an even more subtle case that occurs with event handlers and callbacks. I will use ASP.NET as an example, but the same type of problem can occur in any application. Consider performing an expensive query and wanting to cache the results for 5 minutes. The query is page-specific and based on query-string parameters. To monitor caching behavior, an event handler logs when an item is removed from the cache (see Figure 4).

Figure 4 Logging Items Removed from Cache (Close Figure 4)
protected void Page_Load(object sender, EventArgs e) {
  string cacheKey = buildCacheKey(Request.Url, Request.QueryString);
  object cachedObject = Cache.Get(cacheKey);
  if(cachedObject == null) {
    cachedObject = someExpensiveQuery();
    Cache.Add(cacheKey, cachedObject, null,
      Cache.NoAbsoluteExpiration,
      TimeSpan.FromMinutes(5), CacheItemPriority.Default,
      new CacheItemRemovedCallback(OnCacheItemRemoved));
  }
  ... // Continue with normal page processing
}

private void OnCacheItemRemoved(string key, object value,
                CacheItemRemovedReason reason) {
  ... // Do some logging here
}

This innocuous-looking code contains a major problem. All of these ASP.NET Page instances just became long-lived objects. The OnCacheItemRemoved is an instance method and the CacheItemRemovedCallback delegate contains an implicit this pointer, where this is the Page instance. The delegate is added to the Cache object. So there now exists a dependency from the Cache to the delegate to the Page instance. When a garbage collection occurs, the Page instance remains reachable from a rooted reference, the Cache object. The Page instance (and all the temporary objects it created while rendering) will now have to wait for at least five minutes before being collected, during which time they will likely be promoted to Gen2. Fortunately this example has a simple solution. Make the callback function static. The dependency on the Page instance is broken and it can now be collected cheaply as a Gen0 object.

Conclusion

I have discussed a variety of problems in .NET applications that can lead to memory leaks or overconsumption of memory. Although .NET reduces the need for you to be concerned with memory, you still must pay attention to your application's use of memory to ensure that it is well-behaved and efficient. Just because an application is managed doesn't mean you can throw good software engineering practices out the window and count on the GC to perform magic. You must continue to monitor your application's memory performance counters during the development and testing process. But it's worth it. Remember, a well-behaved application means happy customers.

James Kovacs is an independent architect, developer, trainer, and jack-of-all-trades living in Calgary, Alberta specializing in the .NET Framework, security, and enterprise application development. He is a Microsoft MVP for Solutions Architecture and received his Masters degree from Harvard University. James can be reached at jkovacs@post.harvard.edu or www.jameskovacs.com.


Editor's Note: A New Year—A New Vista

Editor's Note
A New Year—A New Vista

As we start a brand new year here at MSDN Magazine (our 22nd for those scoring at home), we have been the recipients of some great fortune. The Windows Vista team has chosen to time their launch to coincide with this issue, so we've got a bumper crop of OS coverage this month.

First, we'll look at a significant security feature for Windows Vista users, User Account Control. The days where you could assume your users were administrators are gone, and Chris Corio will show you what this means for developers.

Windows Vista and Office 2007 support a new type of add-in called a preview handler. Stephen Toub, our technical editor, lets you in on the secrets of writing your own.

Jeffrey Richter, one of the best-known MSDN Magazine columnists, shows you a thing or two about Windows SideShow. SideShow is a technology that works with secondary displays on laptops. Just like a cell phone that lets you see who's calling without opening up the phone, with SideShow you'll be able to view vital information—schedules, e-mail, and notes—without turning on or opening your laptop.

We also take a look at what's new in Unicode 5.0 and how it's supported in Windows Vista. Are you cool enough to support Phoenecian characters in your app? You know you are.

Our tour of Windows Vista for programmers also makes a stop at the new Credential Provider model. Credential Provider, which replaces GINA, will make it easier to implement new user authentication scenarios.

Of course, we aren't just giving you a couple of Windows Vista features and calling it a day. To start this year off right, we're adding three new columns that will make you wonder how you were able to live without them.

Charles Petzold, the guru of Windows programming, is adding his name to a new column about the .NET Framework 3.0. As Charles writes about Windows Presentation Foundation, he'll share columnist duties with Juval Lowy, a prominent voice in the Windows Communication Foundation community, and Matt Milner, who will wax eloquent on Windows Workflow Foundation. Together, they will discuss—what else?—Foundations.

Brian A. Randell, who is a prominent voice in just about every community you could name, will peel back the curtain on Visual Studio Team System in his new regular column, which has the catchy name "Team System."

There's more good news—now we're global. Do you know how to say "security" in seven languages? Our Web site does. Years ago, translating an article was an arduous task. You'd go to the article you wanted to read in another language, wait until the page loaded, and then wait for years for us to put an automatic translation tool onto the site. Starting with the November 2006 issue, we've streamlined this process by removing the wait. You'll be able to read all our articles in your favorite language. Whether you're looking for information on seguranca, sécurité, seguridad, Sicherheit, безопасность, or 安全性, our Web site is the place to go. (That's Brazilian Portuguese, French, Spanish, German, Russian, and Simplified Chinese, by the way.)

This new translation paradigm is an amazing piece of work. It was designed and implemented by our Web lord, Mike Richter, along with Brad McCabe's team in Redmond, and our technical editors for MSDN Magazine and TechNet Magazine, Stephen Toub and Joshua Hoffman, respectively.

We've got other great stuff online for you as well, like our new official MSDN Magazine blog. Updated regularly, this is a prime spot to hang out, learn about the mag and Microsoft development, and just bask in reflected glory. You can find the blog at blogs.msdn.com/msdnmagazine. Happy surfing! —J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Snesha Arumugam, Mason Bendixen, Matt Gibbs, Ben Gillis, Chris Guzak, Buck Hodges, Kerem Karatal, Aidan Low, Brian McNeil, Kevin Moore, Eric Perlin, Mark Novak, Rob Relyea, Jon Rosenberg, and Ravinder Vuppula.

Aero, Active Directory, ActiveX, IntelliSense, Internet Explorer, Microsoft, MSDN, Outlook, PowerPoint, Visual Basic, Visual C++, Visual C#, Visual J#, Visual Studio, Windows, Win32, Windows Live, Windows Media, Windows NT, Windows Server, Windows SideShow, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. Msdn magazine disclaims any liability whatsoever for any use of such code or other information. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Technology LLC. CMP Technology LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Save Coding Time, Manage Compressed Files, and More

Toolbox
Save Coding Time, Manage Compressed Files, and More
Scott Mitchell

Contents
Save Coding Time
Manage Compressed Files
Inspect HTTP Traffic
The Bookshelf

Save Coding Time

While Visual Studio offers an abundance of built-in productivity tools, a number of third-party products can further increase your proficiency. CodeSMART G2 for Visual Studio by AxTools offers dozens of windows, tools, wizards, and dialogs that can dramatically reduce time spent on repetitive coding tasks.

CodeSMART adds several Explorer windows to Visual Studio®, all of which can be docked, pinned, or floated like any other Visual Studio window. The Code Explorer window combines the best aspects of the Visual Studio Solution Explorer and Class View windows, allowing you to view the files in your project and drill down into the classes and members comprising each file. Furthermore, each method in the Code Explorer includes a Code Flow folder that lists (and nests) the method's conditional statements and control flow constructs. Double-clicking any node in the Code Explorer whisks you to the appropriate location and adds the entry to the History pane for quick access.

CodeSMART Adds Explorer Windows
CodeSMART Adds Explorer Windows (Click the image for a smaller view)
CodeSMART Adds Explorer Windows
CodeSMART Adds Explorer Windows (Click the image for a larger view)

On the design side, CodeSMART's Designer Explorer lists the controls in the currently opened design window, along with the properties of the selected control. When adding a new control to the designer, CodeSmith can be configured to automatically pop up a dialog to set the Text and Name properties.

The Explorer windows and design-time dialogs are great time savers, but CodeSMART really shines with its AutoText and SmartComplete features. Much like Microsoft® Word can auto-complete a word or common phrases from the first few letters, the AutoText feature maps short, simple key combinations into lengthy, often-used code blocks. For example, just type "tc", then space, and-voilà-a complete try...catch block appears with the cursor positioned inside the try. SmartComplete is similar to IntelliSense®, showing language keywords, statements, and so on, but can be customized to include user-defined items. For Visual Studio 2005 users, AutoText and SmartComplete are like Code Snippets; however, AutoText and SmartComplete's rules are easier to create and modify.

CodeSMART offers other code building dialogs, assorted code formatting and documentation tools, and program statistics reports. If you spend too much time writing repetitive code, CodeSMART can help you spend less.

Price: $189 per single-user license.

www.axtools.com

Manage Compressed Files

The ZIP file format has been a popular compression format since its introduction in 1989, offering an easy way to archive and compress files. There are a number of third-party .NET Framework-based components available for working with ZIP files, including ZipForge.NET version 1.0 by ComponentAce.

ZipForge.NET provides a straightforward API for creating, encrypting, searching, and extracting ZIP files. With just four lines of code, you can create a new ZIP file, populating it with files or content from a byte array or stream. Extracting the archive's contents can be accomplished just as easily.

The ZipForge.NET API includes a number of Find methods useful for searching the contents of an archive. These Find methods can be used to enumerate the entire contents of a ZIP file, or to work with only a subset of the files (such as all JPG files).

Performing more complex ZIP-related functions is just as easy. Need to password-protect an archive? Just set the Password property. When working with large archives, event handlers for the OnFileProgress and OnOverallProgress events can be used to monitor the progress of the operation.

Price: Free for the Personal Edition; 149 Euros (around $125) for the single developer, royalty -free Commercial Edition with source code.

www.componentace.com

Inspect HTTP Traffic

Debugging Web applications is often a difficult process due to the logical, physical, and temporal separations among the server- and client-side technologies. Bugs can arise from any of the server-side tiers- the backend database, the application tier, the code in the dynamic Web pages-or can be due to the HTML and JavaScript sent to the browser, or in the serialization or transfer of data between any of these different layers.

In the past, virtually all bugs sprang from an error in code on the server. Today, however, Web pages are becoming increasingly complex, carrying dozens of inline and external stylesheet and script definitions.

HttpWatch Records Traffic
HttpWatch Records Traffic (Click the image for a smaller view)
HttpWatch Records Traffic
HttpWatch Records Traffic (Click the image for a larger view)

When hunting down a bug in a Web application, it helps to start from the client and inspect what markup and script has been received. It's also important to look at what data is being sent back to the server. While you can always use View|Source to examine the received markup, a tool like HttpWatch 4.x from Simtec Limited greatly simplifies and hastens the process.

Once installed, HttpWatch can be launched directly from Microsoft® Internet Explorer® through a toolbar icon. When it is activated, HttpWatch records all HTTP traffic, providing both the raw data and a variety of summary reports for your perusal.

It is amazing how many HTTP requests occur when visiting a typical Web page. There's the request for the Web page itself, of course, plus requests for the images on the page, and requests for external cascading style sheets and script resources as well. In short, the amount of data can be overwhelming. To combat this, HttpWatch provides filters that can be used to limit or highlight the results by content type, URL, HTTP status code, HTTP headers, and so on.

Along with the raw HTTP traffic, HttpWatch also provides numerous summary tabs that show a selected request's headers, cookies, querystring, POST data, and so on. In addition, many of these summary reports can be searched or exported.

Price: $249 per single-user license.

www.httpwatch.com

The Bookshelf

For developers programming against large, high-level frameworks, mastery of the development environment is increasingly essential. Like many .NET developers, I spend hours inside of Visual Studio each and every day. The more I learn about Visual Studio options, shortcuts, windows, toolbars, and features, the more productive I become and the less time I spend fighting with the IDE.

One book I've found particularly helpful in demystifying the Visual Studio environment is Microsoft Visual Studio 2005 Unleashed (Sams, 2006). In this 870-page tome, Lars Powers and Mike Snell meticulously explore the multitude of menus, configuration settings, project types, and features that comprise Visual Studio 2005.

(Click the image for a smaller view)

The book is divided into four parts. The first part provides a quick tour of Visual Studio, while the second part delves into the environment in much greater detail. Part 2 includes chapters on productivity aids such as change tracking, code outlining, smart tags and smart tasks, IntelliSense, and the task list. Other chapters explore myriad debugging options, working with the Visual Studio automation object model, and writing macros, add-ins, and wizards.

Part 3 shows how to accomplish various tasks in Visual Studio. These chapters are useful if you need to discover how to perform a very specific task, such as adding a stored procedure to a database project or creating a custom control for a Windows® Forms application. The text doesn't include any discussion on programming practices or explanations of the code-it just shows you how to complete the task using Visual Studio.

The final part looks at Visual Studio 2005 Team System, exploring common tasks such as using source control, work item tracking, modeling, testing, and so on. If you don't use Visual Studio 2005 Team System, don't discount this book, as there's still a wealth of information in the middle two parts that make it a worthwhile resource.

If you spend countless hours of your day in Visual Studio, consider picking up a copy of Microsoft Visual Studio 2005 Unleashed. It will likely become a permanent fixture on your desk and your first stop for finding answers to your Visual Studio questions.

Price: $59.99.

www.samspublishing.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.

All prices were confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


CLR Inside Out: Introduction to COM Interop

CLR Inside Out
Introduction to COM Interop
Thottam R. Sriram

Code download available at: CLRInsideOut2007_01.exe (1480KB)
Contents
A Simple ATL COM Server
Creating a COM Client
Converting a COM DLL Using Tlbimp
Writing a Managed Client
Marshalling a Structure and a String
Using P/Invoke from Managed Code
Calling Managed Code Using Interfaces
Debugging Managed and Unmanaged Code
Wrap-Up

COM is a wonderful technology. One aspect of the common language runtime (CLR) that makes it an extremely powerful platform is that it allows seamless interactions between Microsoft® .NET applications and unmanaged COM components. However, when I searched the Web, I found few working samples demonstrating the very basic concepts of COM interop. The purpose of this column is to illustrate those basic concepts in order to provide solid working examples that can jump-start users in this technology.

I'll start off with a simple Active Template Library (ATL) COM server and try accessing methods in this server using an unmanaged COM client, then do the same thing with a managed client. I'll walk through the various DLLs to illustrate the translation from unmanaged to managed, and I'll show how to access an exported method in an unmanaged DLL using P/Invoke. The toughest part of this is to figure out marshaling of complex structures, which I don't cover exhaustively in this column-it would be a complete column (or book) on its own. I'll show you how unmanaged code can call back into managed code using interfaces. (You could do this with delegates as well, but I won't cover that in this column.)

Finally, I'll discuss debugging your COM interop project using public symbols. This will give you very basic introduction to WinDbg.exe, unmanaged debugging, and managed debugging using SOS. I'll demonstrate the stack from managed to unmanaged as calls are made in either direction.

A Simple ATL COM Server

Let's start by writing a simple COM server. The client will host the server in-process and execute methods in the server.

To make development easy, I'll use ATL COM. Create a directory called COMInteropSample, then create a new Visual C++® ATL project and give it a name such as MSDNCOMServer. Unselect the Create a New Directory option, but use the defaults for the rest.

Now add a new Class/Interface to the solution. Open Solution Explorer if you don't already see it. Right-click on MSDNCOMServer and select Add | Class. Select ATL and ATL Simple Object and then click Add. Enter the short name of the class as MyCOMServer, then use the defaults for the rest. Click Finish to add the class.

Add a new method to the interface created for you. Go to Class View, right-click the IMyCOMServer interface, and select Add | Add Method. Give the method a name such as MyCOMServerMethod and click Finish.

Now implement the method in the class. Select MyCOMServer.cpp and view the source. You'll see the method that was added by the wizard in the interface in the cpp file. In the TODO section add the following code:

wprintf_s(L"Inside MyCOMServerMethod");
Compile the code to generate the server DLL.

You have now completed your ATL COM server, which implements the interface IMyCOMServer in the class CMyCOMServer and writes a message to the default output stream. The GUID for the interface is defined in the IDL file as an attribute on the interface definition as shown in Figure 1.

Figure 1 IMyCOMServer Definition (Close Figure 1)
[
    object,
    uuid(45FA03A3-FD24-41EB-921C-15204CAF68AE),
    nonextensible,
    helpstring("IMyCOMServer Interface"),
    pointer_default(unique)
]
interface IMyCOMServer : IUnknown {
    [helpstring("method MyCOMServerMethod")]
    HRESULT(MyCOMServerMethod)( void);
};

[
    uuid(3BA5DF7B-F8EF-4EDE-A7B5-5E7D13764ACC),
    version(1.0),
    helpstring("MSDNCOMServer 1.0 Type Library")
]
library MSDNCOMServerLib
{
    importlib("stdole2.tlb");
    [
        uuid(2D6D2821-A7FB-4F99-A61A-2286A47CD4D1),
        helpstring("MyCOMServer Class")
    ]
    coclass MyCOMServer
    {
        [default] interface IMyCOMServer;
    };
};

Creating a COM Client

Let's try to access the COM server now using a native (unmanaged) client. Again, for simplicity, we'll use Visual Studio® and its wizards. Create a new, empty Visual C++ project. Select Source Files from Solution Explorer and add a new C++ file. Give it a name such as COMClient.cpp, then paste the code from Figure 2 into the file.

Figure 2 COMClient.cpp (Close Figure 2)
#include "MSDNCOMServer.h"
#include "wchar.h"
#include "MSDNCOMServer_i.c"

void main()
{
    // Initialize COM
    IMyCOMServer *pUnk = NULL;
    CoInitialize(NULL);

    // Create the com object
    HRESULT hr = CoCreateInstance(CLSID_MyCOMServer,
        NULL, CLSCTX_ALL, IID_IMyCOMServer, (void **)&pUnk);
    if(S_OK != hr)
    {
        wprintf(L"Error creating COM object ... %d", hr);
        return;
    }

    // Call a method on the COM object
    pUnk->MyCOMServerMethod();

    // Free resources and uninitialize COM
    pUnk->Release();
    CoUninitialize();
    return;
}

Add the header files from the server (MSDNCOMServer.h and MSDNCOMServer_i.c), then compile and run the client. You should see "Inside MyCOMServerMethod" printed to the console. This message is coming from the COM server.

Hurray! You've created a COM server and compiled it to a DLL, created a COM client, and called the COM server methods from the client.

Converting a COM DLL Using Tlbimp

Now that we have a COM DLL, let's see how to access it from a managed client. Open a Visual Studio command prompt and go to the directory where you created the COM DLL. Now run the following command:

tlbimp MSDNCOMServer.dll

Tlbimp.exe is the Type Library Importer tool included with the .NET Framework SDK. This command outputs a managed DLL called MSDNCOMServerLIB.dll that serves as a managed wrapper for the unmanaged COM DLL.

Now that we have the unmanaged client running, let's look closely into the original unmanaged COM DLL and the managed DLL generated from Tlbimp.exe. Figure 3 shows a dump from Oleview of the original COM DLL.

Figure 3 Oleview Dump from MSDNCOMServer.dll (Close Figure 3)
// Generated .IDL file (by the OLE/COM Object Viewer)
// typelib filename: MSDNCOMServer.dll
[
    uuid(3BA5DF7B-F8EF-4EDE-A7B5-5E7D13764ACC),
    version(1.0),
    helpstring("MSDNCOMServer 1.0 Type Library"),
    custom(DE77BA64-517C-11D1-A2DA-0000F8773CE9, 100663662),
    custom(DE77BA63-517C-11D1-A2DA-0000F8773CE9, 1158351043),
    custom(DE77BA65-517C-11D1-A2DA-0000F8773CE9, Created by MIDL version
           6.00.0366 at Fri Sep 15 13:10:42 2006)
]
library MSDNCOMServerLib
{
    // TLib : OLE Automation : {00020430-0000-0000-C000-000000000046}
    importlib("stdole2.tlb");

    // Forward declare all types defined in this typelib
    interface IMyCOMServer;

    [
      uuid(2D6D2821-A7FB-4F99-A61A-2286A47CD4D1),
      helpstring("MyCOMServer Class")
    ]
    coclass MyCOMServer {
        [default] interface IMyCOMServer;
    };

    [
      odl,
      uuid(45FA03A3-FD24-41EB-921C-15204CAF68AE),
      helpstring("IMyCOMServer Interface"),
      nonextensible
    ]
    interface IMyCOMServer : IUnknown {
        [helpstring("method MyCOMServerMethod")]
        HRESULT_stdcall MyCOMServerMethod();
    };

};

If you use a .NET decompiler, such as Lutz Roeder's .NET Reflector, and view the managed library output by Tlbimp, you'll see something like Figure 4.

Figure 4 Managed MSDNCOMServerLib (Close Figure 4)
namespace MSDNCOMServerLib
{
    [ComImport, Guid("45FA03A3-FD24-41EB-921C-15204CAF68AE"),
     InterfaceType(1), TypeLibType(0x80)]
    public interface IMyCOMServer
    {
        [MethodImpl(MethodImplOptions.InternalCall,
         MethodCodeType=MethodCodeType.Runtime)]
        void MyCOMServerMethod();
    }

    [ComImport, CoClass(typeof(MyCOMServerClass)),
     Guid("45FA03A3-FD24-41EB-921C-15204CAF68AE")]
    public interface MyCOMServer : IMyCOMServer
    {
    }

    [ComImport, TypeLibType(2),
     Guid("2D6D2821-A7FB-4F99-A61A-2286A47CD4D1"),
     ClassInterface(0)]
    public class MyCOMServerClass : IMyCOMServer, MyCOMServer
    {
        // Methods
        [MethodImpl(MethodImplOptions.InternalCall,
         MethodCodeType=MethodCodeType.Runtime)]
        public virtual extern void MyCOMServerMethod();
    }
}

A lot has happened. First, notice that the library MSDNCOMServerLib is translated to namespace MSDNCOMServerLib. Also, the interface IMyCOMServer is retained with the attributed GUID, and the CoClass MyCOMServer is translated to a public interface MyCOMServer that derives from IMyCOMServer, and a class MyCOMServerClass that implements the interfaces IMyCOMServer and MyCOMServer.

Writing a Managed Client

So far you've seen the process of writing a COM server and an unmanaged COM client, as well as the translation that takes place when the unmanaged DLL is converted to a managed DLL using Tlbimp. Let's consume the library exported by Tlbimp in managed code in order to write a managed COM client:

using System;
using MSDNCOMServerLib;

class Test
{
    static void Main()
    {
        MyCOMServerClass comServer = new MyCOMServerClass();
        comServer.MyCOMServerMethod();
    }
}

The managed code is amazingly simple and compact. I saved this code in a file called ManagedClient.cs and compiled it using the following command:

csc ManagedClient.cs /reference:MSDNCOMServerLib.dll

If you run the executable ManagedClient.exe, you should get the same result we got from the unmanaged client.

Marshalling a Structure and a String

Now let's add some parameters to the server-a String and a struct. This will require modifications to both the server itself and to the clients. In order to accomplish this, you need to start by defining the method on the server.

First, edit the IDL file to include the code that is shown in Figure 5. I added a definition for a struct, and I modified the declaration for MyCOMServerMethod to accept two parameters, a string and a struct.

Figure 5 Modifying MyCOMServerMethod (Close Figure 5)
struct MyTestStruct
{
    int nIndex;
    [string] WCHAR* sName;
    int nValue;
};

[
    object,
    uuid(45FA03A3-FD24-41EB-921C-15204CAF68AE),
    nonextensible,
    helpstring("IMyCOMServer Interface"),
    pointer_default(unique)
]
interface IMyCOMServer : IUnknown{
    [helpstring("method MyCOMServerMethod")]
    HRESULT(MyCOMServerMethod)(
        BSTR *sTestString, struct MyTestStruct *pMyTestStruct);
};

Next, edit the header file on the server and add the following prototype of the method as defined in the IDL.

class ATL_NO_VTABLE CMyCOMServer :
    public CComObjectRootEx<CComSingleThreadModel>,
    public CComCoClass<CMyCOMServer, &CLSID_MyCOMServer>,
    public IMyCOMServer
{
public:
    STDMETHOD(MyCOMServerMethod)(
        BSTR *sTestString, struct MyTestStruct *pMyTestStruct);
};

Modify the implementation of the method to reflect the change in the prototype and also to add more code to manipulate the strings that are passed to it. The final code is shown in Figure 6. This completes the changes to the server, which I'll now recompile.

Figure 6 Modified CMyCOMServer (Close Figure 6)
// MyCOMServer.cpp : Implementation of CMyCOMServer
#include "stdafx.h"
#include "MyCOMServer.h"

// CMyCOMServer
STDMETHODIMP CMyCOMServer::MyCOMServerMethod(
    BSTR *sString, struct MyTestStruct *pStruct)
{
    if((NULL == sString) || (NULL == pStruct) || (NULL ==
        pStruct->sName))
    {
        wprintf_s(L"Invalid input\r\n");
        return E_FAIL;
    }

    // Print the input to the COM Method
    wprintf_s(
        L"Inside MyCOMServerMethod: Param-1 = %s Param-2 = %s\r\n",
        *sString, pStruct->sName);

    // Modifying the input string
    SysFreeString(*sString);
    *sString = SysAllocString(L"Changed String from COM Server");
    if(NULL == *sString)
    {
        wprintf_s(L"Allocation failed ...\r\n");
        return E_FAIL;
    }

    // Modifying the string in the structure
    CoTaskMemFree(pStruct->sName);
    pStruct->sName = reinterpret_cast<WCHAR *>
        (CoTaskMemAlloc(sizeof
             (L"Changed Structure from COM Server")+10));
    if(NULL == pStruct->sName)
    {
        wprintf_s(L"Allocation failed ...\r\n");
        return E_FAIL;
    }

    size_t size = 0;
    size = wcslen(L"Changed Structure from COM Server");
    wcscpy_s(pStruct->sName, size+1,
        L"Changed Structure from COM Server");

    //Print the changed strings in COM Server
    wprintf_s(
        L"Exiting MyCOMServerMethod: Param-1 = %s Param-2 = %s\r\n",
        *sString, pStruct->sName);
    return S_OK;
}

In order to access the new COM method on the managed client, I copy over the DLL to the same directory where I have my managed client. I rerun the following command to get the new MSDNCOMServerLib.dll which now has the new method in it.

tlbimp MSDNCOMServer.dll

You can view the new method signature and implementation in Ildasm, another tool included with the .NET Framework SDK. When you view it, you'll notice that the marshaler has added the code shown in Figure 7 to the call.

Figure 7 COM Server Marshaling Code (Close Figure 7)
namespace MSDNCOMServerLib
{
    [ComImport, Guid("45FA03A3-FD24-41EB-921C-15204CAF68AE"),
     InterfaceType(1), TypeLibType(0x80)]
    public interface IMyCOMServer
    {
        [MethodImpl(MethodImplOptions.InternalCall,
         MethodCodeType=MethodCodeType.Runtime)]
        void MyCOMServerMethod(
            [MarshalAs(UnmanagedType.BStr)] ref string sTestString,
            ref MyTestStruct pMyTestStruct);
    }

    [ComImport, CoClass(typeof(MyCOMServerClass)),
     Guid("45FA03A3-FD24-41EB-921C-15204CAF68AE")]
    public interface MyCOMServer : IMyCOMServer { }

    [ComImport, TypeLibType(2),
     Guid("2D6D2821-A7FB-4F99-A61A-2286A47CD4D1"),
     ClassInterface(0)]
    public class MyCOMServerClass : IMyCOMServer, MyCOMServer
    {
        // Methods
        [MethodImpl(MethodImplOptions.InternalCall,
         MethodCodeType=MethodCodeType.Runtime)]
        public virtual extern void MyCOMServerMethod(
            [MarshalAs(UnmanagedType.BStr)] ref string sTestString,
            ref MyTestStruct pMyTestStruct);
    }

    [StructLayout(LayoutKind.Sequential, Pack=4)]
    public struct MyTestStruct
    {
        public int nIndex;
        [MarshalAs(UnmanagedType.LPWStr)]
        public string sName;
        public int nValue;
    }
}

The input ref string is marshaled as In, OUT BSTR and the struct remains the same, though the string within the structure is marshaled as LPWStr.

Now that the server is ready along with the managed DLL from the Tlbimp set, I can move on to modifying the managed client to pass a string and a struct and check the returned value to see if it is changed. Figure 8 shows the modified client code.

Figure 8 Passing Strings and Structs (Close Figure 8)
using System;
using MSDNCOMServerLib;
using System.Runtime.InteropServices;

class Test
{
    [STAThread]
    static void Main()
    {
        // Calling COM methods from managed Client
        String param1 = "Original String from Managed Client";

        MyTestStruct param2 = new MyTestStruct();
        param2.sName = "Original Structure from Managed Client";

        MyCOMServerClass comServer = new MyCOMServerClass();
        comServer.MyCOMServerMethod(ref param1, ref param2);

        Console.WriteLine("Param1: {0} Param2: {1}\r\n\r\n",
            param1, param2.sName);
    }
}

The managed client is still sleek and simple. It creates the string and an instance of the struct that was imported from the unmanaged DLL. Now compile the above code the same way you did before, then run the executable. You should see the following output:

Inside MsyCOMServerMethod: Param-1 = Original String from Managed Client
Param-2 = Original Structure from Managed Client
Exiting MyCOMServerMethod: Param-1 = Changed String from COM Server
Param-2 = Changed Structure from COM Server
Param1: Changed String from COM Server Param2: Changed Structure from COM Server

The call from Main in managed code goes over to the unmanaged server and prints the input that is passed. The server modifies the strings and prints it on the server. To verify if the modified string is passed back to the client, we print the string in the client as well.

To create an unmanaged function (non-COM) in the same COM DLL, I first have to define the method and implement it in the unmanaged DLL. Select MSDNCOMServer.cpp and paste the code below at the very end of that file.

STDAPI MyNonCOMMethod()
{
    wprintf_s(L"Inside MyNonCOMMethod\r\n");
    return true;
}

You need to export the method from the DLL in order to be able to access it. Do that by adding an entry to this function in the COMServer.def file:

; COMServer.def : Declares the module parameters.

LIBRARY      "COMServer.DLL"

EXPORTS
    DllCanUnloadNow       PRIVATE
    DllGetClassObject     PRIVATE
    DllRegisterServer     PRIVATE
    DllUnregisterServer   PRIVATE
    MyNonCOMMethod        PRIVATE

Compile the server and generate the new DLL with both the COM method and the exported function in it.

Copy the DLL to the directory that contains the managed client and run Tlbimp on it. You can view the exported function available in the managed DLL using Ildasm.

Using P/Invoke from Managed Code

Next we'll modify the managed client to call the unmanaged function using P/Invoke. To do this, you have to define the method in managed code and then make the call. The managed method declaration mirrors the unmanaged declaration and tells the CLR where it can expect to find the actual implementation. Figure 9 shows the changes to the code.

Figure 9 Using P/Invoke (Close Figure 9)
class Test
{
    [DllImport("MSDNCOMServer.dll")]
    static extern bool MyNonCOMMethod();

    [STAThread]
    static void Main()
    {
        // Calling COM methods from managed Client
        ...

        // Calling unmanaged function using pInvoke
        MyNonCOMMethod();
    }
}

We've added a DllImport statement and a call to MyNonCOMMethod to our original managed client. When you compile and run the client, you should see output like this:

Inside MyCOMServerMethod: Param-1 = Original String from Managed Client
Param-2 = Original Structure from Managed Client
Exiting MyCOMServerMethod: Param-1 = Changed String from COM Server
Param-2 = Changed Structure from COM Server
Param1: Changed String from COM Server Param2: Changed Structure from COM Server

Inside MyNonCOMMethod

The call to the MyNonCOMMethod did get executed and prints the correct statement.

Calling Managed Code Using Interfaces

Finally, let's make a managed call from the unmanaged function using reverse P/Invoke. This is not as common a practice, but it's very useful. You can pass a managed interface or a delegate to unmanaged code, and the unmanaged code can invoke the delegate or methods on the interface. Let's get the managed interface going first so it can be invoked from the unmanaged code. To do this, declare the interface in ManagedClient.cs as shown in Figure 10.

Figure 10 Declaring the Managed Interface (Close Figure 10)
using System;
using MSDNCOMServerLib;
using System.Runtime.InteropServices;

[Guid("7DAC8207-D3AE-4c75-9B67-92801A497D44"),
 InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[ComVisible(true)]
public interface IReversePInvoke
{
    void ManagedMethod(ref int val);
}

class ManagedClass : IReversePInvoke
{
    public void ManagedMethod(ref int nVal)
    {
        Console.WriteLine("ManagedMethod: " + nVal);
        nVal = 20;
        Console.WriteLine("ManagedMethod Value changed to: " + nVal);
    }
}

[DllImport("MSDNCOMServer.dll")]
static extern bool MyNonCOMMethod(IReversePInvoke ptr);

[STAThread]
static void Main()
{
    ...
    ManagedClass man = new ManagedClass();
    MyNonCOMMethod(man);
}

In this code I added a declaration for IReversePInvoke, a class that implements the interface, and a modified DLLImport method in the unmanaged DLL to take an interface pointer. I also modified the place where the method is called to pass the ManagedClass instance to MyNonCOMMethod.

You can compile the client now, but it won't run successfully as you haven't yet modified the server to take the interface as a parameter. So run Tlbexp (another tool from the .NET Framework SDK that produces a type library from a managed assembly) on the managed executable to get a .tlb file. Open it in Oleview, copy the prototype of the interface, and paste it into the server IDL file as follows:

[
    odl,
    uuid(7DAC8207-D3AE-4C75-9B67-92801A497D44)
]
interface IReversePInvoke : IUnknown {
    HRESULT _stdcall ManagedMethod([in, out] long* val);
};

Now you have the interface on the server. You have to change the signature of the method and add the call to invoke the managed method on the interface. Here is how we do this:

STDAPI MyNonCOMMethod(IReversePInvoke *ptr)
{
    int param = 10;

    wprintf_s(L"Inside MyNonCOMMethod\r\n");
    HRESULT hr = ptr->ManagedMethod(&param);
    wprintf_s(L"Value in MyNonCOMMethod = %d\r\n", param);
    wprintf_s(L"Exiting MyNonCOMMethod\r\n");

    return true;
}

Now you have the unmanaged function taking the managed interface and invoking a method on the managed interface. Compile this to a DLL, run Tlbimp on it to get an updated managed interop DLL, and compile the managed code against the managed DLL from Tlbimp. Here's the output you get:

Inside MyCOMServerMethod: Param-1 = Original String from Managed Client
Param-2 = Original Structure from Managed Client
Exiting MyCOMServerMethod: Param-1 = Changed String from COM Server
Param-2 = Changed Structure from COM Server
Param1: Changed String from COM Server Param2: Changed Structure from
COM Server

Inside MyNonCOMMethod
ManagedMethod: 10
ManagedMethod Value changed to: 20
Value in MyNonCOMMethod = 20
Exiting MyNonCOMMethod

The call did get back to managed code and printed "10" as expected. The managed method modifies the value to 20 and that is printed in the unmanaged function.

Debugging Managed and Unmanaged Code

Hopefully all of your code has worked successfully. Obviously, though, it's rare to write 100 percent bug-free code. At some point you'll need to know how to debug your code that uses interop.

To successfully debug, make sure your computer has full symbols loaded. Symbols allow a debugger to better understand and interpret what's happening in your code, and Microsoft provides a public server to which debuggers can connect in order to access these symbols for Microsoft-supplied DLLs. To configure the windbg debugger to use this public symbol server, first run windbg:

windbg managedclient.exe

Then use the following command syntax to point it at the symbol server:

.SymFix
.sympath+ http://msdl.microsoft.com/download/symbols
.Reload

Next, you need to tell windbg to set breakpoints for when the mscorwks, COMServerLib, and COMServer DLLs load. This is accomplished using the sxe ld command:

sxe ld mscorwks
sxe ld COMServerLib
sxe ld COMServer

To start the app running under the debugger, type g (for "go") and hit return. During the run of the application, it will break into the debugger due to the loading of the COMServer DLL, at which point we can set another breakpoint on the MyCOMServerMethod (you can then use the bl command to verify that the breakpoint has in fact been set):

bp COMServer!CMyCOMServer::MyCOMServerMethod (wchar_t **,
    struct MyTestStruct *)

Now that mscorwks has been loaded, we want windbg to load the SOS debugger extensions that allow for much more robust debugger interaction with managed applications:

.loadby sos mscorwks

To set a breakpoint on the ManagedMethod method that will be called from the COM server through reverse P/Invoke, we use SOS's !bpmd command:

!bpmd ManagedClient ManagedClass.ManagedMethod

With this breakpoint set, we can again use the g command to proceed. At some point, we'll hit the unmanaged breakpoint on MyCOMServerMethod and then we can view the current stack trace using the windbg kL command, as shown in Figure 11 (which also demonstrates using additional windbg commands to examine the values of variables).

Figure 11 Getting a Stack Trace and Examining Memory (Close Figure 11)
0:000> kL
ChildEBP RetAddr
0012f328 79f21268 MSDNCOMServer!CMyCOMServer::MyCOMServerMethod
0012f3fc 0090a28e mscorwks!CLRToCOMWorker+0x196
0012f438 00f800e5 CLRStub[StubLinkStub]@90a28e
01271c08 79e88f63 ManagedClient!Test.Main()+0x75
0012f490 79e88ee4 mscorwks!CallDescrWorker+0x33
0012f510 79e88e31 mscorwks!CallDescrWorkerWithHandler+0xa3
0012f650 79e88d19 mscorwks!MethodDesc::CallDescr+0x19c
0012f668 79e88cf6 mscorwks!MethodDesc::CallTargetWorker+0x20
0012f67c 79f084b0 mscorwks!MethodDescCallSite::Call_RetArgSlot+0x18
0012f7e0 79f082a9 mscorwks!ClassLoader::RunMain+0x220
0012fa48 79f0817e mscorwks!Assembly::ExecuteMainMethod+0xa6
0012ff18 79f07dc7 mscorwks!SystemDomain::ExecuteMainMethod+0x398
0012ff68 79f05f61 mscorwks!ExecuteEXE+0x59
0012ffb0 79011b5f mscorwks!_CorExeMain+0x11b
0012ffc0 7c816fd7 mscoree!_CorExeMain+0x2c
0012fff0 00000000 KERNEL32!BaseProcessStart+0x23

0:000> dv
           this = 0x00fc2fb8
        sString = 0x0012f364
        pStruct = 0x00195ef8
           size = 0x12f330

0:000> dt sString
Local var @ 0x12f334 Type wchar_t**
0x0012f364
 -> 0x001a61e4  "Original String from Managed Client"

0:000> dt -r pStruct
Local var @ 0x12f338 Type MyTestStruct*
0x00195ef8
   +0x000 nIndex   : 0
   +0x004 sName    : 0x001a4f38  "Original Structure from Managed Client"
   +0x008 nValue   : 0

If we continue with g again, we'll break at the call to ManagedMethod, at which point we can use the SOS command !clrstack to examine the managed stack:

0:000> !clrstack
OS Thread Id: 0x87c (0)
ESP       EIP
0012ef80 00f80168 ManagedClass.ManagedMethod(Int32 ByRef)
0012f148 79f1ef33 [GCFrame: 0012f148]
0012f2a0 79f1ef33 [ComMethodFrame: 0012f2a0]
0012f454 79f1ef33 [NDirectMethodFrameSlim: 0012f454] Test.MyNonCOMMethod(IReversePInvoke)
0012f464 00f80119 Test.Main()
0012f69c 79e88f63 [GCFrame: 0012f69c]

Wrap-Up

For additional information, start with PInvoke.net, a community-driven source. Adam Nathan's COM interop tome, .NET and COM: The Complete Interoperability Guide, is back in print (and his blog is at "Adam Nathan's Blog"). General information about debugging, including the WinDbg download and reference, is available, and you can find a reference on the SOS Debugging Extension (SOS.dll) here.

Send your questions and comments to clrinout@microsoft.com.

Thottam R. Sriram has a Masters Degree in Computer Science from Concordia University, Montreal Canada. He is currently a Program Manager in the CLR team working on COM interop. Thottam worked with the Windows team before joining the CLR team. Thanks to COM interop team members Mason Bendixen, Claudio Caldato, Alessandro Catorcini, Ben Gillis, and Snesha Arumugam for their help with this column.


Data Points: Column Expressions, DataRelations, and Computations

Data Points
Column Expressions, DataRelations, and Computations
John Papa

Code download available at: DataPoints2007_01.exe (191KB)

ADO.NET does more than simply allow you to retrieve and save data to and from a database. It provides myriad features that allow you to manipulate data both for business logic and for display purposes in real-world applications. As demonstrated by the examples in this column, ADO.NET is not just a tool for data access, it can is also be used for data manipulation.

In this month's column, I will concentrate on some of the questions I am asked most often regarding data manipulation with ADO.NET. I'll discuss how you can use expression-based columns and DataRelations to achieve certain goals. I'll also explain how some other features, such as the DataTable's Compute method and the SetOrdinal method, can be used to address common business needs.

Q I want to display order information individually and in rolled-up groups (by day, by month, and by year) using DataTables. I need to allow the order information to be edited, and the totals in the parent DataTables must reflect the changes immediately. I want to show each of the sets of data in separate grids on the form. After I load the data in a DataTable, how do I create the grouped information with the sum of the orders for each group using ADO.NET DataColumn expressions?

A Calculated fields can be created easily by either using a calculated expression in a SQL statement or creating an expression-bound DataColumn. There are differences in how these two approaches operate and the features they expose.

For your scenario and requirements, calculated columns in a SQL statement is not the most efficient solution. A good solution is to add expression-based DataColumns to a series of DataTables.

Before diving too deep into this problem, let's take a look at the end result we're striving for (see Figure 1). This sample shows five grids on a form. The topmost grid contains the individual order line items. The price, quantity, and discount of each order line can be edited in this grid. When these values are edited, the extended price is calculated for that row and the values of the totals in the other four grids automatically recalculate and display the updated total to reflect the changes. The second grid contains a row for each order, including a calculated column that shows the sum of the extended price for that order. The third, fourth, and fifth grids show the sum of each order by day, month, and year, respectively.

Figure 1 Order Total by Day, Month, and Year
Figure 1 Order Total by Day, Month, and Year(Click the image for a smaller view)
Figure 1 Order Total by Day, Month, and Year
Figure 1 Order Total by Day, Month, and Year (Click the image for a larger view)

Each grid is bound to a different DataTable. The first DataTable is loaded with the order lines from the database and has an expression-based column appended to it. After the data is loaded into the DataTable using the DataAdapter's Fill method, a column named ExtendedPrice of type decimal is added to the DataTable. This column is defined as an expression column that calculates its value based on the values of the UnitPrice, Quantity, and Discount for each respective DataRow:

ds.Tables["OrderLines"].Columns.Add("ExtendedPrice",
    typeof(decimal), "(UnitPrice * Quantity) * (1 - Discount)");

The OrderLines DataTable contains the core data for this form. Once it is loaded and its expression column created, the first grouping DataTable can be filled and added to the DataSet (see Figure 2). The SQL statement grabs the OrderID and OrderDate for each order and then the DataTable is filled with the results. Before adding the OrderTotal column to this DataTable, a DataRelation must first be created so the expression in this Orders DataTable can access columns in the OrderLines DataTable.

Figure 2 An Expression Column that Sums (Close Figure 2)
using(SqlCommand cmd = new SqlCommand("SELECT o.OrderID, o.OrderDate " +
    "FROM Orders o WHERE YEAR(o.OrderDate) = @year " +
    "ORDER BY o.OrderDate DESC", cn))
{
    cmd.CommandType = CommandType.Text;
    cmd.Parameters.AddWithValue("@year", year);
    SqlDataAdapter adpt = new SqlDataAdapter(cmd);
    DataTable orders = new DataTable("Orders");
    adpt.Fill(orders);
}
ds.Tables.Add(orders);
ds.Relations.Add("Orders2OrderLines", ds.Tables["Orders"].Columns["OrderID"], ds.Tables["OrderLines"].Columns["OrderID"]);
orders.Columns.Add("OrderTotal", typeof(decimal),
    "Sum(Child.ExtendedPrice)");

The DataRelation is set up between the OrderID columns of the two DataTables. Then the OrderTotal expression column can be created and added to the Orders DataTable:

Sum(Child.ExtendedPrice)

Working first inside and then outside the parenthesis, the Child operator tells the OrderTotal DataColumn to navigate down to the child table (the OrderLines DataTable) and get the ExtendedPrice column. Again using the DataRelation, the Sum operator sums the values of the ExtendedPrice columns for the rows in the OrderLines DataTable whose OrderID values match the OrderID of the parent DataTable. The end result is that the total amount for each order is displayed in the OrderTotal column of the Orders DataTable.

Keep in mind that if there is more than one DataRelation, the syntax needs to be altered to indicate the specific DataRelation. The example from Figure 2 would be altered to look like this:

Sum(Child(Orders2OrderLines).ExtendedPrice)

The same technique can be used to fill the OrdersByDay DataTable, which is loaded with a distinct list of the order dates from the database. Then a DataRelation is established between this DataTable and its OrderDate column and the OrderDate column of the Orders DataTable. This allows the OrdersByDay DataTable to roll up the order totals into an OrderTotalByDay column using expressions, as shown here:

ds.Relations.Add("OrdersByDay2Orders", ds.Tables["OrdersByDay"].Columns["OrderDate"],
 ds.Tables["Orders"].Columns["OrderDate"]);

dayOrders.Columns.Add("OrderTotalByDay", typeof(decimal),
    "Sum(Child.OrderTotal)");

The final two grids are bound to DataTables that are loaded in the same fashion. This technique could be expanded to show other calculations. For example, you could easily show a grid that contains the order totals for a specific customer as long as the customer information was retrieved in the base table. Using a DataView, you could also filter the grid on an expression column.

Q I have a parent DataTable (Orders) and a child DataTable (Order Lines). The parent DataTable has an expression column for OrderTotal. I need to display some columns from the parent table in the child table for each row. How can I grab values from a parent

A Expression-based columns can use the Child operator to access columns in a child DataTable. However, they can also use the Parent operator to access columns in a parent DataTable. For example, to access the value of a column in the parent table, you can modify the code from the previous example to add the following code snippet:

ds.Tables["OrderLines"].Columns.Add("OrderTotalForOrderLinesTable",
    typeof(decimal), "Parent.OrderTotal");

This code adds an expression column to the child table, named OrderLines, that grabs the OrderTotal value from the Orders DataTable. This new column must be created after the child and parent DataTables have been created, the DataRelation between the two has been established, and the OrderTotal column has been created. This is a common technique to use when creating a DataView on a child table that will be bound to a grid.

When you're using the Parent operator with expression columns, you are duplicating the parent data to the child DataTable. Since the tables are synchronized, if someone changes the value in the parent DataTable, then the value in the child table will also display the change.

Q I have a DataGridView that is bound to a DataTable's DefaultView. The source of the DataTable is a stored procedure that retrieves data from my application's database. All of the fields are returned in a specific order, but I need to display these fields in a different order in my DataGridView. How can I easily reorder the columns in the DataGridView?

A Sometimes you will find it is necessary to change the order of columns in a DataTable when it isn't practical to change the sequence of the fields in the SQL statement. The DataColumn exposes a SetOrdinal method that allows you to alter the sequence of the columns in a DataTable.

The ordinal positions are 0-based, so to move the first column to be the fourth displayed, you would use the following code:

myDataTable.Columns[0].SetOrdinal(3);

If you prefer, you can reference the column by its name rather than its ordinal position:

myDataTable.Columns["OrderID"].SetOrdinal(3);

The columns before the third ordinal position are all decremented one slot to make room for the change of positioning.

Q I have a screen that displays a list of customers. The user can select a specific customer from this list and all of the customer data is displayed. The user can then select another customer from the list and view that customer's information. I can either load all of the customer data into a DataTable once and thus only hit the database one time, or I can grab just the list of customers and their IDs and then get the customer information from the database once the user selects a customer. Which of these methods is better?

A This is a common question. Should you take one big hit or several little ones? If you grab a list of customers and all of their information from the database and store the results in memory in a DataSet, you are retrieving a lot of information that may never be used. However, you are potentially reducing trips to the database. For example, if there are 1,000 customers in the customer table and you grab 40 columns for each customer, 40,000 values are retrieved and stored in the DataTable. The list of customers, in comparison, only requires a display value (for example, CompanyName) and an ID (for example, CustomerID). The rest of the columns are only shown on screen if the user selects a specific customer from the customer list.

The other option you've described is to load only the list of customers with the CompanyName and CustomerID. Then when the user selects a customer, the application makes another call to the database to retrieve detailed information for the specific customer. This technique loads significantly less data up front, but hits the database more often.

Which solution is preferable depends on additional factors. For example, if your customer data is so small that it can be gathered very quickly with no performance hit under load tests, then gathering all data up front may be a good choice. However, if the data changes quite often, this solution has a major flaw in that the loaded data will become stale very quickly. Therefore, before choosing to load all of the data up front into memory, you should consider the time it takes to load the data, how much of a memory footprint this data will consume, how much data you are actually loading, and whether the data is likely to change in the database while it is loaded (causing your on screen data to become obsolete). I generally prefer to use this technique only for smaller sets of data that do not change behind the scenes.

The second option is to grab each customer's detailed information only when the user requests it. With this technique, the list will generally load faster since it's loading less data. It will also provide fresher data since the only stagnant data is the list of customers. The downside to this technique is that the application hits the database whenever the user selects a customer. Thus, the calls to get the customer details should be optimized since you are getting customer data using a key field (CustomerID). If these calls to retrieve the customer data are causing a noticeable delay, network bandwidth, the query itself, or issues with passing data through the architecture of your application could be the cause of the slowdown. If you decide on this technique and then experience a slowdown, you'll have to determine the origin of the slowdown and if it can be resolved.

In the end, there's no simple answer for these types of decisions. The answer is almost always that it depends on the situation. In general, though, I find that it is better to load the list with minimal data and go back for additional data if the user selects a specific entry. Remember, performance isn't the only issue here. You also have to consider whether the data is fresh or stale.

Q I need to present a DataGridView loaded with sales information bound to a DataTable. My application must allow the user to perform calculations on the sales data in the grid and to choose which rows and columns to include in the calculation. How can I allow calculations on the DataTable without hitting the database again?

A In this case, the Compute method will do the trick nicely. While expression-based columns operate on a single row at a time, the DataTable's Compute method allows you to perform operations on a set of rows, given a filter and an expression. The filter is used to qualify which rows the Compute method should operate on while the expression indicates the aggregate function expression to perform. For example, you could add a column to the DataTable that calculates the average order total for customers in the USA. This is a great tool to perform calculations on a set of rows quickly without hitting the database again.

Figure 3 shows a sample form that displays a list of orders, along with each order's total. This sample contains all the basic elements that the expression and the filter require. On the left in the Filter groupbox, the user can select the field to filter, the operator to apply to the field, and the criteria for the filter. The list of fields is filled by binding to the list of DataColumns:

DataColumn[] colList = new DataColumn[orders.Columns.Count];
orders.Columns.CopyTo(colList, 0);
ddlFilterColumn.DataSource = colList;
ddlFilterColumn.DisplayMember = "ColumnName";

Figure 3 Calculating with the Compute Method
Figure 3 Calculating with the Compute Method(Click the image for a smaller view)
Figure 3 Calculating with the Compute Method
Figure 3 Calculating with the Compute Method (Click the image for a larger view)

This code creates a DataColumn array, fills it with the columns from the DataTable, and binds it to the ComboBox. By binding the array of DataColumns to the ComboBox, this provides access to other properties of the selected DataColumn that can be used. For example, if the selected column is a string then the value must be wrapped within quotation marks. Since the DataColumn is bound to the ComboBox, the code can figure out whether it should wrap the value in quotes.

The Calculation groupbox contains both a list of aggregate functions and a list of the columns on which the selected aggregate function should operate. The list of columns is bound to the ddlCalculationColumn ComboBox in the same way the ddlFilterColumn ComboBox is bound. The list of aggregate functions is loaded with the seven functions available to the Compute method: Sum, Count, Avg, Min, Max, StDev, and Var.

When the button is clicked, the code shown in Figure 4 executes. It starts by determining the type of the selected column for the filter. If the column is a string or a date, then the criteria value is wrapped with quotation marks. Once the filter and the expression have been formed, they are passed to the Compute method and the result is sent to the textbox.

Figure 4 Performing the Calculation (Close Figure 4)
if (!ValidateControls()) return;

Type colType = ((DataColumn) (ddlFilterColumn.SelectedItem)).DataType;
string filterCriteria = string.Empty;
if (ddlFilterOperator.Text == "Like")
{
    filterCriteria = string.Format("'{0}%'", txtFilterCriteria.Text);
}
else
{
    filterCriteria = (colType.Name == "String" ||
                      colType.Name == "DateTime") ?
        string.Format("'{0}'", txtFilterCriteria.Text) :
                               txtFilterCriteria.Text;
}
string filter = string.Format("{0} {1} {2}", ddlFilterColumn.Text,
    ddlFilterOperator.Text, filterCriteria);
string expression = string.Format("{0}({1})", ddlAggregate.Text,
    ddlCalculationColumn.Text);

object value = this.orders.Compute(expression, filter);
txtResult.Text = value.ToString();

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences, such as VSLive, or blogging at codebetter.com/blogs/john.papa.


Cutting Edge: The Server Side of ASP.NET Pages

Cutting Edge
The Server Side of ASP.NET Pages
Dino Esposito

Code download available at: CuttingEdge2007_01.exe (276KB)
Contents
What's in the Temporary ASP.NET Folder?
Preservation Files
Page Class Dynamic Source Code
The Runtime Public API
Building the Explorer Tool
Examining Sample Pages
Wrap-Up

As an ASP.NET developer, you probably have a pretty good idea about how ASP.NET handles code in an .aspx resource, how that markup is parsed and transformed into a Visual Basic® or C# class on the fly, and so forth. But what happens next? Where are the files generated by ASP.NET stored and how are they used to serve page requests? Last month, I began my look at this process. This month, I'll examine what happens on the server so you can avoid some common pitfalls. I'll review the storage of temporary ASP.NET files and the source code of dynamically generated classes used to serve page responses. In addition, I'll build an explorer tool you can use with any ASP.NET 2.0 or ASP.NET AJAX (formerly code-named "Atlas") application to look into and debug the actual code your pages execute. But before we do that, there are a few things you'll need to know. (As with last month's column, this installment is largely based on undocumented details of how ASP.NET works. These implementation details may change in future versions of the Microsoft® .NET Framework.)

What's in the Temporary ASP.NET Folder?

The processing of an ASP.NET page request requires quite a few temporary files. When you install ASP.NET 2.0 on a Web server machine, a hierarchy of folders is created like so:

%WINDOWS%\Microsoft.NET\Framework\v2.0.50727

The version number here refers to the retail version of ASP.NET 2.0. Each released version of ASP.NET, including each intermediate build, has a unique version number and creates a different tree of folders in order to support side-by-side execution. Therefore, it is extremely important that you specify which version of ASP.NET your app is designed for. Apps running under ASP.NET 1.x and ASP.NET 2.0 are based on physically separate folders. Under the Microsoft.NET\Framework folder you find as many vX.X.XXXX subfolders as there are installed versions of ASP.NET (see Figure 1).

Figure 1 ASP.NET 1.0, 1.1, 2.0, and 3.0 Runtime Files
Figure 1 ASP.NET 1.0, 1.1, 2.0, and 3.0 Runtime Files(Click the image for a smaller view)
Figure 1 ASP.NET 1.0, 1.1, 2.0, and 3.0 Runtime Files
Figure 1 ASP.NET 1.0, 1.1, 2.0, and 3.0 Runtime Files (Click the image for a larger view)

Under the root folder of the installed version, you'll find a number of child directories. The CONFIG folder contains machine configuration files, including machine.config and the base web.config used for all sites. The folder named ASP.NETWebAdminFiles contains the source files that make up the Web Site Administration tool that you can run from within Visual Studio® 2005. Finally, the Temporary ASP.NET Files folder contains all temporary files and assemblies being created to serve pages and resources. You have to look into this subtree to find dynamically created files for your Web pages. Note that the Temporary ASP.NET Files directory is the default location for dynamically created files, but this location is configurable on a per-application basis using the <compilation> section in the web.config file:

<compilation tempDirectory="d:\MyTempFiles" />

The first time an application is executed on the machine, a new subfolder is created under the temporary files directory. The name of the compilation subfolder matches the name of the application's IIS virtual directory. If you're simply testing the application using the Visual Studio 2005 embedded Web server, then the subfolder takes the name of the root folder of the Web application. If you invoke pages from the Web server's root folder, you'll find their temp files under the root subfolder (see Figure 2).

Figure 2 WebApps on a Test Web Server
Figure 2 WebApps on a Test Web Server(Click the image for a smaller view)
Figure 2 WebApps on a Test Web Server
Figure 2 WebApps on a Test Web Server (Click the image for a larger view)

Under the application's compilation subfolder, there's a collection of directories based on hashed names. A typical path where you can find temp files is shown here. (The last two directories have fake but realistic names.)

\v2.0.50727\Temporary ASP.NET Files\MyWebApp\3678b103\e60405c7

You can programmatically retrieve the path to the subfolder where the temp files of a given application are located using the following statement:

Dim tempFilesFolder As String = HttpRuntime.CodegenDir

ASP.NET scavenges the compilation folders and removes stale resources periodically when an application is altered and a recompilation is required, however the size of the subtree rooted in Temporary ASP.NET Files may grow significantly, especially on test machines. As an administrator, you might want to keep an eye on the directories under Temporary ASP.NET Files and make sure that all of them refer to currently active applications. If you accidentally delete the subtree of an active application, don't fret. You'll lose all precompiled pages and resources and reset the application to its original compilation state; however, the next request will trigger a new compilation for each page or for a batch of pages, as configured, so in the end there will be no loss of information or pages (though users will experience that first-hit delay for the next requests). Let's now look at the contents of the compilation folder for a given application.

Preservation Files

For each page in the application, the page compilation process generates a file with the following name:

[page].aspx.[folder-hash].compiled

The [page] placeholder represents the name of the .aspx resource. The [folder-hash] placeholder is a hash value that makes the file name unique and disambiguates files with the same name that originally belonged to different folders. Such files are said to be preservation files because they contain important information that helps the ASP.NET runtime to quickly retrieve the assembly and type name of the HTTP handler that will be used to serve the page request. In addition, a preservation file contains a file hash value that is used to detect if the contents of the file has changed since last access.

All .aspx pages that make up an application are compiled in the same temp folder, even if they have the same name and reside in different folders. How does that work? Suppose your application contains two pages named test.aspx that are located in different folders-Folder1 and Folder2. Both pages will be compiled in the same temp folder, but they can be distinguished by their hash values, which will be different because hash values are calculated on the path information and not just the file name. So in the end, the two test.aspx pages might have preservation file names that differ only in the folder hash value:

Test.aspx.cdcab7d2.compiled
Test.aspx.9d86a5d7.compiled

An internally stored cache of hash values allows the ASP.NET runtime to identify hash values for any given page URL and quickly locate the corresponding preservation file. If no preservation file is found, ASP.NET compiles the page on the fly. This is exactly what happens when you deploy the application without precompilation. When you precompile a site, on the other hand, preservation files for each constituent page are created and placed in the Bin folder.

Preservation files are plain XML files. Figure 3 shows the contents of a sample preservation file.

Figure 3 Sample Preservation File (Close Figure 3)
<?xml version="1.0" encoding="utf-8"?>
<preserve resultType="3" virtualPath="/WebSite2/Default.aspx"
          hash="bce9c4d05"
          filehash="2831dc3af5add65c"
          flags="110000"
          assembly="App_Web_ap-sequk"
          type="ASP.default_aspx">
    <filedeps>
        <filedep name="/WebSite2/Default.aspx" />
        <filedep name="/WebSite2/Default.aspx.vb" />
    </filedeps>
</preserve>

Figure 4 details the attributes of the file. The <fileDeps> section lists the files the current page is dependent on. Any changes to any dependency will cause page recompilation. The FileHash value represents a snapshot of the state of dependencies, while Hash represents a snapshot of the state of the current page file. It's worth noting that a mechanism to detect on-the-fly changes to files that is based entirely on file change notifications will fail the moment you stop or restart the Web application. Persisting the state of pages and dependencies as hash values lets you detect changes at any time.

Figure 4 Attributes in the Preservation File (Close Figure 4)

Attribute | Description
Assembly | Name of the assembly that contains the compiled page.
FileHash | Hash value used to check for changes in all source files the page depends upon.
Flags | Integer value used to persist the state of internal operations, such as building a hash value to monitor changes in dependent files.
Hash | Hash value used to check the current page for any changes.
ResultType | Indicates the type of build result for which this preservation file has been created. Feasible values come from a non-public enumeration named BuildResultTypeCode.
Type | Name of the type that represents the compiled page.
VirtualPath | Virtual path of the page being compiled.

The type attribute sets the name of the dynamically created class that will be used to serve the request. By default, the type name is ASP.[page]_aspx, where [page] stands for the name of the page file. Note, though, that you can change this name by setting the ClassName attribute in the @Page directive of your .aspx file. The root namespace won't change, so the type name could be ASP.[ClassName].

The assembly attribute indicates the name of the dynamically created assembly that contains the page class to serve the request. The name and contents of such an assembly depends on the settings in the <compilation> section of the web.config file.

By default, application pages are compiled in batch mode, meaning that ASP.NET attempts to stuff as many uncompiled pages as possible into a single assembly. The attributes maxBatchSize and maxBatchGeneratedFileSize let you limit the number of pages packaged in a single assembly and the overall size of the assembly. By default, you will have no more than 1,000 pages per batched compilation and no assembly larger than 1MB. In general, you don't want users to wait too long when a large number of pages are compiled the first time. At the same time, you don't want to load a huge assembly in memory to serve only a small page, or to start compilation for each and every page. The maxBatchSize and maxBatchGeneratedFileSize attributes help you find a good balance between first-hit delay and memory usage. If you opt for site precompilation (see Fritz Onion's Extreme ASP.NET column in the January 2006 issue) you don't have to worry about first-hit delay, but you should still think about optimal batching parameters to avoid overloading the Web server's memory.

When batching is on, the first 1,000 pages in the application (the actual number depends on maxBatchSize) are compiled to an assembly named App_Web_[random], where [random] is a random sequence of eight characters. If batching is turned off, each page originates its own assembly. The name of the assembly is the following:

App_Web_[page].aspx.[folder-hash].[random].dll

To turn off batching, add the following to your web.config file:

<compilation batch="false" />

If you explore the compilation folder of a sample application, you'll find a companion preservation file that includes CBMResult in the name, as well as a .ccu file with the same name, like this:

test.aspx.cdcab7d2.compiled
test.aspx.cdcab7d2_CBMResult.ccu
test.aspx.cdcab7d2_CBMResult.compiled

The first file in the list is the preservation file. What about the other two? CCU stands for Code Compile Unit and refers to the CodeDOM tree created to generate the source code for the dynamic page class. The CCU file is a binary file that contains the serialized version of the CodeDOM tree for the page. The CBMResult file is the preservation file used to check if the CCU is up to date, where it is located, and which files it is based upon.

CBMResult files are consumed by the modules that communicate with the ClientBuildManager class-for example, Visual Studio 2005 designers and IntelliSense®. These modules query the structure of the page for statement completion. The CCU file maintains an up-to-date copy of the CodeDOM structure of the page ready to service these requests.

Page Class Dynamic Source Code

As mentioned, an .aspx resource is parsed to a Visual Basic or C# class. The class inherits from System.Web.UI.Page or, more likely, from a class that inherits from System.Web.UI.Page. In the most common scenario, in fact, the dynamic page class has the following prototype:

Namespace ASP
    Public Class test_aspx
        Inherits Test : Implements System.Web.IHttpHandler
        ...
    End Class
End Namespace

In the example, the class Test is defined in the code file class of the page and includes any event handlers and helper routines you write in the companion class file for the page. As you may have already noticed while working with Visual Studio 2005, this code file class lacks definitions for members of the page. For each runat=server tag you find in the .aspx source file, you should expect a member of the corresponding type defined in the code file. The ASP.NET runtime system generates a partial Test class that contains all these members plus two additional properties-Profile and ApplicationInstance. Figure 5 shows the set of classes that contribute to serve the request for a given .aspx resource.

Figure 5 Runtime Classes Used to Serve .aspx Request (Close Figure 5)
'Generated by ASP.NET to add member definitions
Partial Public Class Test
    Implements System.Web.SessionState.IRequiresSessionState

    Protected WithEvents TextBox1 As System.Web.UI.WebControls.TextBox
    Protected WithEvents Button1 As System.Web.UI.WebControls.Button
    Protected WithEvents Msg As System.Web.UI.WebControls.Label
    Protected WithEvents form1 As System.Web.UI.HtmlControls.HtmlForm

    Protected ReadOnly Property Profile() _
            As System.Web.Profile.DefaultProfile
        Get
            Return CType(Me.Context.Profile, _
                System.Web.Profile.DefaultProfile)
        End Get
    End Property

    Protected ReadOnly Property ApplicationInstance() _
            As System.Web.HttpApplication
        Get
            Return CType(Me.Context.ApplicationInstance, _
                System.Web.HttpApplication)
        End Get
    End Property
End Class

'Your code file
Partial Class Test : Inherits System.Web.UI.Page
    Protected Sub Button1_Click(ByVal sender As Object, _
        ByVal e As System.EventArgs) Handles Button1.Click
    End Sub
End Class

'Created to build the expected markup for the .aspx resource
Namespace ASP
    Public Class test_aspx
        Inherits Global.Test
        Implements System.Web.IHttpHandler
        ...
    End Class
End Namespace

The classes in Figure 5 span two distinct source files. The first contains the partial class to complete the class in the code file and the actual page class derived from that to serve the request. The second file is a copy of the code file you created in the project. These files are named after the assembly name. The pattern is as follows: [assembly].X.vb. (or .cs if you're using C#). X is a progressive, 0-based index to uniquely name files.

If you look at the contents of the compilation folder for a sample test.aspx page, you'll find that a third file is created, such as the one in this example:

Namespace __ASP
   Friend Class FastObjectFactory_app_web_test_aspx_cdcab7d2_xg83msu0
      Private Sub New()
          MyBase.New
      End Sub

      Shared Function Create_ASP_test_aspx() As Object
          Return New ASP.test_aspx
      End Function
  End Class
End Namespace

The class name is the name of the page assembly prefixed by the string FastObjectFactory. The class features one shared function (static if written in C#) named Create_XXX where XXX is the name of the page class to instantiate. As the name suggests, this is a helper class that the ASP.NET runtime uses to speed up the creation of page instances-a very frequent operation. Creating such a class takes a very small amount of time compared to the time it takes to compile a page. On the other hand, a factory class is considerably faster than the practice of indirect object creation using Activator.CreateInstance.

The contents of the factory class changes depending on the batch compilation setting. In the default case, when batching is on, the factory class contains as many Create_XXX functions as there are batched pages. The name of the factory class matches the name of the batch assembly:

' Used to serve test.aspx
Shared Function Create_ASP_test_aspx() As Object
   Return New ASP.test_aspx
End Function

' Used to serve default.aspx
Shared Function Create_ASP_default_aspx() As Object
   Return New ASP.default_aspx
End Function

If batching is off, the factory class matches the name of the individual page assembly and contains just one shared function-the factory of the specific page. In this case, each page in the application will have its own factory class.

The Runtime Public API

Armed with the information discussed thus far, exploring the contents of the compilation folder may not be very difficult. Still, a tool to help you quickly locate the information you want would be handy. In a minute, I'll design an explorer tool to navigate the dynamically generated source code of an ASP.NET application, but first let's look at some of the runtime APIs in the .NET Framework 2.0. In particular, there are two classes you might want to know more about: HttpRuntime and ClientBuildManager.

The HttpRuntime class features a number of shared properties that return information about a variety of system paths including the Bin folder of the current application, the ASP.NET installation, the compilation folder, and the current AppDomain ID. You can also easily obtain the list of assemblies loaded in the current AppDomain using the following code:

Dim listOfAssemblies() As Assembly
listOfAssemblies = AppDomain.CurrentDomain.GetAssemblies()

This code is not specific to ASP.NET, but when invoked from within an ASP.NET application it returns the array of assemblies in the AppDomain, including all of those generated for your pages.

The ClientBuildManager class doesn't have many informative properties, except for CodeGenDir which returns the same information as the CodeGenDir property on HttpRuntime. It does, however, feature a number of methods for reading configuration information (such as supported browsers) and for precompiling the application. One of the methods on the class, Get, returns a list of application directories monitored for critical changes that will shut down the AppDomain. These directories are: App_Browsers, App_Code, App_GlobalResources, App_WebReferences and Bin.

Building the Explorer Tool

For debugging, it's often useful to have quick access to the source code of the running page along with other runtime information. Any tool that provides that functionality would have to be compatible with any ASP.NET application and require limited configuration or no configuration at all. Nikhil Kothari's excellent Web Development Helper tool would be perfect, if only it provided ASP.NET runtime information. The tool is implemented as a Browser Helper Object (BHO), a COM-based plug-in for the Microsoft Internet Explorer® user interface. A BHO would have been a great host environment for the spy tool I built for this column but, unfortunately, I'm too lazy to go that route. So I wrote my tool as an HTTP module that sits between your page and the browser, looks up the query string, and kicks in if explicitly invoked. Installing the HTTP module in an ASP.NET application requires only one extra line in the web.config and can be turned on and off easily:

<httpModules>
   <add name="AspExplorerModule" type="Samples.AspExplorerModule" />
</httpModules>

Figure 6 shows most of the code of the Explorer HTTP module. The module registers for the PostMapRequestHandler application event and hooks up the page class. The PostMapRequestHandler event fires when the ASP.NET runtime has determined the HTTP handler object needed to serve the request. If the request contains the source=true parameter on the query string, and the handler is a class that inherits from System.Web.UI.Page, the module starts working.

Figure 6 ASP Explorer HTTP Module (Close Figure 6)
Namespace Samples
    Public Class AspExplorerModule : Implements IHttpModule
        Private _app As HttpApplication

        Public Sub Dispose() Implements IHttpModule.Dispose
        End Sub

        Public Sub Init(ByVal context As HttpApplication) _
                Implements IHttpModule.Init
            If context Is Nothing Then
                Throw New ArgumentNullException("[context] argument")
            End If

            ' Cache the HTTP application reference
            _app = context

            ' Map the app event to hook up the page
            AddHandler context.PostMapRequestHandler, _
                New EventHandler(AddressOf OnPostMapRequestHandler)
        End Sub

        Private Sub OnPostMapRequestHandler( _
                ByVal source As Object, ByVal e As EventArgs)
            ' Get the just mapped HTTP handler and cast to Page
            Dim theHandler As IHttpHandler = _app.Context.Handler
            Dim thePage As Page = TryCast(theHandler, Page)

            ' If OK, register an event handler for PreRender
            If thePage IsNot Nothing AndAlso IsSourceRequest() Then
                HookUpPage(thePage)
            End If
        End Sub

        Private Function IsSourceRequest() As Boolean
            Return String.Equals( _
                app.Request.QueryString.Item("source"), "true", _
                StringComparison.OrdinalIgnoreCase)
        End Function

        Sub HookUpPage(ByVal thePage As Page)
            AddHandler thePage.PreRenderComplete, _
                New EventHandler(AddressOf PreRenderComplete)
        End Sub

        Private Sub PreRenderComplete( _
                ByVal sender As Object, ByVal e As EventArgs)
            Dim thePage As Page = DirectCast(sender, Page)
            thePage.SetRenderMethodDelegate( _
                New RenderMethod(AddressOf RenderSource))
        End Sub

        Private Sub RenderSource( _
                ByVal output As HtmlTextWriter, _
                ByVal container As Control)
            Dim thePage As Page = DirectCast(container, Page)

            Dim source As String
            source = GetWebPageInfo(thePage)
            output.Write(source)
        End Sub

        Private Function GetWebPageInfo(ByVal thePage As Page) As String
            ...
        End Function

    End Class
End Namespace

The ASP Explorer module hooks up the page class and registers its own handler for the PreRenderComplete event. Architected in this way, the HTTP module doesn't alter the runtime processing of the request and doesn't interfere with the compilation of the page either. When the source parameter is specified on the query string and set to true, the module kicks in. As you can see in Figure 6, all the module does is register a rendering delegate for the page using the little-known Page class method, SetRenderMethodDelegate. When a rendering delegate is specified for a page, the wrapped method overrides the standard rendering. In other words, once the module is installed, if you place a call to test.aspx, you'll see the standard output of the page; if you place a call to test.aspx?source=true, you'll see all the runtime information about the page the module can collect.

The ASP Explorer source code defines a class to map the contents of the preservation file for the current page. It reads the preservation file and in the class in Figure 7 copies any information. The SourceFiles property is a collection designed to contain all source files that contribute to the page. The collection is populated with information not found in the preservation file, but still obtained from the compilation folder. In particular, this includes all source files that relate to a given page that are .vb or .cs files whose name begins with the name of dynamic page assembly. The GetWebPageInfo method (see Figure 6) captures any information and builds the output for the request in source mode. The page output includes runtime information as well as the source code of dynamic page classes. Figure 8 shows ASP Explorer in action.

Figure 7 Helper Class to Load Information from Preservation Files (Close Figure 7)
Public Class PreservationInfo
    Private _file As String
    Private node As XmlNode

    Public Sub New(ByVal file As String)
        _file = file
        Load()
    End Sub

    Private Sub Load()
        Dim doc As New XmlDocument
        doc.Load(_file)
        node = doc.DocumentElement

        If node Is Nothing OrElse _
                Not String.Equals(node.Name, "preserve") Then Return

        Dim buf As String

        ' ResultType
        buf = node.Attributes("resultType").Value
        Int32.TryParse(buf, ResultType)

        ' VirtualPath
        VirtualPath = node.Attributes("virtualPath").Value

        ' Hash
        Hash = node.Attributes("hash").Value

        ' FileHash
        FileHash = node.Attributes("filehash").Value

        ' Flags
        buf = node.Attributes("flags").Value
        Int32.TryParse(buf, Flags)

        ' Assembly
        AssemblyName = node.Attributes("assembly").Value

        ' Type
        TypeName = node.Attributes("type").Value

        ' Dependencies
        Dim list As XmlNodeList = node.SelectNodes("filedeps/filedep")
        FileDeps = Array.CreateInstance(GetType(String), list.Count)

        For i As Integer = 0 To list.Count - 1
            FileDeps(i) = list.Item(i).Attributes("name").Value
        Next

        ' Source files
        Dim searchPath As String = String.Format("{0}.*", AssemblyName)
        Dim files() As String = _
            Directory.GetFiles(HttpRuntime.CodegenDir, searchPath)

        SourceFiles = Array.CreateInstance(GetType(String), files.Length)
        For i As Integer = 0 To files.Length - 1
            Dim fileInfo As New FileInfo(files(i))
            If fileInfo.Extension = ".vb" Or _
                   fileInfo.Extension = ".cs" Then
                SourceFiles(i) = files(i)
            End If
        Next
    End Sub

    Public ResultType As Integer
    Public VirtualPath As String
    Public Hash As String
    Public FileHash As String
    Public Flags As Long
    Public AssemblyName As String
    Public TypeName As String
    Public FileDeps() As String
    Public SourceFiles() As String
End Class
Figure 8 ASP Explorer Module in Action
Figure 8 ASP Explorer Module in Action(Click the image for a smaller view)
Figure 8 ASP Explorer Module in Action
Figure 8 ASP Explorer Module in Action (Click the image for a larger view)

Examining Sample Pages

Now that you have the tool, let's take a brief look at the structure of the code that ASP.NET generates for each .aspx file. It is interesting to note that without the parsing and compilation facilities provided by the ASP.NET runtime, you would have to write that code yourself in order to run an ASP.NET page!

The dynamic page class (the test_aspx class in Figure 5) overrides a few methods on the System.Web.UI.Page class: FrameworkInitialize, ProcessRequest, and GetTypeHashCode. Nothing special happens in ProcessRequest, which simply calls its base method. GetTypeHashCode returns the hash code for the page that uniquely identifies the page's control hierarchy. The hash value is calculated on the fly when the page is compiled and is inserted as a constant in the source.

The most interesting override is FrameworkInitialize. The method governs the creation of the page's control tree and calls into a private method named __BuildControlTree. This method populates the Controls collection of the page class with fresh instances of the controls corresponding to runat=server tags in the .aspx source. __BuildControlTree parses all server-side tags and builds an object for each of them.

<asp:textbox runat="server" id="TextBox1" text="Type here" />

Here's the typical code you get for the preceding markup:

Private Function __BuildControlTextBox1() As TextBox
    Dim __ctrl As New TextBox()
    Me.TextBox1 = __ctrl
    __ctrl.ApplyStyleSheetSkin(Me)
    __ctrl.ID = "TextBox1"
    __ctrl.Text = "Type here"
    Return __ctrl
End Function

What if the control has an event handler or a data binding expression? Let's first consider a button with the Click event handler. You need one additional line:

__ AddHandler __ctrl.Click, AddressOf Me.Button1_Click

For data binding expressions <%# ... %>, the code generated is similar except that the DataBinding event is used:

AddHandler __ctrl.DataBinding, AddressOf Me.DataBindingMsg

The code associated with the handler depends on the nature of the bound control and the code being bound. For the Text property of a Label control it looks like this:

Public Sub DataBindingMsg(ByVal sender As Object, ByVal e As EventArgs)
   Dim target As Label = DirectCast(sender, Label)
   target.Text = Convert.ToString(..., _
       CultureInfo.CurrentCulture);
End Sub

The expression passed to Convert.ToString is exactly the code in the <%# ... %> expression. The type casting is also dependent on the types involved.

If Master Pages and themes are present, the number of source files and the list of dependencies grow, but with the ASP Explorer tool you can track them down at will.

Wrap-Up

ASP.NET performs on-demand dynamic code compilation of the resource types it owns. This feature greatly facilitates rapid iterative development of Web applications but it does require ASP.NET to be able to write files to disk. The compilation folder is a critical folder where much of the ASP.NET magic takes place. You can peer into it for fun-and sometimes even to diagnose and debug tough problems. Of course, most of the functionality discussed here is internal to ASP.NET and, as such, might change in future builds without being noted. As of today, though, that's how ASP.NET 2.0 works. By the way, feel free to use the ASP Explorer tool with ASP.NET AJAX applications too. It works just fine.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Foundations: Using Templates to Customize WPF Controls

Foundations
Using Templates to Customize WPF Controls
Charles Petzold

Code download available at: Foundations2007_01.exe (158KB)
Contents
Dumping the Defaults
The Visual Tree
Template Triggers
Making Progress
Anatomy of a ScrollBar
Button Commands
Creating a 3D Slider
Remember Moderation

With the release of Windows Vista™ and the Microsoft® .NET Framework 3.0 come a host of new technologies for developers to learn about, discuss, and use. New tools, libraries, and paradigms will change how you build managed apps, offering powerful possibilities. We're introducing this new monthly column to cover the underlying technologies you will use for developing your applications. Industry experts you are already familiar with will take turns delving into the Windows® Presentation Foundation, Windows Communication Foundation, and Windows Workflow Foundation. Let's get started.

Traditionally, customizing an existing control in Windows has been a four-step process. It begins with inspiration. Then comes the research and exploration. This inevitably leads to weeping and gnashing of teeth. And finally it concludes with a complete rewrite. Controls are often resistant to customization due to deeply inaccessible code that links the visuals of the control with its functionality. This code is vital to the control so it must be either completely accepted or completely bypassed and replaced.

The pain of control customization has evidently been felt by the developers of the Windows Presentation Foundation, available as part of the .NET Framework 3.0. They've come up with an exciting and powerful solution known as the "template."

The Windows Presentation Foundation template is at the same time so simple and so powerful that it actually took me a while to wrap my head around the concept. I quickly understood Windows Presentation Foundation styles (which are often confused with templates), but templates needed time to sink in.

Every predefined control in the Windows Presentation Foundation that has a visual appearance also has a template that entirely defines that appearance. This template is an object of type ControlTemplate that is set to the Template property defined by the Control class.

When you use a Windows Presentation Foundation control in your application, you can replace that default template with one of your own design. You retain the basic functionality of the control-including all the keyboard and mouse handling-but you can give it an entirely different look. This is what is meant when Windows Presentation Foundation controls are described (rather inelegantly) as "lookless." A control has a default look, but this look is not intrinsic to the inner workings of the control.

Writing templates in code is rather awkward. It is much easier to use Extensible Application Markup Language (XAML). Because templates can be represented entirely in XAML, visual design tools will be available to help. If you're thinking about programming a custom control that will work like an existing Windows Presentation Foundation control but with a different look, stop now! You might very well be able to do it with a template.

The downloadable source code consists of seven standalone XAML files that I'll discuss throughout this column. No C# code was compiled during the making of this column! If you have the .NET Framework 3.0 SDK installed, you can edit the files using XAMLPad or XAML Cruncher, a similar program from my book, Applications = Code + Markup: A Guide to the Microsoft Windows Presentation Foundation.

The Windows Presentation Foundation supports other types of templates for displaying control content, but in this column I'll only discuss objects of type ControlTemplate.

Dumping the Defaults

Every predefined control in the Windows Presentation Foundation that has a visual appearance has a default template. If you are interested in writing custom templates, study these defaults.

The DumpControlTemplate from Chapter 25 of my book displays a control's default Template property in convenient XAML format. (You can download the source code.) But if you'd rather write a template dumper yourself, here's the code that performs the crucial step:

XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;
settings.IndentChars = new string(' ', 4);
settings.NewLineOnAttributes = true;
StringBuilder strbuild = new StringBuilder();
XmlWriter xmlwrite = XmlWriter.Create(strbuild, settings);
XamlWriter.Save(ctrl.Template, xmlwrite);

This code assumes that ctrl is an instance of a class that derives from Control and that ctrl has been rendered on the screen. (It is my experience that the Template property will be null otherwise.) The XamlWriter.Save call raises an exception if the Template property is null, as it will be for some controls that don't have a visual appearance. At the end of this code, calling ToString on strbuild provides a complete XAML document that contains the template.

The XAML in this default template may be a little wordier than the XAML you might write. For example, you might write this:

<Trigger Property="IsEnabled" Value="False">

But in the XAML file generated by XamlWriter.Save, you'll see markup that looks like this:

<Trigger Property="UIElement.IsEnabled">
    <Trigger.Value>
        <s:Boolean>False</s:Boolean>
    </Trigger.Value>

The s prefix is defined with an xmlns namespace declaration for the .NET System namespace.

The Visual Tree

Let's begin by looking at a template for a fairly simple but non-trivial control: the CheckBox. Say you want something that looks a bit more dramatic than the standard CheckBox. Perhaps you want the user's selection to be displayed as a big green checkmark or a big red X that appears over the entire content of the CheckBox. The file BigCheckCheckBox.xaml shows a template for such a CheckBox.

In a XAML file, a template is an element of type ControlTemplate. In a small Windows Presentation Foundation program, ControlTemplate elements are generally defined in a Resources section in the root element of a XAML file. For larger applications, or when defining templates that are shared among applications, ControlTemplate elements are in their own XAML files with root elements of ResourceDictionary.

In either case, the ControlTemplate element generally has three parts. It begins with an optional Resources section that can define styles or brushes used by the template. (The template in BigCheckCheckBox.xaml has no Resources section.) The template then provides the definition of the template's visual tree. This tree describes the desired appearance of the control as a layout of elements and perhaps other controls. The template concludes with a Triggers section, which indicates how elements of the visual tree change in response to changes in the control's properties. Figure 1 and Figure 2 show most of the ControlTemplate elements for the custom CheckBox. The code in the first figure shows the visual tree and the second figure continues the ControlTemplate element with the Triggers section.

Figure 2 The Triggers Section (Close Figure 2)
<ControlTemplate.Triggers>
    <Trigger Property="IsChecked" Value="True">
        <Setter TargetName="path"
                Property="Data"
                Value="M 0 5 L 3 10 10 0" />

         <Setter TargetName="path"
                Property="Stroke"
                Value="Green" />
    </Trigger>

    <Trigger Property="IsChecked" Value="{x:Null}">
         <Setter TargetName="path"
                Property="Data"
                Value="M 0 2.5 A 5 2.5 0 1 1 5 5
                       L 5 8 M 5 10 L 5 10" />

        <Setter TargetName="path"
                Property="Stroke"
                Value="Blue" />
    </Trigger>

    <Trigger Property="IsEnabled" Value="False">
        <Setter Property="Foreground"
                Value="{DynamicResource
                   {x:Static SystemColors.GrayTextBrushKey}}" />
    </Trigger>
    ...
</ControlTemplate.Triggers>
Figure 1 Visual Tree that Describes a CheckBox (Close Figure 1)
<ControlTemplate x:Key="templateBigCheck"
                 TargetType="{x:Type CheckBox}">

    <Border BorderBrush="{TemplateBinding BorderBrush}"
            BorderThickness="{TemplateBinding BorderThickness}"
            Background="{TemplateBinding Background}">

        <Grid>
            <!-- ContentPresenter displays content of CheckBox -->
            <ContentPresenter
                Content="{TemplateBinding Content}"
                ContentTemplate="{TemplateBinding ContentTemplate}"
                Margin="{TemplateBinding Padding}"
                HorizontalAlignment="{TemplateBinding
                                        HorizontalContentAlignment}"
                VerticalAlignment="{TemplateBinding
                                        VerticalContentAlignment}" />

            <!-- This Border displays a semi-transparent red X -->
            <Border>
                <Border.Background>
                    <VisualBrush Opacity="0.5">
                        <VisualBrush.Visual>
                            <Path Name="path"
                                  Data="M 0 0 L 10 10 M 10 0 L 0 10"
                                  Stroke="Red"
                                  StrokeStartLineCap="Round"
                                  StrokeEndLineCap="Round"
                                  StrokeLineJoin="Round" />
                        </VisualBrush.Visual>
                    </VisualBrush>
                </Border.Background>
            </Border>
        </Grid>
   </Border>

The excerpt from BigCheckBox.xaml shown in Figure 1 illustrates the use of the ControlTemplate start tag and the visual tree. Because this template is a resource, it must contain an x:Key attribute with its resource name. You'll notice the TargetType attribute is used-this is not strictly required but it makes the rest of the template simpler by eliminating the need to preface every property with a class name.

The top-level element of this visual tree is a Border, an element that can play parent to a single child. In this sample, three properties of this Border are assigned with the TemplateBinding markup extension. TemplateBinding is used to bind properties of elements in the visual tree to properties of the control.

The TemplateBinding expressions are more interesting in the ContentPresenter element, which formats content for buttons and other controls that derive from ContentControl. It is ContentPresenter that allows just about anything to be displayed inside a Button or CheckBox. Notice that a TemplateBinding is used to set the Margin property of the ContentPresenter to the Padding property of the CheckBox control being templated. Margin is extra space outside an element; padding is space inside a control not occupied by its content, so you can see the rationale behind this binding.

The ContentPresenter appears inside a single-cell Grid panel along with another Border. Grid cells are often used to host multiple children that must appear in layers. The second Border therefore appears on top of the ContentPresenter. This Border is given a background that consists of a 50 percent opaque brush that contains a red X mark. Notice that the Path element that renders the red X mark is assigned a name of path. This name is referenced in the Triggers section of the template.

Template Triggers

The last section of a ControlTemplate is generally dedicated to Trigger elements. These change properties of the elements that make up the visual tree based on changes in properties of the control. Figure 2 shows a large portion of the Triggers section for the custom CheckBox template.

By default, the IsChecked property of the CheckBox is false and the CheckBox displays a red X mark. The first Trigger element indicates that when IsChecked becomes true, the Data property and the Stroke property of the element named path should be set to display a green check mark. The next Trigger element accommodates a null value of IsChecked (this can occur when the CheckBox is set for tri-state operation) by displaying a blue question mark. The last Trigger element changes the Foreground property of the control to a gray brush when the IsEnabled property is false. (The BigCheckCheckBox.xaml file has an additional Trigger element that displays a dotted line around the control's content when the control has input focus.)

The Triggers section of a template can be quite extensive. A Trigger element for the IsMouseOver property is often included so the control reacts when the mouse passes over the control.

You can then reference the template in an element that creates a CheckBox, like so:

<CheckBox Template="{StaticResource templateBigCheck}" ...

Figure 3 shows three CheckBox controls displayed by the BigCheckCheckBox.xaml file. Two have text content and the third (which has its IsThreeState property set to true and its IsChecked property set to null) contains a bitmap.

Figure 3 CheckBox Controls with a Custom Template
Figure 3 CheckBox Controls with a Custom Template(Click the image for a smaller view)
Figure 3 CheckBox Controls with a Custom Template
Figure 3 CheckBox Controls with a Custom Template (Click the image for a larger view)

Making Progress

You might be skeptical of how well the concept of templates can be adapted to more complex controls. After all, some controls have moving parts and more involved interaction with the user. To dissuade you from such skepticism, I'll spend the rest of this column looking at templates for the three controls that derive from RangeBase. These are ProgressBar, ScrollBar, and Slider.

To work properly, templates for more complex controls require certain types of elements with specific names. For example, a visual tree for a ProgressBar must have two elements of type FrameworkElement (or that derive from FrameworkElement) named PART_Track and PART_Indicator. These two elements are known as "named parts" of the template. The names are shown in the SDK documentation of the ProgressBar class as TemplatePart attributes. If your template does not include two elements with these names, the control won't work correctly.

If the ProgressBar is displayed in its default horizontal orientation, then the internal logic of the control sets the Width property of the element named PART_Indicator to a fraction of the ActualWidth property of the element named PART_Track. That fraction is based on the Minimum, Maximum, and Value properties of the ProgressBar. If the orientation of the ProgressBar is vertical, then the Height and ActualHeight properties of the two elements are used instead.

The ProgressBar control actually has two default templates for the two orientations. (This is true of ScrollBar and Slider, as well.) If you want your new ProgressBar to support both orientations, you should write two separate templates and select them in the Triggers section of a Style element that you also define for the ProgressBar.

Figure 4 is an excerpt from the BareBonesProgressBar.xaml file that shows a complete ProgressBar element with the ControlTemplate object as a property of that element. The file also includes a ScrollBar used to test the ProgressBar through a binding. As you manipulate the ScrollBar, the blue rectangle (the PART_Indicator element) varies in width from zero to the width of the red rectangle (the PART_Track element). Note that even though BareBonesProgressBar.xaml gives the PART_Track rectangle an explicit width, you should actually avoid explicit dimensions whenever possible.

Figure 4 Barebones ProgressBar Template (Close Figure 4)
<ProgressBar Margin="50" HorizontalAlignment="Center"
             Value="{Binding ElementName=scroll, Path=Value}">
    <ProgressBar.Template>
        <ControlTemplate>
            <StackPanel>
                <Rectangle Name="PART_Track"
                           Height="20" Width="300" Fill="Red" />

                <Rectangle Name="PART_Indicator"
                           Height="20" Fill="Blue" />
            </StackPanel>
        </ControlTemplate>
    </ProgressBar.Template>
</ProgressBar>

It's more common for the indicator element to be inside the track element, and you'll probably consider using a Border element for the track. But beware: if you also give that Border a non-zero BorderThickness, then that thickness will be part of the Border element's total width, and the width inside the border will be slightly less. If you want to use a Border with a non-zero border thickness for display purposes, make the track another Border with a zero border thickness inside that first border, and put something else inside that second Border for the indicator. (And if you're using a Border element without making use of its border or background properties, consider using a Decorator, which is the borderless, background-less class from which Border descends.)

Figure 5 shows a rather extensive ControlTemplate object from the ThermometerProgressBar.xaml file. This template has a Resources section but no Triggers section. Although the visual tree makes use of some explicit coordinates for borders and corners, the overall dimensions of the ProgressBar are not defined. That responsibility is left to any markup that defines a ProgressBar using a template like this, for example:

<ProgressBar
       Template="{StaticResource
       templateThermometer}"
       Orientation="Vertical" Minimum="0"
       Maximum="100"
       Width="50" Height="350" ...

Figure 5 ControlTemplate for Thermometer ProgressBar (Close Figure 5)
<ControlTemplate x:Key="templateThermometer"
                 TargetType="{x:Type ProgressBar}">

    <!-- Define two brushes for the thermometer liquid -->
    <ControlTemplate.Resources>
        <LinearGradientBrush x:Key="brushStem"
                             StartPoint="0 0" EndPoint="1 0">
            <GradientStop Offset="0" Color="Red" />
            <GradientStop Offset="0.3" Color="Pink" />
            <GradientStop Offset="1" Color="Red" />
        </LinearGradientBrush>

        <RadialGradientBrush x:Key="brushBowl"
                             GradientOrigin="0.3 0.3">
            <GradientStop Offset="0" Color="Pink" />
            <GradientStop Offset="1" Color="Red" />
        </RadialGradientBrush>
   </ControlTemplate.Resources>

   <!-- Two-row Grid divides thermometer into stem and bowl -->
   <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="*" />
            <RowDefinition Height="Auto" />
        </Grid.RowDefinitions>

        <!-- Second grid divides stem area in three columns -->
        <Grid Grid.Row="0">
            <Grid.ColumnDefinitions>
                <ColumnDefinition Width="25*" />
                <ColumnDefinition Width="50*" />
                <ColumnDefinition Width="25*" />
            </Grid.ColumnDefinitions>

            <!-- This border displays the stem -->
            <Border Grid.Column="1" BorderBrush="SteelBlue"
                    BorderThickness="3 3 3 0"
                    CornerRadius="6 6 0 0" >

                <!-- Track and Indicator elements -->
                <Decorator Name="PART_Track">
                    <Border Name="PART_Indicator"
                            CornerRadius="6 6 0 0"
                            VerticalAlignment="Bottom"
                            Background="{StaticResource brushStem}" />
                </Decorator>
            </Border>
        </Grid>

        <!-- The bowl outline goes in the main Grid second row -->
        <Ellipse Grid.Row="1"
                 Width="{TemplateBinding Width}"
                 Height="{TemplateBinding Width}"
                 Stroke="SteelBlue" StrokeThickness="3" />

        <!-- Another grid goes in the same cell -->
        <Grid Grid.Row="1" >
            <Grid.RowDefinitions>
                <RowDefinition Height="50*" />
                <RowDefinition Height="50*" />
            </Grid.RowDefinitions>

            <Grid.ColumnDefinitions>
                <ColumnDefinition Width="25*" />
                <ColumnDefinition Width="50*" />
                <ColumnDefinition Width="25*" />
            </Grid.ColumnDefinitions>

            <!-- This is to close up the gap between bowl and stem -->
            <Border Grid.Row="0" Grid.Column="1"
                    BorderBrush="SteelBlue"
                    BorderThickness="3 0 3 0"
                    Background="{StaticResource brushStem}" />
        </Grid>

        <!-- Another ellipse to fill up the bowl -->
        <Ellipse Grid.Row="1"
                 Width="{TemplateBinding Width}"
                 Height="{TemplateBinding Width}"
                 Stroke="Transparent" StrokeThickness="6"
                 Fill="{StaticResource brushBowl}" />
    </Grid>
</ControlTemplate>

The resulting ProgressBar is shown in Figure 6. Notice that the Orientation property of this ProgressBar must be set to Vertical-otherwise it won't work properly. You can either remember to set the Orientation whenever you use this template, or you can define a Style for the ProgressBar that sets the Orientation and also references the template. (I'll show an example of this technique in a moment.)

Figure 6

Among the downloadable code for this column is also SpeedometerProgressBar.xaml, which produces the more radical ProgressBar shown in Figure 7.

Figure 7 Speedometer ProgressBar
Figure 7 Speedometer ProgressBar

This one is a bit tricky. The template includes two rectangles that are invisible-they have no height or fill color or stroke color, as shown here:

  <Rectangle Name="PART_Track" Width="180" />
  <Rectangle Name="PART_Indicator" />

The width of the PART_Track element is set to the number of degrees in a semicircle. The Polygon element for the red pointer is subjected to a RotateTransform whose Angle property is bound to the ActualWidth of PART_Indicator, like so:

<RotateTransform
   Angle="{Binding ElementName=PART_Indicator,
                 Path=ActualWidth}" />

Anatomy of a ScrollBar

Both ProgressBar and ScrollBar derive from the abstract RangeBase class. Just about all that RangeBase provides are Value properties, the ValueChanged event, and definitions of Minimum, Maximum, SmallChange, and LargeChange. (ProgressBar uses just the Minimum, Maximum, and Value properties.)

ScrollBar is more complex than ProgressBar, reacting to user input in several ways. This behavior is distributed among five child controls in the standard ScrollBar. The moveable Thumb is a control named Thumb. Directly on either side of the thumb are two RepeatButton controls that typically implement Page Up and Page Down commands. (RepeatButton is similar to the regular Button except that it responds to a sustained mouse press with multiple Click events.) These two RepeatButton controls change size whenever the Value property of the ScrollBar changes, and sometimes one or the other shrinks down to nothing. On the extreme ends of the ScrollBar are two more RepeatButton controls of constant size labeled with arrows that typically implement Line Up and Line Down commands.

The Thumb and inner RepeatButton controls are children of an element named Track, which is responsible for all of their interaction, movement, and size changes. Most of the core functionality and messy logic of the ScrollBar is handled by Track.

When you define a ControlTemplate for a ScrollBar, the visual tree needs to contain a Track element named PART_Track. That's the only required named part for a ScrollBar template.

Track is not a control. It derives instead from FrameworkElement, and it does not have a Template property because the Template property is defined by Control. Although you can't provide a template for Track, you can provide templates for the three controls that make up the Track element.

Track defines three properties that correspond to its three children: Thumb (of type Thumb), DecreaseRepeatButton (of type RepeatButton), and IncreaseRepeatButton (also of type RepeatButton). By default, these three properties are null, which means the template's visual tree should contain explicit definitions of these three controls. If you also want to provide the two buttons on the ends of the ScrollBar, the visual tree should include elements for these two buttons. You can give these five child controls their own templates, or you can use the existing templates and just assign some properties.

Button Commands

The four RepeatButton controls in the standard ScrollBar template must be able to communicate with the inner logic of ScrollBar. This is accomplished using predefined RoutedCommand objects that the ScrollBar class defines as static read-only fields. ScrollBar defines no fewer than 17 of these static read-only fields that correspond to various operations the ScrollBar can perform. Some of these commands are for use with the ScrollBar context menu; some are for a ScrollBar used as part of a ScrollViewer control. Still others, specifically those that begin with the words Line and Page, are for templates.

Figure 8 shows a ScrollBar template from the NoFrillsScrollBar.xaml file. The Track element and the two RepeatButton controls at the ends are organized in a Grid panel. Three properties of the Track element are set to two more RepeatButton controls and a Thumb.

Figure 8 A Simple ScrollBar Template (Close Figure 8)
<ControlTemplate x:Key="templateNoFrillsScroll"
                 TargetType="{x:Type ScrollBar}">
    <Grid>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="Auto" />
            <ColumnDefinition Width="1*" />
            <ColumnDefinition Width="Auto" />
        </Grid.ColumnDefinitions>

        <RepeatButton Grid.Column="0"
                      Command="ScrollBar.LineLeftCommand"
                      FontFamily="Wingdings" Content="E" />

        <Track Grid.Column="1" Name="PART_Track">
            <Track.DecreaseRepeatButton>
                <RepeatButton Command="ScrollBar.PageLeftCommand" />
            </Track.DecreaseRepeatButton>

            <Track.IncreaseRepeatButton>
                <RepeatButton Command="ScrollBar.PageRightCommand" />
            </Track.IncreaseRepeatButton>

            <Track.Thumb>
                <Thumb />
            </Track.Thumb>
        </Track>

        <RepeatButton Grid.Column="2"
                      Command="ScrollBar.LineRightCommand"
                      FontFamily="Wingdings" Content="F" />
    </Grid>
</ControlTemplate>

This is a fully functional ScrollBar, but as you can see in Figure 9, the four RepeatButton controls look like regular buttons. The two buttons on the ends were originally quite tiny, so I increased their sizes by setting their content to pointing-hands characters taken from the Wingdings font. Other than that, none of the five controls have any properties set except for the Command property and the Grid.Column attached property.

Figure 9 ScrollBar Based on Figure 8
Figure 9 ScrollBar Based on Figure 8(Click the image for a smaller view)
Figure 9 ScrollBar Based on Figure 8
Figure 9 ScrollBar Based on Figure 8 (Click the image for a larger view)

The template in Figure 8 is intended for a horizontal ScrollBar. A vertical orientation would require three rows rather than three columns in the Grid, and it would reference RoutedCommand fields from the ScrollBar class that include the words Up and Down rather than Left and Right.

The default ScrollBar template uses the ScrollChrome class from the Microsoft.Windows.Themes namespace for rendering some of its controls. You'll also notice that the template seems to be much longer than it needs to be because many of the properties of the controls in the visual tree are defined through styles rather than attributes. For example, where you could use something like this:

<RepeatButton IsFocusable="False" ...
instead you see something like this:
<RepeatButton ... >
    <RepeatButton.Style>
        <Style TargetType="RepeatButton">
            ...
            <Setter Property="UIElement.IsFocusable">
                <Setter.Value>
                    <s:Boolean>False</s:Boolean>
                </Setter.Value>
            </Setter>
            ...

I suspect the default templates contain markup like this because these properties were originally defined in Style elements in the Resources section of the templates. But it's important to know that when you write your own templates, you can set properties directly in the elements.

The default template sets both the IsFocusable and IsTabStop properties of each RepeatButton to false-when you experiment with NoFrillsScrollBar.xaml, you'll see why. Click the leftmost button. Now press the Tab key a couple times. You'll see input focus shift from RepeatButton to RepeatButton, which is certainly not what's supposed to happen.

Because there are four RepeatButton controls in the template, it's probably easiest to use a Style element for setting uniform properties on all of them. You can add the following markup to NoFrillsScrollBar.xaml right after the ControlTemplate start tag to solve the focus problem:

<ControlTemplate.Resources>
    <Style TargetType="{x:Type RepeatButton}">
        <Setter Property="Focusable" Value="False" />
        <Setter Property="IsTabStop" Value="False" />
    </Style>
</ControlTemplate.Resources>

You can also use the Resources section of ControlTemplate to define templates for the child controls that make up the visual tree of the ScrollBar. I did this in the SpringLoadedScrollBar.xaml file. Figure 10 shows the visual tree section of that template.

Figure 10 Visual Tree that Refers to Other Templates (Close Figure 10)
<Grid>
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="50" />
        <ColumnDefinition Width="1*" />
        <ColumnDefinition Width="50" />
    </Grid.ColumnDefinitions>

    <!-- Line-left button on left side -->
    <RepeatButton Grid.Column="0"
                  Command="ScrollBar.LineLeftCommand"
                  Foreground="{TemplateBinding Foreground}"
                  Template="{StaticResource templateArrow}" />

    <!-- Named track occupies most of the ScrollBar -->
    <Track Grid.Column="1" Name="PART_Track" >
        <Track.DecreaseRepeatButton>
            <RepeatButton Command="ScrollBar.PageLeftCommand"
                          Foreground="{TemplateBinding Foreground}"
                          Template="{StaticResource templateSpring}" />
        </Track.DecreaseRepeatButton>

        <Track.IncreaseRepeatButton>
            <RepeatButton Command="ScrollBar.PageRightCommand"
                          Foreground="{TemplateBinding Foreground}"
                          Template="{StaticResource templateSpring}" />
        </Track.IncreaseRepeatButton>

        <Track.Thumb>
            <Thumb Background="{TemplateBinding Foreground}" />
        </Track.Thumb>
    </Track>

    <!-- Line-right button on right side -->
    <RepeatButton Grid.Column="2"
                  Command="ScrollBar.LineRightCommand"
                  Foreground="{TemplateBinding Foreground}"
                  Template="{StaticResource templateArrow}"
                  LayoutTransform="-1 0 0 1 0 0" />
</Grid>

Notice that the four RepeatButton elements in the template set the Foreground property to the Foreground property of the ScrollBar itself, and set a Template defined earlier in the Resources section. These other templates (not shown in Figure 10) use the Foreground property of the RepeatButton to color their visuals. Figure 11 shows two ScrollBar controls that use this template but with different Foreground settings.

Figure 11 Using Different Foreground Settings
Figure 11 Using Different Foreground Settings(Click the image for a smaller view)
Figure 11 Using Different Foreground Settings
Figure 11 Using Different Foreground Settings (Click the image for a larger view)

I added DropShadowBitmapEffect elements to the four buttons. In fact, I liked the effect so much that I decided to add a Triggers section that sets an additional drop shadow when the mouse passes over the button.

The template for the spring-loaded ScrollBar assumes a horizontal orientation, and it looks best when the height is 50 device-independent units. These settings are not part of the template itself, and there's really no place for them in the template. You might want to consider putting them in a Style definition, as I'll demonstrate in a moment.

The size of the Thumb in the ScrollBar is based on the ScrollBar ViewportSize property and is commonly used to show the proportion of the document currently in view. I couldn't find a way to override the Thumb size without setting ViewportSize. If you want a larger Thumb, you should write a template for Slider instead-you can actually write a template that makes a Slider look pretty much just like a ScrollBar.

Creating a 3D Slider

Unlike ScrollBar, the default Slider control does not have two small change buttons placed on either end. But you can add such controls to a Slider template. The Slider defines six RoutedCommand objects as static get-only properties, and includes DecreaseSmall, IncreaseSmall, DecreaseLarge, and IncreaseLarge. The Slider can also display optional tick marks on one or both sides.

Perhaps the first thing you'll notice when you look at the default Slider template is that it's over a thousand lines long-that's over three times the length of the default ScrollBar template. Part of the reason for this is that the Slider template doesn't rely on a class from Microsoft.Windows.Themes. Instead, everything is built right into the template.

This template defines different thumb shapes when tick marks are displayed on just one side of the Slider, and the template contains many gradient brushes used in coloring this thumb. If you want to include optional TickBar elements in your Slider template, they should have their Visibility properties set to Visibility.Collapsed. The Triggers section of the ControlTemplate should contain Trigger elements that set the Visibility properties to Visibility.Visible based on the TickPlacement property of Slider. For this reason, it's necessary to assign names to the TickBar elements, but they don't have to be special names.

For my custom vertical Slider, I wanted my Thumb to resemble a lever on a sound-mixing board. My goal was for this lever to look like an elevated piece of plastic with a three-dimensional appearance. To emulate three-dimensional perspective, it had to change shape as it moved up and down. You should see the bottom of the lever when it's pushed to the top of the Slider and the top of the lever when it's positioned at the bottom of the slider.

Windows Presentation Foundation has some 3D graphics capabilities that are ideal for a task of this nature. The Slider3D.xaml file contains the template. Figure 12 shows five of these 3D Slider controls set to different positions. Notice how the lever changes shape depending on its position.

Figure 12 3D Slider Controls
Figure 12 3D Slider Controls(Click the image for a smaller view)
Figure 12 3D Slider Controls
Figure 12 3D Slider Controls (Click the image for a larger view)

A vertical Slider generally begins with a three-column Grid panel; the two outer columns are for the two TickBar elements. In the default vertical Slider template, the center Grid contains a Border element that provides the image of the groove, and a Track element containing the Thumb and two RepeatButton controls. I took the same approach in my template and defined the two RepeatButton controls as just Border elements with Transparent backgrounds. They don't obscure the groove but they still respond to the mouse.

I decided that the Thumb should be 50 pixels high (it's already 50 pixels wide by virtue of the ColumnDefinition in the Grid). And the visual tree contains another Border with a transparent Background. At this point, the Slider is fully functional, though you can't see the thumb. The child of this transparent Border is a Viewport3D element, and here's where it gets interesting.

The way I look at it, there are basically two parts to 3D graphics programming: the grunt work and the fiddling. The grunt work mostly involves defining MeshGeometry3D elements, which are the visible objects defined entirely as a collection of connected triangles in 3D coordinate space. For the Thumb, I defined an object that is a rectangular pyramid with the top cut off:

<MeshGeometry3D
    Positions="-2 -1 0, -1 -0.25 4, -2 1 0, -1 0.25 4,
                2 -1 0,  1 -0.25 4,  2 1 0,  1 0.25 4"

    TriangleIndices="0 1 2, 1 3 2, 0 2 4, 2 6 4,
                     0 4 1, 1 4 5, 1 5 7, 1 7 3,
                     4 6 5, 7 5 6, 2 3 6, 3 7 6"

    TextureCoordinates="0 1, 0.2 0.6, 0 0, 0.2 0.4,
                        1 1, 0.8 0.6, 1 0, 0.8 0.4" />

The Positions attribute consists of three-dimensional coordinates for each of the eight vertices of the figure. The bottom of the figure (where Z equals zero) extends from X values of -2 to +2 and Y values of -1 to +1. The top rectangle (where Z equals 4) extends from X values of -1 to +1 and Y values of -0.25 to +0.25.

Each triplet in the TriangleIndices attribute defines a surface of the figure in terms of indices into the Positions array. For example, the first TriangleIndices triplet is 0, 1, and 2-this refers to the first, second, and third vertices of the Positions collection. The TextureCoordinates collection contains a two-dimensional point for every three-dimensional vertex in the figure. These points correspond to points in a brush based on a GeometryDrawing used to cover the outside of the three-dimensional object:

<GeometryDrawing Brush="LightGray"
    Geometry="F 1 M 0 0 L 1 0 L 1 1 L 0 1 Z
              M 0.2 0.4
              L 0.8 0.4 0.8 0.6 0.2 0.6 Z
              M 0 0 L 0.2 0.4
              M 1 0 L 0.8 0.4
              M 1 1 L 0.8 0.6
              M 0 1 L 0.2 0.6">
    <GeometryDrawing.Pen>
        <Pen Brush="DarkGray"
             Thickness=".05" />
    </GeometryDrawing.Pen>
</GeometryDrawing>

This brush simply covers the whole object with a light-gray color and gives a dark-gray accent to the edges.

When I refer to fiddling with 3D graphics programming, I mean working with the lighting and camera. With lighting, you generally want a combination of DirectionalLight and AmbientLight. The former by itself is too stark; the latter tends to wash out everything. The camera in the template is initially set up to look straight down the Z axis at the lever, but it also has a rotation transform to view the lever from above and below. I couldn't get this to work with a TemplateBinding to the Value property of the Slider, so I used a regular binding instead:

<AxisAngleRotation3D Axis="1 0 0"
    Angle="{Binding RelativeSource={RelativeSource
        AncestorType={x:Type Slider}}, Path=Value}" />

Because the Value property determines the rotation angle of the camera, I needed to impose Minimum and Maximum properties of -25 and 25, respectively. This allows the camera to swing a total of 50 degrees-any more and the lever would get clipped. Less than 50 degrees is OK, but then the 3D effect is muted. The Slider3D.xaml file uses a Style to set these two properties as well as the Orientation property and the Template property, as shown here:

<Style x:Key="styleSlider3D"
       TargetType="Slider">
    <Setter Property="Orientation" Value="Vertical" />
    <Setter Property="Minimum" Value="-25" />
    <Setter Property="Maximum" Value="25" />
    <Setter Property="Template"
            Value="{StaticResource templateSlider3D}" />
</Style>

A Slider using this template actually refers to this style rather than the template.

Remember Moderation

As you've seen, a little hand-coded XAML is all you need to make standard controls look quite different. If you're like most programmers I know, you'll start horsing around and writing templates that violate all sense of logic and good taste.

Before you go crazy with templates, keep in mind that one of the great values of Windows is its consistent user interface. When things look familiar, users know what to expect. So keep your creative instincts thoroughly tempered and use templates in moderation. (The MSDN®Magazine editors made me grudgingly add this section.)

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a Contributing Editor to MSDN Magazine and the author of Applications = Code + Markup: A Guide to the Microsoft Windows Presentation Foundation (Microsoft Press, 2006). His Web site is www.charlespetzold.com.


Team System: Team Foundation Server Version Control

Team System
Team Foundation Server Version Control
Brian A. Randell

Code download available at: VSTS2007_01.exe (199KB)
Contents
Digging into Team Foundation Server
Connecting to the Server
Documents under Version Control
Fit and Finish
Conclusion

I didn't know it at the time, but in February 2004 this column was conceived. Around that time, I was at the Microsoft campus in Redmond for a software design review covering an upcoming product that was code-named "Burton." In every review session, I raised my hand and asked the same question: "Will there be extensibility points?" Over the two days, I consistently got an answer that made me smile: "Yes, Brian, you can customize it." Burton, it turns out, became Visual Studio® Team System, and customizing it is what this column is all about.

In the first of what I expect to be many columns, I'll poke around and show you how to extend and enhance Visual Studio Team System on both the client and server. Along the way, I'll explain how you can build an add-in for Microsoft® Word that allows you to check Word documents in and out of the Team Foundation Server version control repository. Before we start, though, head over to msdn.microsoft.com/vstudio/extend and get yourself a copy of the Visual Studio 2005 SDK. The SDK contains documentation and samples for customizing and extending the entire Visual Studio product line. In order to build this add-in, you'll need a copy of Visual Studio 2005 Professional (or any of the role-based SKUs including Team Suite), the Visual Studio 2005 Tools for Office Second Edition Beta, and the Visual Studio 2005 Team Explorer. Since this column is about extending Team System, I won't be covering the details related to hooking the code into the Word infrastructure, but all the code will be provided for you to see. If you want more details about writing Office add-ins, visit msdn.microsoft.com/office/tool/vsto.

Digging into Team Foundation Server

Before starting, you need to understand the core interaction between Team Foundation Server and my client-side add-in shown in Figure 1-or any other client for that matter. If you're going to work with Team Foundation Server, the first thing you need to do is connect to a valid server. To do this, you must know the server name, the protocol to use (HTTP or HTTPS), and the port. To perform any version control operations, you need a workspace with a working folder. A workspace represents the client-side copy of the artifacts that are under version control on the Team Foundation Server, and it serves as an isolated area to perform your work. Each user can have multiple workspaces on a client machine. Each workspace, in turn, supports multiple mappings between a local file path, known as a working folder, and a path in the version control repository. The first time you use the source code control features inside Visual Studio 2005 in conjunction with a Team Foundation Server, the Team Explorer tools create a default workspace using the name of your machine. So the add-in needs to perform all of these operations in order to be able to get documents in and out of the Team Foundation Server version control repository.

Figure 1 Checking a Word Doc into Team Foundation Server Version Control
Figure 1 Checking a Word Doc into Team Foundation Server Version Control(Click the image for a smaller view)
Figure 1 Checking a Word Doc into Team Foundation Server Version Control
Figure 1 Checking a Word Doc into Team Foundation Server Version Control (Click the image for a larger view)

Microsoft factored the APIs for programming Team System across numerous assemblies. The core set of APIs the add-in uses to interact with Team Foundation Server are contained in three assemblies installed as part of a Team Explorer installation. They are Microsoft.TeamFoundation.Client.dll, Microsoft.TeamFoundation.VersionControl.Client.dll, and Microsoft.TeamFoundation.VersionControl.Common.dll. The Team Explorer setup application installs the assemblies in the Global Assembly Cache (GAC). However, the setup does not register them to show up in the Visual Studio 2005 Add References dialog. You'll need to either modify your Windows registry so that the Add References dialog displays the assemblies (see support.microsoft.com/kb/306149), or manually browse to the assemblies. You'll find browsable copies of the assemblies at %Program Files%\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\.

Connecting to the Server

To get started, you'll want to create a class library, which needs references to the aforementioned assemblies. In the library you'll create a class called tfsvcUtil that will contain a set of shared members (static in Visual C#®) to perform the heavy lifting with the Team Foundation Server APIs. As noted, before the add-in can do any work, it needs to connect to a valid Team Foundation Server. When using Team Explorer to connect to a box running Team Foundation Server, Team Explorer attempts to connect using your current credentials. If the credentials are invalid, Team Explorer displays a dialog asking for a valid user name and password. This is how the add-in should behave as well.

To do any work with Team Foundation Server, you must first acquire an instance of the root object, TeamFoundationServer. The APIs allow you to connect to your server in a couple of ways, depending on your need to reuse the same connection. In the case of the add-in, you'll be using a single reference for the lifetime of the add-in so you'll want to use the TeamFoundationServerFactory.GetServer method, specifically the overload that accepts both a server name and a reference to an object that implements ICredentialsProvider. Because you'll want to be able to provide alternate credentials if authentication fails, you should create an instance of UICredentialsProvider before calling GetServer. Once you call GetServer, you'll want to authenticate against the server by calling the Authenticate method on the TeamFoundationServer instance returned from GetServer. Assuming you are connected, you'll need to acquire a reference to the Team Foundation Server Version Control service.

As mentioned earlier, Microsoft designed Team Foundation Server to be extensible. You can see this in the way the services have been factored on the server. Team Foundation Server provides a set of core services around version control, work item tracking, and team builds. Each core service has its own set of classes that you use when you want to interact with a particular feature. The top-level object you use to work with the Version Control service is VersionControlServer class. You retrieve an instance of this type by calling the GetService method on a valid TeamFoundationServer reference. Figure 2 lists the code necessary to do everything just discussed. There you'll also find some basic error handling and some additional class level variables that you'll use later.

Figure 2 Connect and Authenticate (Close Figure 2)
' Team Foundation Server Version Control Utilities
Public Class tfsvcUtil
  Private Shared serverValidated As Boolean = False
  Private Shared m_tfs As TeamFoundationServer = Nothing
  Private Shared m_tfsVCS As VersionControlServer = Nothing
  Private Shared m_userWorkspace As Workspace = Nothing
  Private Shared m_serverName As String = Nothing

  Private Const MSG_UNEXPECTED As String = _
    "Unexpected exception in tfsUtil.{0}."

  Public Shared ReadOnly Property IsServerReady() As Boolean
    Get
      Return serverValidated
    End Get
  End Property

  Public Shared Sub Connect(ByVal fqServerName As String, _
      ByVal showLoginUI As Boolean)
    Try
      m_serverName = fqServerName
      If showLoginUI Then
        Dim icp As ICredentialsProvider = New UICredentialsProvider
        m_tfs = TeamFoundationServerFactory.GetServer( _
          m_serverName, icp)
      Else
        m_tfs = TeamFoundationServerFactory.GetServer(m_serverName)
      End If

      m_tfs.Authenticate()

      m_tfsVCS = CType(m_tfs.GetService( _
        GetType(VersionControlServer)), VersionControlServer)

      serverValidated = True
    Catch ex As Exception
      Throw New tfsUtilException( _
        String.Format(MSG_UNEXPECTED, "Connect"), ex, m_serverName)
    End Try
  End Sub

  Public Shared Sub Connect(ByVal serverName As String, _
    ByVal protocol As String, ByVal port As Integer, _
    ByVal showLoginUI As Boolean)

    m_serverName = String.Format("{0}://{1}:{2}", _
      protocol, serverName, port)

    tfsvcUtil.Connect(m_serverName, showLoginUI)
  End Sub
End Class

If you examine the code in Figure 2, you'll note there are two versions of the Connect method. One requires that the Team Foundation Server name be in a fully qualified format that includes the protocol, server name, and port. The other version accepts the server connection data as individual items. Both versions expose a parameter that specifies whether the Team Foundation Server should show the login UI in cases where there is an authentication failure. This allows you to reuse the code in situations where you want it to fail if the provided credentials are not satisfactory.

Once you can connect to the server, the next decision is to figure out where you want to put your documents. In order to put documents under version control, a Team Foundation Server needs to have at least one Team Project created with version control enabled. Currently, Microsoft does not provide an API to create a Team Project, so this is something you must do manually before attempting to put documents under version control. Your team project's root trunk will be named $/%TeamProjectName%. From there you can define any number of folders and subfolders. The VersionControlServer object has a GetTeamProject method that you can use to access the ServerItem property to get the correct string representing the root. You should be aware of one thing. If you create a Team Project without defining the source control folder, a valid option when using the New Team Project wizard, you will receive a Microsoft.TeamFoundation.VersionControl.Client.VersionControlException exception when you call GetTeamProject. The string returned from the ServerItem property provides the base to build your folder structure for your documents.

Connecting to the server is only one part of the equation. In order to move documents in and out of the server, the client machine needs some configuration also. As mentioned earlier, Team Foundation Server Version Control uses the concept of a workspace to manage the client portion of version control. If you want to check a file in, check it out, or even check if a file is under version control, you need a workspace. As noted, the first time you use the Team Explorer and work with files under version control, it will create a default workspace for you using your machine name. You manage your workspaces from within Visual Studio 2005 via File | Source Control | Workspaces. However, the add-in can't count on the workspace having been created, so you'll need to add some code to create one if necessary.

Before you create a workspace, you'll want to check to see if one already exists and whether the location of the document you want to add to version control is already mapped to the active Team Project. You start with the Workstation class and use the Workstation object to obtain the WorkspaceInfo instance for the document you wish to add to source code control.

' Get workstation object
Dim userWorkstation As Workstation = Workstation.Current

' We need the path of the of the document:
Dim docPath As String = Environment.GetFolderPath( _
  Environment.SpecialFolder.MyDocuments) & "\Test.doc"
Dim docWSInfo As WorkspaceInfo = _
  userWorkstation.GetLocalWorkspaceInfo(docPath)

If the document's path is mapped to a server path, then the call to GetLocalWorkspaceInfo will return a hydrated instance; otherwise, it will be Nothing. You need to understand a key issue with workspaces and folder mappings. When you check to see if a mapping exists, GetLocalWorkspaceInfo isn't looking for an exact match; partial matches are accepted. If, for example, you have a path of C:\Docs\Specs\MySpec.doc and a folder mapping exists for C:\Docs\Specs\, a WorkspaceInfo instance will be returned as expected. However, if a mapping exists for C:\Docs, the API returns a valid instance even if a mapping does not exist for C:\Docs\Specs\. In general, this implicit association is what you want. Team Foundation Server will map recursively file system subfolders of an existing mapping to subfolders in the version control repository. If you don't have a mapping, create a one by using the CreateMapping method of the Workspace object. Getting a Workspace when you have a valid WorkspaceInfo object is simply a matter of calling GetWorkspace on the VersionControlServer instance.

Documents under Version Control

At this point, the core functions of the add-in-checking in and checking out Word documents-are ready to run. Assuming you can log in to a machine running Team Foundation Server, access a valid Team Project and a local Workspace, you can attempt to put a file under version control. You'll find the relevant methods you need to do your work as members of the Workspace class. For the add-in, you'll need to be able call PendAdd, PendEdit, and CheckIn.

PendAdd is the method to use the first time you place a document under version control. If you examine the API's documentation, you'll see there are six overloads. You have the flexibility to queue a single item or a set of items at once. The method has versions that support recursion as well as the ability to adjust the lock level. For the add-in, you'll be checking in one document at a time, so the first overload that accepts a file name is sufficient. When you call PendAdd, it returns an integer representing the number of items queued for check in. To perform the actual checking, you need to query the Workspace for a list of pending changes using the GetPendingChanges method. Microsoft overloaded this method with nine versions. For the add-in, you'll use the version that accepts a string as input. Because you're passing in a single file path, PendAdd should return 1. When you call GetPendingChanges, it should return a PendingChange array with a single element. You need to be careful with using other versions of GetPendingChanges. The reason is that you could attempt to check in files that are not ready, for example code you're editing in Visual Studio 2005.

Once you have the array of pending changes, you can pass the PendingChange array instance and any check-in comments to the CheckIn method to perform the actual check-in operation. If the method is successful, Team Foundation Server returns an integer representing the Changeset number for the check-in. A changeset is a set of committed modifications to one or more items that are stored in the Team Foundation Server version control repository. It can also contain work item data, check-in notes, and policy override information. As part of the check-in process, the Team Foundation Server client API will turn the read-only bit on for the document. Figure 3 lists two methods from the tfsvcUtil class that perform the initial PendAdd and then the CheckIn. As you examine the code, you should see the extra code necessary to get the Workspace. In addition, there's a helper method GetDefaultWorkspace that will get or create the default Workspace if necessary. This code assumes you've successfully executed the Connect method.

Figure 3 Add Document to Version Control (Close Figure 3)
Public Shared Function AddDocumentToVCR( _
  ByVal docPath As String, _
  ByVal serverPath As String) As Boolean

  If serverValidated Then
    ' We need the Workstation
    Dim userWorkstation As Workstation = Workstation.Current
    ' We need the Workspace
    Dim docWSInfo As WorkspaceInfo = _
      userWorkstation.GetLocalWorkspaceInfo(docPath)

    If docWSInfo Is Nothing Then
      ' there's no existing mapping
      ' so let's get the default workspace
      m_userWorkspace = GetDefaultWorkspace(userWorkstation)
    Else
      m_userWorkspace = m_tfsVCS.GetWorkspace(docWSInfo)
    End If

      ' Now the question is does the workspace
      ' we're working with have a mapping
      Dim docWorkingFolder As New WorkingFolder(serverPath, _
        IO.Path.GetDirectoryName(docPath))
      ' It appears that CreateMapping is happy to let us try and
      ' create a mapping even if an entry already exists
      ' This eliminates the need to search
      m_userWorkspace.CreateMapping(docWorkingFolder)

    Return m_userWorkspace.PendAdd(docPath) = 1
  Else
    Throw New tfsUtilException(MSG_SERVER_NOT_VALIDATED)
  End If
End Function

Public Shared Function CheckInDocument(ByVal docPath As String, _
  ByVal comment As String) As Integer

  If serverValidated Then
    If m_userWorkspace Is Nothing Then
      Dim userWorkstation As Workstation = Workstation.Current
      Dim docWSInfo As WorkspaceInfo = _
        userWorkstation.GetLocalWorkspaceInfo(docPath)
      m_userWorkspace = m_tfsVCS.GetWorkspace(docWSInfo)
    End If

    Dim pc As PendingChange() = m_userWorkspace.GetPendingChanges(docPath)

    If pc IsNot Nothing AndAlso pc.Length > 0 Then
      Return m_userWorkspace.CheckIn(pc, comment)
    Else
      Return -1
    End If
  Else
    Throw New tfsUtilException(MSG_SERVER_NOT_VALIDATED)
  End If
End Function
Private Shared Function GetDefaultWorkspace( _
    ByVal userWorkstation As Workstation) As Workspace

  ' Get the users default's workspace
  Dim defaultWorkspaceInfo As WorkspaceInfo = _
    userWorkstation.GetLocalWorkspaceInfo(m_tfsVCS, _
      Environment.MachineName, m_tfs.AuthenticatedUserName)

  Dim defaultWorkspace As Workspace = Nothing
  If defaultWorkspaceInfo Is Nothing Then
    ' Need to create it
    defaultWorkspace = m_tfsVCS.CreateWorkspace( _
      Environment.MachineName, m_tfs.AuthenticatedUserName, _
      "Auto-generated by tfsUtil")
  Else
    ' Get other
    defaultWorkspace = m_tfsVCS.GetWorkspace(defaultWorkspaceInfo)
  End If

  Return defaultWorkspace

End Function

You check out a file for edit by calling PendEdit on the Workspace object. Microsoft overloaded this method as well, providing five versions. As before, the version you'll use is the one that accepts a single file name. You'll find the code for this operation in the download that accompanies this column at msdn.microsoft.com/msdnmag/code07.aspx. It's almost an exact copy of PendAdd.

Fit and Finish

The core code necessary for the add-in to work with Team Foundation Server is nearly ready, but there are a few more things left to do:

  • oProvide the Team Foundation Server with information such as server name, protocol, and port.
  • oSpecify the Team Project and server path.
  • oSpecify the document and comments as part of a check-in operation.
  • oBuild a Word 2003 application add-in to hook all of the forms and code into a command bar, and handle the Word document related events.

These items require a GUI if the code presented earlier is going to work from an add-in hosted in Word 2003. Unfortunately, even though Microsoft wrote the Team Explorer functions and UI for version control in managed code, the actual dialog boxes used for specifying Team Foundation Server information or adding a document to version control are not exposed. In the code download, you'll find three Windows Forms that satisfy the aforementioned requirements.

The first form you need is frmAddTFS. If you look at Figure 4, you'll see it mimics the Microsoft dialog. It lets you specify the sever name, protocol, and port. The form contains only a small amount of code. You could extend it to look up machines running Team Foundation Server via Active Directory®, check for SSL and more.

Figure 4 The Add Team Foundation Server Form
Figure 4 The Add Team Foundation Server Form

The second form, frmAddtoSCC (displayed in Figure 5), is more complicated in design and implementation. To mimic the Microsoft version, I added some additional helper methods to the tfsvcUtil class: GetAllProjects and GetFolders. GetAllProjects does what it says in one line of code. GetFolders uses the GetItems method of the VersionControlServer object to get a single level of folders for the provided path. As you expand a node, the form calls this method to populate the next level of folders, allowing you to specify where you want your document stored in the repository. There's a bit of extra code you can poke around in that loads the tree view control, handles support for adding sub folders, and so forth.

Figure 5 The Add Document to Version Control Form
Figure 5 The Add Document to Version Control Form(Click the image for a smaller view)
Figure 5 The Add Document to Version Control Form
Figure 5 The Add Document to Version Control Form (Click the image for a larger view)

The frmCheckIn (the dialog you saw in Figure 1) shows you what document you're checking in and provides a textbox for comments. If you've used the Team Explorer version, you'll note that this is much simpler. In a future column, I'll enhance this form to support work items and check-in notes.

Finally, a few notes related to the Word add-in. Although I'm not covering all the gory details of the add-in, I want to point out a couple of things. First, you'll find three additional helper methods in the downloadable version of tfsvcUtil: Disconnect, GetDocumentVCStatus, and IsDocumentUnderVersionControl. Disconnect is a cleanup method so you can change machines running Team Foundation Server. GetDocumentVCStatus is used by the add-in to get the version control status for the active document to keep the commandbar buttons in sync. IsDocumentUnderVersionControl checks to see if the document exists in version control, if you've mapped the folder that contains the document into a workspace.

Second, the Team Foundation Server client API wants an exclusive lock on the document to perform a check-in. In order to make this work, the add-in must close the document, perform the check-in, and then reopen the document.

Conclusion

Visual Studio Team System provides extensibility options for almost every area of the product, and the Version Control service is a shining example. Using the rich APIs, we created a simple add-in that lets you put Word documents into the Team Foundation Server version control repository where they can be easily shared and managed.

Send your questions and comments for Brian to mmvsts@microsoft.com.

Brian A. Randell is a senior consultant with MCW Technologies LLC. Brian spends his time speaking, teaching, and writing about Microsoft technologies. He is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


Extreme ASP.NET: Client-Side Web Service Calls with AJAX Extensions

Extreme ASP.NET
Client-Side Web Service Calls with AJAX Extensions
Fritz Onion

Code download available at: ExtremeASPNET2007_01.exe (160KB)
Contents
Calling Web Services with AJAX
How It Works
Serialization
Conclusion

Since its inception, ASP.NET has fundamentally been a server-side technology. There were certainly places where ASP.NET would generate client-side JavaScript, most notably in the validation controls and more recently with the Web Part infrastructure, but it was rarely more than a simple translation of server-side properties into client-side behavior-you as the developer didn't have to think about interacting with the client until you received the next POST request. Developers needing to build more interactive pages with client-side JavaScript and DHTML were left to do it on their own, with some help from the ASP.NET 2.0 script callbacks feature. This has changed completely in the last year.

At the Microsoft Professional Developer's Conference in September 2005, Microsoft unveiled a new add-on to ASP.NET, code-named "Atlas," which was focused entirely on leveraging client-side JavaScript, DHTML, and the XMLHttpRequest object. The goal was to aid developers in creating more interactive AJAX-enabled Web applications. This framework, which has since been renamed with the official titles of Microsoft® AJAX Library and the ASP.NET 2.0 AJAX Extensions, provides a number of compelling features ranging from client-side data binding to DHTML animations and behaviors to sophisticated interception of client POST backs using an UpdatePanel. Underlying many of these features is the ability to retrieve data from the server asynchronously in a form that is easy to parse and interact with from client-side JavaScript calls. The topic for this month's column is this new and incredibly useful ability to call server-side Web services from client-side JavaScript in an ASP.NET 2.0 AJAX Extensions-enabled page.

Calling Web Services with AJAX

If you have ever consumed a Web service in the Microsoft .NET Framework, either by creating a proxy using the wsel.exe utility or by using the Add Web Reference feature of Visual Studio®, you are accustomed to working with .NET types to call Web services. In fact, invoking a Web service method through a .NET proxy is exactly like calling methods on any other class. The proxy takes care of preparing the XML based on the parameters you pass, and it carefully translates the XML response it receives into the .NET type specified by the proxy method. The ease with which developers can use the .NET Framework to consume Web service endpoints is incredibly enabling, and is one of the pillars that make service-oriented applications feasible today.

The ASP.NET 2.0 AJAX Extensions enable this exact same experience of seamless proxy generation for Web services for client-side JavaScript that will run in the browser. You can author an .asmx file hosted on your server and make calls to methods on that service through a client-side JavaScript class. For example, Figure 1 shows a simple .asmx service that implements a faux stock quote retrieval (with random data).

Figure 1 StockQuoteService.asmx (Close Figure 1)
<%@ WebService Language="C#"
               Class="MsdnMagazine.StockQuoteService" %>

using System;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
// From Microsoft.Web.Extensions.dll assembly
using Microsoft.Web.Script.Services;

namespace MsdnMagazine
{
    [WebService(Namespace = "http://msdnmagazine.com/ws")]
    [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
    [ScriptService]
    public class StockQuoteService : WebService
    {
        static Random _rand = new Random(Environment.TickCount);

        [WebMethod]
        public int GetStockQuote(string symbol)
        {
            return _rand.Next(0, 120);
        }
    }
}

In addition to the standard .asmx Web service attributes, this service is adorned with the ScriptService attribute that makes it available to JavaScript clients as well. If this .asmx file is deployed in an ASP.NET AJAX-Enabled Web application, you can invoke methods of the service from JavaScript by adding a ServiceReference to the ScriptManager control in your .aspx file (this control is added automatically to your default.aspx page when you create a Web site in Visual Studio using the ASP.NET AJAX-enabled Web site template):

<asp:ScriptManager ID="_scriptManager" runat="server">
  <Services>
    <asp:ServiceReference Path="StockQuoteService.asmx" />
  </Services>
</asp:ScriptManager>

Now from any client-side JavaScript routine, you can use the MsdnMagazine.StockQuoteService class to call any methods on the service. Because the underlying mechanism for invocation is intrinsically asynchronous, there are no synchronous methods available. Instead, each proxy method takes one extra parameter (beyond the standard input parameters)- a reference to another client-side JavaScript function that will be called asynchronously when the method completes. The example page shown in Figure 2 uses client-side JavaScript to print the result of calling the stock quote Web service to a label (span) on the page.

Figure 2 Sample Web Service Client Page (Close Figure 2)
<%@ Page Language="C#" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
 <title>ASP.NET AJAX Web Services: Web Service Sample Page</title>

 <script type="text/javascript">
    function OnLookup()
    {
      var stb = document.getElementById("_symbolTextBox");
      MsdnMagazine.StockQuoteService.GetStockQuote(
        stb.value, OnLookupComplete);
    }

    function OnLookupComplete(result)
    {
      var res = document.getElementById("_resultLabel");
      res.innerHTML = "<b>" + result + "</b>";
    }
  </script>
</head>
<body>
    <form id="form1" runat="server">
    <asp:ScriptManager ID="_scriptManager" runat="server">
      <Services>
        <asp:ServiceReference
             Path="services/StockQuoteService.asmx" />
      </Services>
    </asp:ScriptManager>
    <div>
    <h1>ASP.NET AJAX Web Services: Web Service Sample Page</h1>
    Enter symbol:
    <asp:TextBox runat="server" id="_symbolTextBox" />
    <br />
    <input onclick="OnLookup();" id="_lookupButton" type="button"
           value="Lookup" />
    <br />
    <asp:Label runat="server" id="_resultLabel" />
    </div>
    </form>
</body>
</html>

If something goes wrong with a client-side Web service call, you definitely want to let the client know, so it's usually wise to pass in another method that can be invoked if an error, abort, or timeout occurs. For example, you might change the OnLookup method shown previously as follows, and add an additional OnError method to display any problems:

function OnLookup()
{
  var stb = document.getElementById("_symbolTextBox");
  MsdnMagazine.StockQuoteService.GetStockQuote(
    stb.value, OnLookupComplete, OnError);
}

function OnError(result)
{
  alert("Error: " + result.get_message());
}

This way if the Web service call fails, you will notify the client with an alert box. You can also include a userContext parameter with any Web service calls made from the client, which is an arbitrary string passed in as the last parameter to the Web method, and it will be propagated to both the success and failure methods as an additional parameter. In this case, it might make sense to pass the actual symbol of the stock requested as the userContext so you can display it in the OnLookupComplete method:

function OnLookup()
{
  var stb = document.getElementById("_symbolTextBox");
  MsdnMagazine.StockQuoteService.GetStockQuote(
    stb.value, OnLookupComplete, OnError, stb.value);
}

function OnLookupComplete(result, userContext)
{
  // userContext contains symbol passed into method
  var res = document.getElementById("_resultLabel");
  res.innerHTML = userContext + " : <b>" + result + "</b>";
}

If you find that you're making many different calls to a Web service, and that you re-use the same error and/or complete methods for each call, you can also set the default error and succeeded callback method globally. This avoids having to specify the pair of callback methods each time you make a call (although you can choose to override the globally defined methods on a per-method basis). Here is a sample of the OnLookup method that sets the default succeeded and failed callback methods globally instead of on a per-call basis.

// Set default callbacks for stock quote service
MsdnMagazine.StockQuoteService.set_defaultSucceededCallback(
                 OnLookupComplete);
MsdnMagazine.StockQuoteService.set_defaultFailedCallback(
                 OnError);
function OnLookup()
{
  MsdnMagazine.StockQuoteService.GetStockQuote(stb.value);
}

Another interesting alternative to building a complete .asmx file for your Web service methods is to embed the Web service methods directly in the page class. If it doesn't make sense to build a complete Web service endpoint for the methods you want to call, you can expose a Web method from your page that is callable from client-side JavaScript by adding a server-side method to your page (either directly in the page or in the codebehind) and annotating it with the WebMethod attribute. You will then be able to invoke it via the client-side object PageMethods. The example in Figure 3 shows the stock quote service sample rewritten to be entirely contained in a single page instead of split into a separate Web service.

Figure 3 A Single-Page Stock Quote Service (Close Figure 3)
<%@ Page Language="C#" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
<title>ASP.NET AJAX Web Services: Web Service Sample Page</title>
  <script runat="server">
        private static Random _rand = new Random();

        [WebMethod]
        public static float GetStockQuoteFromPage(string symbol)
        {
            return _rand.Next(0, 120);
        }
  </script>

  <script type="text/javascript">
  function OnLookup()
  {
    var stb = document.getElementById("_symbolTextBox");
    PageMethods.GetStockQuoteFromPage(stb.value,
                  OnLookupComplete);
  }

  function OnLookupComplete(result)
  {
    var res = document.getElementById("_resultLabel");
    res.innerHTML = "<b>" + result + "</b>";
  }
  </script>
</head>
<body>
    <form id="form1" runat="server">
    <div>
    <h1> ASP.NET AJAX Web Services: Web Service Sample Page </h1>
     Enter symbol:
        <asp:TextBox runat="server" id="_symbolTextBox" />
        <br />
        <input onclick="OnLookup();" id="_lookupButton"
               type="button" value="Lookup" />
        <br />
        <asp:Label runat="server" id="_resultLabel" />
    </div>
    </form>
</body>
</html>

Bear in mind that these client-side proxies can only be generated from ASP.NET .asmx endpoints, Windows Communication Foundation .svc endpoints, or WebMethods embedded directly in a page, and are not a general mechanism for calling arbitrary Web services. In fact, there is a general restriction on the underlying XmlHTTPRequest object that requests be restricted to the same domain from which the page was loaded (for security reasons), so this technique could not be used to call arbitrary Web services regardless of whether the client-side proxies supported it. If you do find the need to call external Web services, your best bet is to set up a bridge .asmx endpoint in your own application that calls into a .NET proxy class (generated with wsdl.exe or Add Web Reference in Visual Studio) for the external Web service.

How It Works

It might seem surprising at first that you can take a standard .asmx Web service and access it from client-side JavaScript within a browser with almost no changes. The secret lies in the registration of a new .asmx HTTP handler, added to the configuration file of every ASP.NET AJAX-enabled Web site:

<httpHandlers>
  <remove verb="*" path="*.asmx"/>
  <add verb="*" path="*.asmx"
       type="Microsoft.Web.Services.ScriptHandlerFactory"
       validate="false"/>
</httpHandlers>

This newly registered handler will invoke the standard Web service handler (System.Web.Services.Protocols.WebServiceHandlerFactory) if it is a standard Web service request made to an .asmx endpoint. If, however, the request has a trailing /js in the URL or contains a query string with an mn= variable (such as ?mn=GetStockQuote), the handler will either return a chunk of JavaScript that creates a client-side proxy for the Web service (the /js option), or it will invoke the corresponding method defined in the WebService-derived class and package up the response in a JavaScript Object Notation (JSON)-encoded string (the ?mn= option).

When a page includes a client-side reference to an .asmx service (via the ServiceReference element within the ScriptManager control), it injects a script element that references the .asmx file with a trailing /js, creating a proxy in the client. For example, the stock quote page I built earlier rendered with the following script element in it:

<script src="StockQuoteService.asmx/js"
        type="text/javascript"></script>

This is, of course, in addition to the script references added for the Microsoft AJAX Libraries, which include the client-side features needed to interact with this proxy. If you try navigating to this endpoint yourself, you will see the following JavaScript (elided):

Type.registerNamespace('MsdnMagazine');
MsdnMagazine.StockQuoteService=function() {
  this._timeout = 0;
  this._userContext = null;
  this._succeeded = null;
  this._failed = null;
}
MsdnMagazine.StockQuoteService.prototype={
GetStockQuote:Sys.Net._WebMethod._createProxyMethod(this,
     "GetStockQuote",
     "MsdnMagazine.StockQuoteService.GetStockQuote",
     "symbol"), ...
}

This JavaScript is using features of the Microsoft AJAX Libraries (like namespaces and the WebMethod class) that are included with every page that includes a ScriptManager control. The proxy method created by this JavaScript is initialized to invoke the .asmx endpoint with the query string ?mn=GetStockQuote in this case, so that whenever you call MsdnMagazine.StockQuoteService.GetStockQuote from the client, it turns into an asynchronous Web request for the same .asmx endpoint. This combination of client-side proxy generation and server-side support for JavaScript-initiated Web service calls means you can include client-side calls to your .asmx Web services in an intuitive way.

Serialization

The default serialization of AJAX-based Web services is JSON. If you look at a trace of the page shown in the last section in action, the bodies of the Web service request and response look like this:

Request: {"symbol":"ABC"}
Response: 51

This is definitely not the standard XML format you are probably accustomed to seeing when calling your .asmx Web services. Since .asmx endpoints were built to serialize into XML, one of the major additions of the ASP.NET 2.0 AJAX Extensions is a JSON serializer. There are actually two of them-one implemented in JavaScript for use on the client, and one implemented in .NET for use on the server, specifically when an AJAX client invokes an .asmx endpoint. The server-side serializer is available through the Microsoft.Web.Script.Serialization.JavaScriptSerializer class and the client-side serializer is available through Sys.Serialization.JavaScriptSerializer. One of the major advantages of using JSON as a serialization format over XML is that you can deserialize objects in JavaScript by simply evaluating a JSON string. The deserialize method of the client serializer class ends up being very short (error checking removed):

Sys.Serialization.JavaScriptSerializer.deserialize=
    function(){eval('('+data+')');}

The serialize method of the JavaScriptSerializer, on the other hand, is considerably more involved. Another advantage to using JSON is its relatively compact representation compared with the XML equivalent.

Much as the XmlSerializer used by standard Web services serializes types to XML, you can take almost any .NET type and serialize it into JSON with JavaScriptSerializer. If you want to try it yourself, it's just a matter of calling the Serialize method of the JavaScriptSerializer class. Figure 4 shows a sample console application that serializes the complex Person type (shown in Figure 5), as an example. (It must include an assembly reference to the Microsoft.Web.Extensions.dll installed in the Global Assembly Cache (GAC) with the ASP.NET AJAX Extensions.)

Figure 5 Sample Person Type (Close Figure 5)
namespace MsdnMagazine.SampleTypes
{
    public class Person
    {
        private string _firstName;
        private string _lastName;
        private int _age;
        private bool _married;

        public bool Married
        {
            get { return _married; } set { _married = value; }
        }

        public int Age
        {
            get { return _age; } set { _age = value; }
        }

        public string FirstName
        {
            get { return _firstName; } set { _firstName = value; }
        }

        public string LastName
        {
            get { return _lastName; } set { _lastName = value; }
        }
    }
}
Figure 4 JSON Serialization Console Application (Close Figure 4)
using System;
using System.Collections;
using System.Collections.Generic;
using Microsoft.Web.Script.Serialization;

class App
{
    static void Main()
    {
        Person p = new Person();
        p.FirstName = "Bob";
        p.LastName = "Smith";
        p.Age = 33;
        p.Married = true;

        JavaScriptSerializer jss = new JavaScriptSerializer();
        string serializedPerson = jss.Serialize(p);
        Console.WriteLine(serializedPerson);
    }
}

The output of the console application would be the Person class in JSON format, or:

{"Married":true,"Age":33,"FirstName":"Bob","LastName":"Smith"}

Like XmlSerializer, JavaScriptSerializer will only serialize publicly accessible data in a type, and there is no support for resolving cyclic references. But any types that can be serialized by a standard .asmx Web service will work fine with this serializer (and yes, this includes DataSet). With this knowledge in place, you can build a more complex Web service, knowing that it will handle complex types as easily as any SOAP-based Web service defined in an .asmx file.

The example in Figure 6 shows a Web service called MarriageService that implements a Marry method to take two Person objects (as defined earlier) and modify their attributes appropriately. (The accompanying ASP.NET page is included with the code download for this issue.)

Figure 6 MarriageService.asmx (Close Figure 6)
<%@ WebService Language="C#"
    Class="MsdnMagazine.MarriageService" %>

using System;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
using Microsoft.Web.Script.Services;
using MsdnMagazine.SampleTypes;

namespace MsdnMagazine
{
    [WebService(Namespace = "http://msdnmagazine.com/ws")]
    [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
    [ScriptService]
    [GenerateScriptType(typeof(Person))]
    public class MarriageService : WebService
    {
        [WebMethod]
        public Person[] Marry(Person[] couple)
        {
          if (couple.Length != 2)
            throw new ArgumentException("2 persons only!");
          if (couple[0].Married || couple[1].Married)
            throw new ArgumentException("Tell your spouse first!");

          couple[0].LastName += "-" + couple[1].LastName;
          couple[1].LastName = couple[0].LastName;
          couple[0].Married = couple[1].Married = true;
          return couple;
        }
    }
}

If you'd rather work with XML in your client-side script, that option is available as well. In addition to the standard WebMethod attribute used when defining Web services, there is a new attribute in the Microsoft.Web.Script.Services namespace called ScriptMethod, which has a ResponseFormat property that can be set to either Json or Xml (it defaults to Json).

namespace PS
{
    [ScriptService]
    [WebService(Namespace = "http://pluralsight.com/ws")]
    [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
    public class StockQuoteService : WebService
    {
        [WebMethod]
        public int GetStockQuote(string symbol)
        {
            return (new Random()).Next(0, 120);
        }
    }
}
The serialized response would then be:
<?xml version="1.0" encoding="utf-8"?><int>74</int>

It is up to you when you invoke this method from client-side JavaScript to handle the XML response. If you plan on running a transformation on it or are already using MSXML, this may be useful.

Conclusion

It's worth pointing out that the ASP.NET 2.0 AJAX Extensions provide two pre-built services to tap into specific ASP.NET 2.0 application services from client-side code: ProfileService and AuthenicationService. With these two client-side proxy classes you can set and retrieve profile values for individual clients, as well as perform authentication (via the default Membership provider) and grant authentication cookies completely within client script.

While many discussions and demonstrations of the ASP.NET 2.0 AJAX Extensions focus on the flashy controls that enable responsive user interfaces, one of the most impressive and useful features is the ability to call Web services directly from client-side JavaScript. With a full .NET Framework/JSON serializer, direct integration with the familiar .asmx Web services, support for batching, and auto-generated bridges to external Web services, the breadth and depth of support for Web services makes this perhaps the most compelling feature of all.

Send your questions and comments for Fritz to xtrmasp@microsoft.com..

Fritz Onion is a cofounder of Pluralsight, a Microsoft .NET training provider, where he heads the Web development curriculum. Fritz is the author of Essential ASP.NET (Addison Wesley, 2003) and Essential ASP.NET 2.0 (Addison Wesley, 2006). Reach him at pluralsight.com/fritz.


Security Briefs: Using Protocol Transition—Tips from the Trenches

Security Briefs
Using Protocol Transition—Tips from the Trenches
Keith Brown

Code download available at: SecurityBriefs2007_01.exe (172KB)
Contents
A Dilemma of Privilege
Factoring Out High Privilege
Kernel Objects and Handles
Kernel Objects and ACLs
A Sample Gateway
Security Considerations

I covered protocol transition and constrained delegation in detail in a previous Security Briefs column, where I dove in deep and looked at the underlying S4U Kerberos extensions that actually make it work. For more of a layman's introduction to protocol transition, check out Item 63 from my book, The .NET Developer's Guide to Windows Security.

Now that Windows Server® 2003 is more widely deployed, I've been getting mail from readers who are trying to use protocol transition to build secure gateways into their intranets. It's a great feature when you're looking to wire up some alternate form of authentication on the front end but would like to reap the benefits of using Kerberos on the back end.

With protocol transition, a gateway (such as a Web portal) can authenticate the user with any technique that makes sense, and then establish a logon for a Windows® account, which can be impersonated and used to access resources on servers. This means you can use all of the built-in user security features in Windows on the back end, including groups, access control lists (ACLs), Authorization Manager (AzMan) roles, COM+ roles, and so on.

While mapping to Windows accounts has been a common feature of many gateways-for example, IIS supports mapping certificates onto user accounts-protocol transition offers the ability to do this without forcing the gateway to store passwords for all the accounts it uses. Normally you need a password to establish a logon session, but with protocol transition, you can bestow trust on a gateway to establish a logon session without knowing the corresponding password. The resulting session will be limited in its reach around the network: Kerberos tickets will only be issued to this session for services listed in the gateway's allowed-to-delegate-to (A2D2) list.

Consider the breach of a gateway that stores passwords versus one that uses protocol transition. When an attacker gains access to a user's password, he can impersonate that user anywhere on the network, for as long as he wants, at least until the user changes her password. But if an attacker compromises a gateway that's trusted for protocol transition, there are no passwords to find there. Sure, he can log on all users whose accounts aren't marked "Sensitive and cannot be delegated," but he will only be able to use those credentials to access services on the A2D2 list. The damage will be constrained in space and time: you can quickly shut down the attack once it's discovered, whereas an attacker who has gained access to user passwords will likely have a much easier time continuing his attack from a safe location.

The way you bestow this sort of trust on a gateway is to grant the security principal for the gateway process permission to use protocol transition, and list the specific services to which it may delegate. Figure 1 shows an example from the lab environment in which I tested the code for this column. This is the Delegation property page for a user account called MyWebIdentity, which is an otherwise low-privileged account assigned to an IIS 6.0 application pool where my sample Web portal runs. As you can see, MyWebIdentity is configured for protocol transition because I've selected the option "Trust this user for delegation to specified services only" and "Use any authentication protocol." Note that I've used the A2D2 list to constrain MyWebIdentity so that the only Kerberos tickets it can obtain via protocol transition will be for the file system on a machine in the domain called FileServer. CIFS stands for the Common Internet File System, the name of the Microsoft® file server.

Figure 1 Configuring an Account for Protocol Transition
Figure 1 Configuring an Account for Protocol Transition(Click the image for a smaller view)
Figure 1 Configuring an Account for Protocol Transition
Figure 1 Configuring an Account for Protocol Transition (Click the image for a larger view)

By the way, you may be wondering how I got the Delegation tab to show up on a normal user account. That's because I assigned a service principal name (SPN) to the user account, which tells Active Directory® that it's designed to be used for a service, not a human. The Delegation tab will only show up for security principals that have at least one SPN assigned, which is why you normally only see it on computer accounts. If you're not sure what an SPN is, please consult "What is a Service Principal Name".

A Dilemma of Privilege

While protocol transition is a really neat feature, the basic ability to establish a logon session for a user without knowing her password is scary. Technically you don't even need to have been granted permission to do protocol transition to do this; anyone can use the S4U2Self Kerberos extension to establish a logon for a user under Windows Server 2003. You won't be able to talk to any other servers unless you've been granted permission to use protocol transition, but what's to stop a normal user from elevating her privileges with code that uses the S4U logon constructor of WindowsIdentity that I discussed in my previous column on S4U Kerberos extensions?

string admin = "administrator@mydomain.local";
using (WindowsIdentity id = new WindowsIdentity(admin)) // S4U logon
using (WindowsImpersonationContext ctx = id.Impersonate()) {
    string SAM = @"c:\windows\repair\sam";
    Console.WriteLine(File.ReadAllText(SAM));
}

Here she's attacking the security account database on the local machine by temporarily upgrading her privileges to the level of domain administrator.

Clearly, this would be bad were it possible. But it's not. You see, while the attacker will be able to successfully execute the first few lines of the previous code, the security token she receives when she performs an S4U logon will be very weak and will not support impersonation. Her call to File.ReadAllText will fail because the token she received for the administrator's account can only be used to identify the administrator, not to impersonate him.

Only a highly privileged user can use the S4U logon constructor on WindowsIdentity to get a token that can be successfully used with impersonation. Under the covers, the Win32® API that this constructor calls, LsaLogonUser, checks whether the calling process's identity has SeTcbPrivilege, which SYSTEM has by default. This is the most powerful privilege in Windows, as it indicates that you are effectively part of the operating system itself, or the "trusted computing base," which is where the acronym TCB comes from. If you're already that trusted, there's nothing you can't do anyway, and the security system will then issue S4U logons that you can impersonate.

This is where my readers started running into trouble. They wanted to use protocol transition, but quickly found that unless their gateway processes were running as SYSTEM, any logons they established via this feature were useless. They couldn't impersonate them to get to back-end resources. Some people went ahead and elevated their gateways to run as SYSTEM and just felt bad about doing it, while others refused and sought other, more traditional solutions.

Factoring Out High Privilege

This dilemma can be solved by factoring the gateway process in half. There's literally only a single line of code that needs the TCB privilege:

new WindowsIdentity(userPrincipalName)

It's reckless to elevate the privilege level of the entire gateway just because this one line of code needs high privilege. Any time you have a process that accepts input from remote users, runs all the time, and runs with high privilege, you've got a dangerous situation. The solution is to remove that one line of code from the gateway program entirely, and move it into a high-privileged helper process. This is not difficult to do if you're a seasoned Win32 systems developer, but if you're new to the platform or Win32, this might just be a showstopper. So in the remainder of this column, I'm going to show you how to build a secure logon service that hosts this one line of highly sensitive code. Along the way you will learn lots of different things, like how to pass tokens around between processes using the Microsoft .NET Framework, how to extend System.Security.AccessControl, and how to use COM+ to host highly privileged processes.

Kernel Objects and Handles

Each Windows process has a handle table that references Windows kernel objects such as files, semaphores, tokens, and so on. Win32 handles are integers that index into this table, and as such, they are only meaningful in the process that opened the handle. You can't write the value of a file handle onto the clipboard, for example, and expect a different process to be able to read that value and start using it as a handle.

If you want to transfer an open handle to another process, you must use the Win32 DuplicateHandle API. Since the WindowsIdentity that the logon service is going to obtain wraps one of these handles, serializing it for communication isn't really an option. You've got to use DuplicateHandle to insert an entry into the target process's handle table.

So here's the plan. The gateway service is going to make a call out to the logon service, using some interprocess communication technique that is easy to secure. As part of this call, the gateway will supply two arguments: the name of the user to be logged on and the process ID of the gateway's process so that the logon service knows in which direction to push the resulting token handle. The logon service will then return the integer handle value that represents the index into the caller's handle table. The caller can then use this value as a Win32 handle and construct its own WindowsIdentity to wrap around the handle.

Here's a .NET interface that represents this contract:

public interface ILogonService {
    int LogonUserViaProtocolTransition(string upn, int callerPID);
}

To help you decide how to best implement this interface, consider the following requirements. First, the process that hosts this logon service must run with the TCB privilege, and the best way to do that is to run as SYSTEM. Second, the ILogonService interface must be carefully secured so that only the designated gateway can use it. Otherwise, it'll act as a privilege escalation lever that will enable any user to elevate privilege by logging on a domain account with local admin rights. Happily, there exists a technology that solves both of these problems easily, and it's built into Windows Server 2003: our good friend COM+.

Did I hear someone say COM+ was dead? Far from it! In my opinion, COM+ is the only way to fly when you're factoring out privilege in this way. Using System.EnterpriseServices, you can write a COM+ component in the managed language of your choice. Then you can leverage the COM+ hosting model to automatically host your component in a service that runs as SYSTEM. Plus, you don't have to write a single line of code for that service. Oh and don't forget the built-in role-based security in COM+, which will allow you to restrict access to the logon method on your component so that only the security principal who is authorized to run your gateway will be allowed to use the logon service.

After building the LogonService assembly, I popped it into the COM+ catalog by running RegSvcs.exe, opened up the Component Services admin tool and found the COM+ server package for the LogonService. To make this into a service that runs as SYSTEM, all I had to do was bring up the property sheet for this package, go to the Activation tab, and check the box that says, "Run application as an NT Service." After doing this, I flipped back to the Identity tab where the "Local System" identity option had now become available. By selecting that option, I'd finished configuring my logon service to run as SYSTEM.

Kernel Objects and ACLs

Another thing to keep in mind about Windows kernel objects is that each is secured with an access control list (ACL), including the token you'll be sending back. Technically, since you're going to be duplicating a handle back to the caller, the ACL shouldn't matter as long as the caller only uses that particular handle. But it's wise to adjust the ACL on the token to ensure that the caller's security principal has permission to use it, especially given that the caller is probably going to be using .NET wrappers like WindowsIdentity to manipulate the token handle. You might end up with a caller being denied permission to use the token you've given him.

Unfortunately, the .NET Framework doesn't yet have a class that allows you to manipulate tokens directly. WindowsIdentity is the closest thing to a Token class in the framework today. But I took this opportunity to extend the classes in System.Security.AccessControl to support reading and writing ACLs on tokens. I know I've said this in the past, but my hat is off to the folks on the Windows team who developed the AccessControl namespace. It was surprisingly easy to extend their NativeObjectSecurity to support tokens. My implementation of TokenSecurity can be found in Figure 2. (Full code is available in the download for this column.) It may look like a lot of code, but if you look closely, all it's doing is customizing the ACL for a token. The most interesting code here is the TokenRights enumeration; all the rest of the code is boilerplate.

Figure 2 Extending AccessControl to Support Token Security (Close Figure 2)
public enum TokenRights : int {
    AssignPrimary       = 0x0001,
    Duplicate           = 0x0002,
    Impersonate         = 0x0004,
    Query               = 0x0008,
    QuerySource         = 0x0010,
    AdjustPrivileges    = 0x0020,
    AdjustGroups        = 0x0040,
    AdjustDefault       = 0x0080,
    AdjustSessionId     = 0x0100,
    Read            = 0x00020008,
    Write           = 0x000200E0,
    Execute         = 0x00020000,
    All             = 0x000F00FF, // TOKEN_ALL_ACCESS_P
    AllPlusSessionId= 0x000F01FF, // TOKEN_ALL_ACCESS
    MaximumAllowed  = 0x02000000,
    AccessSystemSecurity = 0x01000000,
}

public class TokenAccessRule : AccessRule {
    public TokenAccessRule(IdentityReference identity,
        TokenRights tokenRights, AccessControlType aceType)
        : base(identity, (int)tokenRights,
            false, InheritanceFlags.None,
            PropagationFlags.None, aceType) {
    }
    public TokenRights TokenRights {
        get { return (TokenRights)base.AccessMask; }
    }
}

public class TokenAuditRule : AuditRule {
    public TokenAuditRule(IdentityReference identity,
        TokenRights tokenRights, AuditFlags auditFlags)
        : base(identity, (int)tokenRights,
            false, InheritanceFlags.None,
            PropagationFlags.None, auditFlags) {
    }
    public TokenRights TokenRights {
        get { return (TokenRights)base.AccessMask; }
    }
}

public class TokenSecurity : NativeObjectSecurity {
    public TokenSecurity()
        : base(false, ResourceType.KernelObject) {
    }
    public TokenSecurity(SafeTokenHandle token)
        : this(token,
               AccessControlSections.Owner |
               AccessControlSections.Group |
               AccessControlSections.Access) {
    }
    public TokenSecurity(SafeTokenHandle token,
        AccessControlSections includeSections)
        : base(false, ResourceType.KernelObject,
            token, includeSections) {
    }
    public override Type AccessRightType {
        get { return typeof(TokenRights); }
    }
    public override Type AccessRuleType {
        get { return typeof(TokenAccessRule); }
    }
    public override Type AuditRuleType {
        get { return typeof(TokenAuditRule); }
    }
    public override AccessRule AccessRuleFactory(
        IdentityReference identity, int accessMask,
        bool isInherited, InheritanceFlags inheritanceFlags,
        PropagationFlags propagationFlags,
        AccessControlType aceType) {

        return new TokenAccessRule(identity,
            (TokenRights)accessMask, aceType);
    }
    public override AuditRule AuditRuleFactory(
        IdentityReference identity, int accessMask,
        bool isInherited, InheritanceFlags inheritanceFlags,
        PropagationFlags propagationFlags, AuditFlags auditFlags) {

        return new TokenAuditRule(identity,
            (TokenRights)accessMask, auditFlags);
    }
    public void AddAccessRule(TokenAccessRule rule) {
        base.AddAccessRule(rule);
    }
    public void AddAuditRule(TokenAuditRule rule) {
        base.AddAuditRule(rule);
    }
    public void Persist(SafeTokenHandle token,
        AccessControlSections includeSections) {
        base.Persist(token, includeSections);
    }
}

The implementation of the logon service is shown in Figure 3. The first thing you'll probably notice are the System.EnterpriseServices attributes that prepare the LogonService assembly and class for hosting in the COM+ catalog. You should pay special attention to the security attributes that require callers to be authenticated and to be in a particular role to call the logon service. (I used a role called ProtocolTransitionUsers, and I ensured that the security principal under which my gateway runs was a member of this role when I deployed the logon service and configured it in the COM+ catalog.)

Figure 3 The Logon Service (Close Figure 3)
[assembly: ApplicationName("PluralsightLogonService")]
[assembly: ApplicationActivation(ActivationOption.Server)]
[assembly: ApplicationAccessControl(true,
  Authentication = AuthenticationOption.Privacy,
  ImpersonationLevel = ImpersonationLevelOption.Identify,
  AccessChecksLevel = AccessChecksLevelOption.ApplicationComponent)]

 public interface ILogonService {
    int LogonUserViaProtocolTransition(string upn, int callerPID);
}

[ComponentAccessControl(true)]
[SecureMethod]
public class LogonService : ServicedComponent, ILogonService {
    public LogonService() { }

    [SecurityRole("ProtocolTransitionUsers")]
    [SecurityRole("Marshaler")]
    public int LogonUserViaProtocolTransition(
        string upn, int callerPID) {

        SecurityIdentifier callerSID = null;
        CoImpersonateClient();
        try {
            callerSID = WindowsIdentity.GetCurrent().User;
        }
        finally {
            CoRevertToSelf();
        }

        using (SafeProcessHandle targetProcess = OpenProcess(
                    PROCESS_DUP_HANDLE, false, callerPID)) {
            if (targetProcess.IsInvalid) {
                Marshal.ThrowExceptionForHR(
                    Marshal.GetLastWin32Error());
            }

            // this is the line of code that requires high privilege
            using (WindowsIdentity identity =
                new WindowsIdentity(upn)) {

                SafeTokenHandle token = new SafeTokenHandle(
                    identity.Token, false);

                throwIfUserIsAdmin(token);
                grantPermission(token, callerSID);

                // push the token handle into the caller's process
                int targetHandle;
                if (!DuplicateHandle(
                        GetCurrentProcess(), token,      // source
                        targetProcess, out targetHandle, // target
                        0, false, DUPLICATE_SAME_ACCESS)) {
                    Marshal.ThrowExceptionForHR(
                        Marshal.GetLastWin32Error());
                }
                return targetHandle;
            }
        }
    }

    private void grantPermission(SafeTokenHandle token,
                                 SecurityIdentifier callerSID) {
        // use our extension to System.Security.AccessControl!
        TokenSecurity tokenSecurity = new TokenSecurity(token,
            AccessControlSections.Access);
        tokenSecurity.AddAccessRule(new TokenAccessRule(callerSID,
            TokenRights.All, AccessControlType.Allow));
        tokenSecurity.Persist(token, AccessControlSections.Access);
    }

    //Pinvoke stubs
    ...
}

You may also notice the calls to CoImpersonateClient and CoRevertToSelf. These are Win32 APIs that most DCOM programmers are familiar with. They allow me to discover which security principal is calling the logon service, and I capture the security identifier for the caller into a variable which I later use to grant permission on the token.

The rest of the code consists of logging the user on via the S4U logon constructor on WindowsIdentity, followed by duplicating the token back into the caller's process. Fortunately since the logon service will run as SYSTEM, it'll have permission to duplicate handles into any process it wants by default. You see, by design, SYSTEM has full permission to all objects on the system, including all processes. Note the call to throwIfUserIsAdmin. This helper method checks to see if the resulting token has local administrative privileges. This helps prevent a compromised gateway from using the logon service to gain administrative privilege, which would make this whole solution pointless!

One last thing you might notice in my sample code is that I've written some custom SafeHandle-derived classes to help manage handles. This is a good idea in general when you find yourself doing a lot of P/Invoke work. The code for these is easy to understand, and can be found in the SafeHandles.cs file in the accompanying download. Search the Internet for "Shawn Farkas SafeHandle" and you'll find a great blog entry by one of my favorite guys on the Microsoft .NET Framework team that will show you how easy it is to build your own SafeHandle classes.

A Sample Gateway

Before I show you my sample gateway, I want to emphasize something really important: it's easy to misuse protocol transition! There's no way this feature can ensure that you properly authenticate incoming users. When the domain administrator enables protocol transition for your Web application's identity, she's entrusting you with the job that the domain controller normally does-authenticating users.

It's now the gateway's job to authenticate incoming users in the strongest possible fashion, and only then to use its special privilege to perform protocol transition and log on a corresponding Windows user account to represent that user.

Because this article is focused on protocol transition, as opposed to the myriad ways you might be authenticating users on the front end, my gateway sample does not authenticate incoming users at all. It simply allows you to type in a user name and the path to a file on the network that you want to attempt to read using that user's S4U logon. You can take my example and extend it to add whatever front-end authentication scheme makes sense for you to get started using protocol transition from a low-privileged gateway process. This is worth reiterating: do not just copy and paste my gateway sample into your own app, as you'll be leaving yourself wide open to attack!

Figure 4 shows the Web form that my sample gateway uses. You specify the user principal name and a file path as input, and it calls out to the logon service to establish an S4U logon for that user, wraps the resulting handle in a WindowsIdentity, impersonates the user, and reads the file. You can play with opening files on the same machine as the gateway, or with opening files on the network. You'll need to configure the A2D2 list with all the servers you plan on reading files from, and-although this is specific to the file system, which actually goes through a daemon called the Workstation service that runs as SYSTEM on each machine-you'll need to set the A2D2 list for the machine account of the gateway, as well as the user account under which the gateway runs.

Figure 4 Sample Gateway Web App
Figure 4 Sample Gateway Web App(Click the image for a smaller view)
Figure 4 Sample Gateway Web App
Figure 4 Sample Gateway Web App (Click the image for a larger view)

Figure 5 shows the code that runs the Web page, calling out to the logon service. It's important to note that when you construct a WindowsIdentity from a token handle, the WindowsIdentity constructor duplicates the incoming handle. You need to get that original handle closed. This is why I'm careful to use my SafeTokenHandle class in a C# using statement, which will cause the original token handle to be closed at the end of the using block. This is important for long running processes like the gateway, especially for sensitive handles like tokens.

Figure 5 Codebehind for the Sample Gateway Web App (Close Figure 5)
public partial class _Default : System.Web.UI.Page {
    protected void btnRead_Click(object sender, EventArgs e) {
        // connect to our privileged logon service (runs in COM+)
        ILogonService logonService = new LogonService();
        int tokenHandleValue =
            logonService.LogonUserViaProtocolTransition(
                txtUser.Text, GetCurrentProcessId());

        // WindowsIdentity ctor dups the incoming handle, so we use
        // SafeTokenHandle to ensure that the original gets closed
        using (SafeTokenHandle token = new SafeTokenHandle(
            tokenHandleValue))
        using (WindowsIdentity id = new WindowsIdentity(
            token.DangerousGetHandle()))
        using (WindowsImpersonationContext wic = id.Impersonate()) {
            txtOutput.Text = File.ReadAllText(txtPath.Text);
        }
    }

    [DllImport("kernel32.dll", SetLastError = true)]
    static extern int GetCurrentProcessId();
}

Security Considerations

All security countermeasures trade off against something. This solution reduces the privilege level of the gateway process significantly; you no longer have to run your gateway as SYSTEM. Remember that an attacker who successfully takes control of a process can do anything that process can do (including calling out to the logon service to obtain protocol transition logons). An attacker running as SYSTEM can read the local security accounts database, potentially gaining access to other systems on the network once he's cracked the hashes of any weak passwords found there. Running the gateway under lower privilege provides defense-in-depth against these sorts of attacks. So what's the downside? You've introduced a second process, along with additional context switching overhead to call into it. Your solution will be more tricky to deploy, as you now have to install the logon service COM+ component. Are these trade-offs worth it for your system? Measure and decide for yourself.

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is a co-founder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight's Applied .NET Security course, as well as several books, including The .NET Developer's Guide to Windows Security, which is available both in print and on the web. Learn more at pluralsight.com/keith.


{ End Bracket }: From Bouncy Balls to Better Estimates

{End Bracket}
From Bouncy Balls to Better Estimates
James Waletzky

Care to guess how many bouncy balls are in the container on my desk? You don't know the size of the balls, the volume of the container, or how full the container is, which makes an educated guess difficult. An estimate would be a pure stab in the dark. Does this problem sound familiar when applied to developing software? Let's explore the reasons why predicting the amount of development effort is extremely difficult early in a project, and what we can do to improve the accuracy of our estimates.

One type of estimate that we are often asked for early in development is a SWAG-or "Silly Wild-Ass Guess." A SWAG is a gut-feel estimate of effort to develop a software project. Project managers often ask for a SWAG to predict ship dates before we have suitable details on the requirements of our project, let alone any knowledge of the design. Your probability of guessing the number of bouncy balls on my desk is akin to estimating the amount of effort in a project with many variables.

As we remove uncertainty from the project by identifying features, detailing requirements, and creating a design, the picture becomes clearer. We may not have all the information required to make a precise estimate, but at this point we know what we don't know, and we can plan for the future. If I told you that the bouncy balls on my desk are one-half inch in diameter, sit in a container that could hold a loaf of bread, and the container is three-quarters full, your estimate of the number of balls in the container could be fairly accurate. With software projects, eventually we remove enough variability and minimize enough risk such that we can provide a reasonably accurate estimate. At the beginning of a project, however, there are many unknowns.

(Click the image for a smaller view)

(Click the image for a larger view)

Fortunately, there's a way to fast forward the process. My early-cycle development estimates have significantly improved since I discovered Wideband Delphi, an under-utilized team-based estimation technique that was popularized by Barry Boehm in the 1970s. A lightweight process, Wideband Delphi significantly improves the accuracy of estimates. Here is the process:

  1. Gather some people to help. Wideband Delphi recommends a team of two to five people. Bring in some people who have experience building a similar type of application, as well as some who have never built this kind of application. Multiple perspectives are valuable.
  2. Provide an overview of the work to be estimated. The person most familiar with the work gives the group a brief outline of the task at hand, focusing on requirements.
  3. Derive individual estimates. Every person in the room thinks about the problem and privately notes his assumptions. Then, on an index card, each person writes down his estimate in ideal effort-hours, and places it facedown on the table.
  4. Reveal the estimates and discuss assumptions. A facilitator reveals the anonymous estimates. What does everyone expect to happen here? The result is similar to multiple people guessing the number of bouncy balls on my desk without much information-estimates vary widely. A facilitated discussion about the assumptions that factored into the estimates ensues.
  5. Iterate from Step 3 until estimates match up. Each individual learns from the others in the room, updates his assumptions, and provides a new estimate. Estimates start converging. Continue this process until convergence is satisfactory.

Why does this technique work? Wideband Delphi incorporates the experience of others, is feedback-driven, allows estimators to go with their gut by keeping the first round anonymous, and facilitates learning. You quickly discover invalid or missed assumptions, while refining high-level requirements and design. The team agrees upon assumptions, documents decisions, and now has rationale for their numbers.

Wideband Delphi has substantially improved the accuracy of our estimates and decreased the risk in our projects by leveraging multiple minds to anticipate problems. For example, my team maintains a list of prioritized requirements from which we choose a subset to develop in a 30-day iteration. We use Wideband Delphi when we add requirements to this list to help us predict the size of work items and when we derive our detailed estimates for a development iteration. The end result has been on-time software deliveries and a better work-life balance!

For more information on Wideband Delphi and other estimation techniques, I recommend that you read Software Estimation: Demystifying the Black Art by Steve McConnell. It'll help you to deliver your software projects on time-or at least to improve the accuracy of your bouncy ball estimates.

James Waletzky is a Software Design Engineer in the Microsoft Engineering Excellence group. James focuses on improving engineering practices at Microsoft through training, consulting, and tools, while emphasizing continuous improvement.