 MSDN Magazine November 2004

MSDN Magazine November 2004

Attack Surface: Mitigate Security Risks by Minimizing the Code You Expose to Untrusted Users

In this article, Microsoft security expert Michael Howard discusses the cardinal rules of attack surface reduction. His rules - reduce the amount of code executing by default, reduce the volume of code that is accessible to untrusted users by default, and limit the damage if the code is exploited - are explained along with the techniques to apply the rules to your code.

Michael Howard App Lockdown: Defend Your Apps and Critical User Info with Defensive Coding Techniques

Whether you're storing database connection strings, user credentials, or logon info, you'll need to practice good defensive programming techniques to avoid those surprise situations in which your data is exposed. In this article, author Kenny Kerry shows you how.

Kenny Kerr Cryptography: Employ Strong Encryption in Your Apps with Our CryptoUtility Component

When storing sensitive data, you need to be able to identify threats, determine how these threats interact with each other, and how issues can combine to constitute a vulnerability that will leave your data exposed. With a good understanding of the various cryptographic algorithms, salt, hashes, ACLs, and other available techniques, you'll be in a better position to protect your critical data.

Michael Stuart and J Sawyer Trustworthy Code: Exchange Data More Securely with XML Signatures and Encryption

You can sign any kind of data using XML Signature, including part of an XML document, other XML documents, or other data of any format. However, in practice, XML signatures are most frequently used to sign other data represented in XML. In this article, the authors discuss the new standard and how you can benefit from it in your apps.

Mike Downen and Shawn Farkas Safety in Windows: Manage Access to Windows Objects with ACLs and the .NET Framework

Until now, Microsoft did not provide explicit support in the .NET Framework for manipulating security settings. With the .NET Framework 1.x, access can only be granted to users via a series of cumbersome P/Invoke calls. By introducing the concepts of security objects and rules, the .NET Framework 2.0 allows developers to manipulate security settings of objects in a few easy steps using managed code. Want to know more? Read on.

Mark Novak Intrusion Prevention: Build Security Into Your Web Services with WSE 2.0 and ISA Server 2004

Once you've addressed security in your code, it's time to look at the environment it runs in. Firewalls stop unauthorized traffic from getting into your network, and smart Web service-specific firewalls, like the one that comes with Internet Security and Acceleration (ISA) Server 2004, bring XML intrusion prevention to your system for that added layer of safety.

Dino Esposito Editor's Note: Vote Early and Often for MSDN Magazine

As this issue of MSDN Magazine goes to press, Election Day is drawing near. Like many candidates for office, this magazine sports some bold cover lines and bright, appealing pictures. Unlike those same politicians, however, we will keep working for you month after month, year after year, even after you elect to purchase the magazine.

New Stuff: Resources for Your Developer Toolbox

Jungo Software Technologies has announced the release of Go-HotSwap 6. 22, a complete off-the-shelf software package offering a solution for hardware vendors, system integrators, and operating system vendors, aiming to provide hot-swap capabilities to their users.

Nancy Michell Web Q&A: ADO.NET Joins, HTML to XHTML, ASP.NET ViewState, and More

Edited by Nancy Michell Data Points: Updating Data in Linked Servers, Information Schema Views, and More

Every day a developer somewhere needs to write code to iterate through SQL Server™ system objects, query and update tables in linked servers, handle optimistic concurrency, and retrieve column and stored procedure metadata.

John Papa Test Run: API Test Automation in .NET

The most fundamental type of software test automation is automated API testing. API testing essentially entails testing the individual methods that make up a software system rather than testing the overall system itself.

James McCaffrey Advanced Basics: Digital Grandma

As a parent of a young child, I take a lot of pictures—many more than anyone would ever be interested in seeing. Well, anyone except my mother. This is her first grandchild and the one or two pictures I send to her each week only brush the surface of her grandmotherly needs.

Duncan Mackenzie Cutting Edge: The ASP.NET 2.0 Wizard Control

ASP.NET has a lot to offer to both the low-level programmer willing to control every little step of the code and the busiest of developers who needs to point-and-click his way through Web app development using just a few existing components.

Dino Esposito Service Station: Improving Web Service Interoperability

If interoperability is the main promise of Web services, why is it that so many developers and organizations have a difficult time achieving it in practice? With all due respect to our hard-working standards bodies, the primary culprits are the imperfect specifications guiding today's implementations.

Aaron Skonnard .NET Matters: ThreadPoolPriority, and MethodImplAttribute

Stephen Toub C++ Q&A: Calling Virtual Functions, Persisting View State, POD Type

Paul DiLascia { End Bracket }: A Tidal Wave of Change

I surf the Net somewhat obsessively when I have spare time, and I often read various discussion forums such as Slashdot and Neowin. Now, the journalism at those sites isn't always the highest quality, and I have noticed a marked lack of perspective on the level of changes that have occurred in this industry.

Larry Osterman


Attack Surface: Mitigate Security Risks by Minimizing the Code You Expose to Untrusted Users

Attack Surface

Mitigate Security Risks by Minimizing the Code You Expose to Untrusted Users

Michael Howard
This article discusses:
  • Identifying and reducing attack surface
  • Reducing the amount of code executing by default
  • Reducing the volume of code accessible to untrusted users
  • Limiting damage if hackers do attack your code
This article uses the following technologies:
Windows Server 2003, Windows XP, Security
Contents
Reduce the Amount of Running Code
Reduce Access to Entry Points by Untrusted Users
Reducing Privilege to Limit Damage Potential
Anonymous Code Paths
Watch Out for Those Little Protocols
Reduce Attack Surface Early
Measuring Attack Surface
Big Attack Surface = Big Security Work
Other Resources

C
ode fails. It's a sad fact of life. It is these security failures and code vulnerabilities that interest me most. In the industry, we worry a lot about improving code quality. While code quality is exceptionally important, most code will eventually fail so we cannot focus exclusively on getting the code right. Imagine for a moment your code is perfect. It's only perfect by today's standards—a snapshot of best practices at the time it was developed. Yet the vulnerability research landscape is constantly evolving. Four years ago, integer overflow attacks were almost unheard of; now they are the attack de jour! Imagine broadening the scope to all the code you've ever delivered to customers.

You need to change your outlook from "my code is very good quality" to "though my code is the best it can be with today's knowledge, it likely still has security defects." Once you get into this state of mind, the attack surface reduction (ASR) philosophy becomes easy to digest. The attack surface of an app is the union of code, interfaces, services, protocols, and practices available to all users, with a strong focus on what is accessible to unauthenticated users.

The core tenet of ASR is that all code has a nonzero likelihood of containing one or more vulnerabilities. Some vulnerabilities will result in customer compromise. Therefore, the only way to ensure no customer compromises is to reduce code usage to zero. ASR is a compromise between perfect safety and unmitigated risk that minimizes code exposed to untrusted users. Code quality and attack surface reduction combined can help produce more secure software. Perfect code cannot do it alone.

Remember, never lose track of your customers: they're the people who will need to apply any of your security updates. If the code has a flaw, does your customer have to patch the 4 machines using the faulty functionality or the 400 with the software installed by default? It will make a huge difference to the person handling it.

In this article, I will explain a simple process to help you reduce the attack surface and to help improve the security of your applications. ASR has three main goals:

  • Reduce the amount of code executing by default
  • Reduce the volume of code that is accessible to untrusted users by default
  • Limit the damage if the code is exploited

Reduce the Amount of Running Code

This is the easy step—just turn off some features! Apply the 80/20 rule and ask yourself, "Do 80 percent of my customers use this feature?" If the answer is no, then turn it off and make it easy to turn the feature back on. In Windows Server™ 2003, we turned off over 20 services because if the service is not running it cannot be attacked. For example, the Web server, IIS 6.0, is not installed by default; when you do install it, it serves only static files by default. All forms of dynamic Web content are opt-in. Therefore, if there is a security defect in WebDAV, for example, the only affected customers are those actively using IIS and WebDAV rather than all users of Windows Server 2003.

Another example, outlined in Microsoft® Security Bulletin MS04-020, "Vulnerability in POSIX Could Allow Code Execution," does not affect Windows® XP or Windows Server 2003. Applying the 80/20 rule, 80 percent of the users do not use the POSIX subsystem (to be honest, I know of no one who does!), so it was simply removed from the operating system.

Obviously, turning features off can cause applications that used to operate correctly to fail, and you may add complexity when people want to use the feature. Now I'm a security guy, so I always err on the side of security, but I'm painfully aware of the impact on nontechnical users when applications fail. You could make an extremely secure system by running no code whatsoever, but that wouldn't be very useful, now would it?

The good news is ASR is not just an "on or off" proposition; you can reduce attack surface by limiting who can access the code once it is running. This has an interesting side effect: many features are still usable, but are inaccessible by attackers. That's next.

Action: Apply the 80/20 rule to all functional areas. Do 80 percent of your users use the functionality? If not, consider turning it off.

Reduce Access to Entry Points by Untrusted Users

Going back to the IIS example, another way of reducing who can access WebDAV functionality on a given Web site is to restrict access to any site using WebDAV. By default, most Web sites are accessible to all users (regardless of their intent), so any attacker on the Internet could probably exploit a defect. However, if the Web server limited access only to trusted users, or users on a specific subnet, then fewer users could take advantage of the defect. This is good because the code works for those who require the functionality. One of my favorite examples is the firewall in Windows XP Service Pack 2. When we turned the firewall on, one very common scenario broke: the small home network, where a central computer acts as a file and print server for other computers in the home. Disabling this functionality by shutting down the ports at the firewall was simply not an option as it breaks potentially hundreds of thousands of users. The tradeoff was to open the ports, but only on the user's local subnet. Valid users can access files and print them, but the bad guys cannot. See the Sidebar Windows XP Service Pack 2: Information for Developers for more information on Service Pack 2.

Another way to reduce the number of people who can communicate with potentially vulnerable code is to authenticate the caller before she can access the code (of course attackers will try to attack the authentication code, too, but that's a story for another day!) Again, this is something we did in Windows XP SP2. We limited access to remote procedure call (RPC) endpoints to valid Windows users; anonymous users (potential bad guys) cannot access the endpoint and cannot therefore attack the code. Remember, most bad guys are unauthenticated, and by raising the authentication bar a little you can eliminate a large population of potential attackers. This is something I do with every Web server I install—one of the first settings I disable is anonymous access.

Action: Restrict access to network endpoints used by your application to the local subnet or IP range.

Action: Limit access to network endpoints using authentication. Simply validating the user reduces your component's attack surface.

Reducing Privilege to Limit Damage Potential

Let's assume for a moment your code does have vulnerabilities, and you decide not to limit access to the code through some of the simple measures I've just suggested. You can still reduce the damage potential by operating the code with reduced privileges. Let's look at another example. On June 8, 2004, Microsoft issued a security bulletin (Crystal Reports Web Viewer Security Update for June 2004) that affected Crystal Reports from Business Objects. The technical bulletin offered the following warning:

An attacker who successfully exploited the vulnerability could retrieve and delete files through the Crystal Reports and Crystal Enterprise Web viewers on an affected system. The number of files that are impacted by this vulnerability would depend on the security context of the affected component that is used by the Crystal Web viewer.

Because the ASP.NET worker process runs by default as the lower privilege ASPNET account, and is not an administrator account (such as SYSTEM), the damage is limited to reading files in the file system, not deleting them. This was a change made to ASP.NET before its initial release in February 2003. Sure, it's still a bad bug: an attacker could read your BOOT.INI file, but at least they can't delete it. The bug is still bad because the attacker could also read your C:\WINDOWS\repair\sam file or, on Unix or Linux your /etc/passwd file (this bug affects all platforms that Crystal Reports runs on, including AIX, Solaris, and Linux—did you apply the patch?). Now, the bad guy could attempt offline password-cracking attacks.

If your code must operate with high privileges, remove any privileges not explicitly required by the application. For example, I'm working on a tool that requires a "dangerous" Windows privilege—the backup privilege. It's scary because any process running under an account with this privilege can read any file in the file system, regardless of the access control list (ACL) on the file. That's why it's called the backup privilege—you don't want a backup application to backup only certain files and not others because of an ACL conflict. There are two ways to solve this issue: I can run the service as SYSTEM, which has the privilege, or I can run the service under a specific account that only has this privilege. The problem with SYSTEM is that it has the backup privilege, but it also has the restore privilege, debug privilege, and the "act as part of operating system" privilege, and it's an administrative account, too. I think you can guess what I chose.

You should never run services as SYSTEM, or daemons as root, or user accounts with administrative rights unless you have exhausted all other possibilities. A defect in the code could lead to catastrophic failure. Set the default account to a lower privilege and if a corner-case scenario does not work, then let your administrator know which scenarios fail and allow her to make the change, if she feels the benefit outweighs the risk.

When people think of attack surface reduction, they often think of security design, which is good, but developers, not just designers and architects, can also help drive the process by analyzing anonymous code paths.Windows XP Service Pack 2: Information for Developers and Brian Johnson

Last year, just before the 2003 Professional Developers Conference, a group of us at Microsoft met to talk about how we were going to inform developers of the changes Windows XP Service Pack 2 would bring. These changes were first enumerated in a paper by John Montgomery, and Jason Garms spoke at the 2003 PDC about the upcoming changes so that developers would know what to do to make sure their applications worked with the service pack on release.

In the time since the 2003 PDC, Microsoft has created an SP2 product information site and made the SP2 Beta release generally available so that developers and IT professionals had the information they needed to test their applications.

Now that Windows XP Service Pack 2 has been officially released, developers should learn about changes in the operating system that may affect the applications they are currently building. The updates that are included in Service Pack 2 can be divided into four categories: networking, memory, e-mail and messaging, and Web development.

Networking The most important change in networking is that the Windows firewall is now turned on by default. This can have consequences for developers with applications that communicate over the network. In addition, all ports are now closed unless they are in use, access controls have been placed on DCOM, and the Remote Procedure Call.SubSystem (RPCSS) is now run with a reduced set of privileges.

Memory Buffer In the time since Windows XP was first introduced, Microsoft has developed a number of ways to help mitigate the potential damage done caused by buffer overrun attacks, one of the most common vectors of attack against software. Many of the core components in Windows have been recompiled with the /GS switch and other compiler technologies to help protect against such attacks. This recompilation is one of the reasons SP2 is such a significant update. In addition, Microsoft has worked with compiler vendors on Data Execution Prevention technology which will help prevent malicious code from running by enforcing a separation between areas of memory where applications are allowed to execute and areas that only hold data. This will help protect against a whole class of attacks on machines equipped with processors that support this feature.

E-Mail and Instant Messaging The default settings for Microsoft Outlook® Express and for Windows Messenger now provide attachment controls which are designed to act as a speed bump against attachments that carry a malicious payload. Microsoft has also created new APIs to assist in helping users and administrators make better trust decisions about attachments.

Web Development An important update offered in SP2 for Web developers is the inclusion of a pop-up manager, designed to reduce the number of pop-up windows that occur when one is browsing the Web. With SP2, a user usually must confirm that he wants a pop-up window to show before the browser will open the window. The pop-up manager also protects against running untrusted ActiveX controls. Developers need to be aware of how that mechanism works so that they can either guide their users through the confirmation process, or they can redesign their Web applications so that this restriction doesn't break the applications. In addition, the security in the Local Machine zone has been enhanced to help protect against malicious script and harmful Web downloads.

Microsoft is updating the tools delivered to developers with SP2 in mind. The Microsoft Platform SDK has been updated to support SP2, so you'll want to consider updating your PSDK installation to better handle changes in the operating system. In addition, new Service Packs for the .NET Framework 1.0 and 1.1 have been released to help developers take advantage of security enhancements and Execution Protection on those processors that support it. Looking forward, the .NET Framework 2.0 and Visual Studio® 2005 are being designed with a major focus on the security technologies offered in SP2.Microsoft has created a number of resource portals on the Web where you can find related articles and training on SP2. First, be sure to check out the Windows XP Service Pack 2 site on the MSDN Security Developer Center. There you will find links to the SP2 download, a link to the debug (checked) build of the service pack for developers, and links to articles, training, and webcasts designed to help make you successful with this new technology.

You'll also want to visit the following sites for additional information: Windows XP Service Pack 2 on TechNet and Support for Windows XP Service Pack 2 for Developers.

Action: Reduce the privileges under which your code must execute.

Anonymous Code Paths

One part of the threat modeling process involves building data flow diagrams (DFDs), or using a similar technique, such as Unified Modeling Language (UML) Interaction diagrams, to identify the entry points into the application. It is these entry points that can be accessed by attackers. As a developer, you first need to make sure that all the entry points into your code are factored into the DFD. Next, you should look at each entry point to determine what privilege is required to access that entry point. Then trace through the DFD, look for all the potential data flows, data stores, and processes that the code touches, or, more to the point, code that an attacker can touch. Let's assume your code has only two network-facing entry points; one is anonymous and the other can only be accessed by administrators, because your code authenticates the user before they can continue execution. Now sit back and think for a moment: where's the bad guy going to go? He's going to attack the code along the anonymous data path (also called a threat path) simply because he can. He's not going to exercise the administrative code path because, if he can, then he's already an administrator and your problems are only just starting.

Action: As you review the DFD, look for anonymous threat paths. It may be prudent to raise the authorization bar on these paths using some of the methods outlined.

Watch Out for Those Little Protocols

Microsoft issued a security bulletin in early 2004 that affected Windows 2000, Windows XP, and Windows Server 2003. One of the defects fixed was in the Private Communications Transport (PCT) protocol, which is part of the Microsoft Secure Sockets Layer (SSL) library. PCT was used in the past when U.S. federal cryptographic export restrictions existed. Today, there are no such restrictions, so PCT is not often used anymore. That's why in Windows Server 2003, PCT was disabled altogether, and because this bug was in PCT, Windows Server 2003 was unaffected by default, even though the defect was in the code. Windows 2000, however, was affected because PCT is enabled by default on SSL-enabled Windows 2000 Web servers, so for them this fix is critical.

Another protocol to watch out for is User Datagram Protocol (UDP). It's interesting that many applications that use TCP also listen for UDP traffic when they don't need to. If your application uses only TCP, then don't listen for UDP traffic. Remember Slammer? Slammer was an effective worm because the payload fit in one UDP packet. Because there is no three-way handshake like there is in TCP, the worm could travel very rapidly indeed. See Analysis of the Sapphire Worm for an interesting analysis of the worm's impact.

It's also really important you reduce your attack surface as early as possible in the design process.

Action: Apply the 80/20 rule to every network protocol you consume.

Reduce Attack Surface Early

Waiting to reduce your attack surface late in product development is often very difficult because changing functionality late in the game guarantees you'll cause regression errors. Other features probably depend on the functionality to work in a specific way, and you just changed it!

We had an issue like this when we reduced the attack surface of Microsoft Internet Explorer in Windows Server 2003; Windows Update stopped working, so we had to make some small changes in order to make it work.

When designing your application, you should have a section in the design outlining what the attack surface will look like. You should identify items like the following:

  • The networking protocols you enabled by default.
  • The endpoints that should support authentication and authorization. Focus on other anonymous endpoints.
  • The off-by-default features. Look for code that auto-starts or will execute when accessed, such as services, daemons, ISAPI filters and applications, SOAP services, and Web roots.
  • Reusable components (ActiveX® controls, COM objects, and .NET Framework assemblies, especially those marked with the AllowParticallyTrustedCallersAttribute).
  • Process identities for all the code you run.
  • User accounts installed.

When developers know early what the attack surface looks like, they can design and code accordingly from the start, rather than waiting for a big, nasty surprise at the end. Waiting too long causes huge code churn and bad feelings as designs must change to accommodate the disruption.

Measuring Attack Surface

As each week in the development cycle passes by, you should measure the attack surface of your product. Start with a baseline, and then each week count all the items identified in the previous section using various scanning tools. In some cases, you may need to write your own tools if you have entry points specific to your application. If the attack surface count goes up, determine why it went up, and see if you can drive it back down. When engineers know you are measuring the attack surface, they will try not to stand out by increasing the attack surface in the first place. See Figure 1 for an example.

Figure 1 Comparing Attack Surface Against a Baseline (Close Figure 1)

Baseline | Baseline + 1 month | Comment
3 x TCP ports | 2 x TCP ports | Good; one fewer port to worry about.
1 x UDP port | 2 x UDP port | Which functionality opened the new UDP port? Why is it open by default? Is it authenticated? Is it restricted to a subnet?
2 x Services (both SYSTEM) | 3 x Services (2 x SYSTEM, 1 x LocalService) | Why is another service running by default? Why are any running as SYSTEM?
3 x ActiveX controls | 4 x ActiveX controls | Why is the new control installed? Is it safe for scripting?
No additional user accounts | 1 x application account | Turns out this is a member of the administrators group too! Why? What's the password?

Action: Define your minimal attack surface early in development, and measure it during development.

Big Attack Surface = Big Security Work

After all of this, if you decide that you must ship an application with a large attack surface, obviously, that's bad. Not only does it mean your customers may be attacked by default, but it means you have a lot of code to review. I like to look at it this way: a large attack surface means you have a great deal of faith in your code quality. The only way you can trust the code is by having very extensive code reviews and by making sure the code is defensive and conservative. Imagine two similar products, one from you and the other from a competitor. Your competitor's product does not run by default, runs with least privilege when it does run, and only opens one TCP socket, which is restricted to the local subnet by default. Your product runs by default as SYSTEM, opens one UDP and three TCP sockets, and they are all unauthenticated and Internet facing. Enter an attacker—which system is he going to attack? I'll give you a hint: yours! If there is a flaw in your code, he can probably compromise the server on which your code runs. Better yet, he can attack your other customers too, because they all have the functionality running by default.

Finally, let users and admins know what you did. Document the features affected by the reduced attack surface and explain how customers can tweak the system if they run into problems because of these settings. Be conservative in your recommendations and make sure your customers know the implications of enabling a port or feature. If your application relies on a firewall and users want to access some functionality using a specific port, just tell them which port to open; by all means don't tell them to turn the firewall off!

Action: If you have a large attack surface, you must spend more time making sure the code is excellent quality, conservative, and defensive. A large attack surface says: "Hackers, come get me!"

Other Resources

Jon Pincus (Microsoft Research), Jeanette Wing (Carnegie Mellon University), and I authored an academic paper entitled "Measuring Relative Attack Surfaces" that was presented in Proceedings of Workshop on Advanced Developments in Software and Systems Security. Also, a student of Wing's, Pratyusa Manadhata, has coauthored a paper with her, "Measuring A System's Attack Surface." Both are available at http://www-2.cs.cmu.edu/~wing.

A group of security consultants from Ernst and Young LLP wrote a paper entitled, "Using Attack Surface Area and Relative Attack Surface Quotient to Identify Attackability of Windows Server 2003" about the use of the relative attack surface analysis technique.

Michael Howard is a senior program manager, a founding member of the Secure Windows Initiative group at Microsoft, and a coauthor of Writing Secure Code (Microsoft Press, 2002). He focuses on designing, building, testing, and deploying applications that are designed to withstand attack, yet are still useful to millions of nontechnical users.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


App Lockdown: Defend Your Apps and Critical User Info with Defensive Coding Techniques

App Lockdown

Defend Your Apps and Critical User Info with Defensive Coding Techniques

Kenny Kerr
This article discusses:
  • Securing user credentials and logon information
  • Protecting client and server data using encryption
  • Defending COM clients and servers
This article uses the following technologies:
C++, COM, Security, C#, .NET
Code download available at: AppLockdown.exe (165KB)
Contents
Protecting Credentials
Protecting Data on the Client
Protecting Named Pipe Clients
Protecting COM Clients
Protecting Data on the Server
Defending Named Pipe Servers
Defending COM Servers
Conclusion

I
n today's connected world in which every application is a potential target, you must extend your defensive programming efforts to cover security. Everything you have learned about defensive programming helps you write more secure code, but this is not enough. You need to go much further to build explicit defenses into your software.

In this article, I'll focus on protecting users, securing their credentials and private information, and defending servers. I will cover a wide range of common programming scenarios and explore practical ways in which you can write code that is more resistant to attack.

Protecting Credentials

Let's start with how you can protect your users' credentials. For best credentials security, you should avoid managing user credentials explicitly because passwords need to be kept secret and secrets are hard to secure. Therefore, you should rely on logon sessions and single sign-on. If you need to run an application as a different user, use the shell's Run As option or type runas at the command prompt. However, when your back is to the wall and you must manage credentials yourself, here's what you should do.

First, you will need to install the Platform SDK from February 2003 or later to use the functions described here. Once installed, add the Platform SDK directories named Include and Lib to your Visual C++® search paths. If you prefer to use P/Invoke from C# or Visual Basic® .NET, you do not need to install the Platform SDK.

The next step is to prompt the user for credentials. To provide a consistent user experience and write less security-related code, you should use the CredUIPromptForCredentials function for GUI applications, and CredUICmdLinePromptForCredentials for console applications. Both provide essentially the same functionality, with the GUI version providing a few more options specific to a graphical interface such as allowing you to change the default banner bitmap, the caption, and the dialog box message. These functions can be used to prompt the user for Windows® domain credentials or generic credentials, as illustrated in Figure 1.

Figure 1 Using CredUIPromptForCredentials (Close Figure 1)
std::wstring target = L"server";
std::vector<wchar_t> userName(CREDUI_MAX_USERNAME_LENGTH + 1);
std::vector<wchar_t> password(CREDUI_MAX_PASSWORD_LENGTH + 1);

CREDUI_INFO info = { sizeof (CREDUI_INFO) };
info.hwndParent = 0; // set to the parent window

DWORD result = ::CredUIPromptForCredentials(&info,
    target.c_str(), 0 /*reserved*/, 0 /*auth error not used*/,
    &userName[0], static_cast<ULONG>(userName.size()),&password[0],
    static_cast<ULONG>(password.size()), 0 /*ignore save state*/,
    CREDUI_FLAGS_GENERIC_CREDENTIALS);
if (NO_ERROR == result) {
    // User chose OK.
} else if (ERROR_CANCELLED == result) {
    // Use chose Cancel.
} else {
    // An error occurred.
}

Credentials are stored in the user profile using the target name as the key, so be sure to use meaningful and unique target names. The CREDUI_INFO structure allows you to specify the handle to the parent window for the modal dialog box. It also lets you override the default caption, message text, and banner bitmap that will be displayed to the user.

Then there is the matter of integer overflow. Did you notice the two static_cast operations? These are used to convert the result of the std::vector<wchar_t>::size method to ULONG, which is what the CredUIPromptForCredentials function expects. Although ULONG will always be a 32-bit unsigned integer, size_t is big enough to span the full range of a pointer. In other words, size_t is a 32-bit unsigned integer on 32-bit Windows and a 64-bit unsigned integer on 64-bit Windows. The static_cast is used to suppress the compiler warning informing you that the cast may not be safe on 64-bit versions of Windows.

By suppressing the compiler warning, you are taking full responsibility to ensure that the cast will always be safe, even if run on 64-bit platforms. In the previous example, you can see that the size of the vector is less than std::numeric_limits<ULONG>::max(), so it is safe in this case. Needless to say there is a lot of room for error and that's why I wrote the PromptForCredentials class to make it easier and safer to manage credentials. The class is available in the code download for this article. The following example shows the PromptForCredentials class in use:

PromptForCredentials prompt;
prompt.Target(L"server");
prompt.ParentWindow(0); // set to the parent window
prompt.Flags(CREDUI_FLAGS_GENERIC_CREDENTIALS);

if (prompt.ShowDialog()) {
    // TODO: use prompt.UserName() and prompt.Password()
    prompt.ScrubPassword();
}

As you can see, the code is simple and takes care of error handling. The PromptForCredentials destructor will automatically call the ScrubPassword method, but it is a good idea to zero out any secrets in memory as soon as possible. Internally, the ScrubMemory method calls SecureZeroMemory (which is just a substitute for the RtlSecureZeroMemory function) in order to zero out the password buffer.

Figure 2 Prompt for Credentials Sample
Figure 2 Prompt for Credentials Sample

To allow you to experiment more easily with the wide range of options and flags provided by the CredUIPromptForCredentials function, I wrote the Prompt for Credentials sample application displayed in Figure 2. It allows you to combine the different options and flags to find the functionality you need.

Protecting Data on the Client

Protecting sensitive data involves encrypting the data while it is held in memory and before it's written to disk. If the data is persisted, it must also be protected by a strong access control list (ACL). This section will show you how to perform these tasks correctly.

With good cryptography libraries such as the CryptoAPI, CAPICOM, and System.Security.Cryptography freely available, writing encryption code is pretty easy, but managing encryption keys and passwords remains a challenge. Windows XP and Windows Server™ 2003 include data protection functions that allow you to encrypt and decrypt data without having to manage encryption keys. You can use the CryptProtectData function to encrypt data that can then only be decrypted using the CryptUnprotectData function when called from a logon session based on the same user account. Figure 3 illustrates an example.

Figure 3 Using CryptProtectData to Encrypt Data (Close Figure 3)
std::vector<BYTE> buffer;

// TODO: get secret from user into the buffer

DATA_BLOB plaintext =  { static_cast<DWORD>(buffer.size()), &buffer[0] };
DATA_BLOB ciphertext = { 0 };

CheckError(::CryptProtectData(&plaintext, 0 /*no description*/,
    0 /*no additional entropy*/, 0 /*reserved*/, 0 /*no prompt*/,
    CRYPTPROTECT_UI_FORBIDDEN, &ciphertext));

// TODO: write the ciphertext to disk

::SecureZeroMemory(&buffer[0], buffer.size());
::LocalFree(ciphertext.pbData);

CheckError is a helper function that checks the return value of a function. If the return value indicates failure, an appropriate exception is thrown. CheckError, along with many other useful functions and classes, can be found in the download for this article. As you can see, there is no need to manage an encryption key. First, you need to get the secret from the user—for example, use the GetWindowTextLength and GetWindowText functions to read some text from a control into the buffer. Next, the plaintext DATA_BLOB structure is populated to describe the location and size of the data to the CryptProtectData function. Internally, CryptProtectData allocates enough memory to hold the encrypted data using the LocalAlloc function, and returns both a pointer to this memory and the size in the ciphertext DATA_BLOB structure. At this point, you can write the secret to disk if necessary and free the memory using the LocalFree function (you should also remember to scrub the plaintext secret as soon as possible using the SecureZeroMemory function). Decryption is performed using the CryptUnprotectData function, which is virtually identical to CryptProtectData. The first parameter points to a DATA_BLOB structure referring to the ciphertext, and the last parameter points to a DATA_BLOB structure for which the function will allocate memory using LocalFree to hold the plaintext secret.

Managing DATA_BLOB structures is a cumbersome task that can result in errors if you forget to free and scrub the memory where it is appropriate to do so. It is also difficult to use in the face of exceptions. The DataBlob class shown in Figure 4 can be used to simplify these issues.

Figure 4 DataBlob Class (Close Figure 4)
namespace Kerr {
    class DataBlob : public DATA_BLOB {
    public:
        enum Flags {
            FreeMemory = 0x0001,
            ScrubData  = 0x0002
        };

        explicit DataBlob(DWORD flags) : m_flags(flags) {
            cbData = 0;
            pbData = 0;
        }

        explicit DataBlob(
          std::vector<BYTE>& buffer, DWORD flags=ScrubData) :
                   m_flags(flags) {
            if (std::numeric_limits<DWORD>::max() < buffer.size()) {
                AtlThrow(E_INVALIDARG);
            }

            if (FreeMemory & m_flags) {
                AtlThrow(E_INVALIDARG);
            }

            cbData = static_cast<DWORD>(buffer.size());
            pbData = &buffer[0];
        }

        ~DataBlob() {
            if (0 != pbData) {
                if (ScrubData & m_flags) {
                    ::SecureZeroMemory(pbData, cbData);
                }

                if (FreeMemory & m_flags) {
                    ::LocalFree(pbData);
                }
            }
        }

    private:
        DWORD m_flags;
    };
}

The DataBlob destructor will automatically scrub and free the memory where appropriate. This works great for storing data securely in the file system, but if all you need to do is protect some data in memory while the application is running, this approach can consume too much memory. You should encrypt memory in your process's address space because it is possible for the memory to be paged out to the swap file or to the hibernation file. If an attacker can compromise some other aspect of your computer's security, such as gaining physical access to it, he may be able to read your secret data.

Windows Server 2003 introduced the CryptProtectMemory and CryptUnprotectMemory functions for precisely this reason. On Windows 2000 and Windows XP you can use the RtlEncryptMemory and RtlDecryptMemory functions for the same purpose, but use caution as they may not be available on newer platforms. CryptProtectMemory and CryptUnprotectMemory are designed to efficiently encrypt a block of memory in-place. This is useful if you want to hold onto some sensitive information, such as a set of credentials, for an extended period or use them at different times. What is interesting about these functions is that the data to be encrypted must be a multiple of given block size. Encryption algorithms generally encrypt streams of data one block at a time. Data that is shorter than the block size must be padded to fill an entire block. Therefore, if you want to encrypt arbitrary data, you need to ensure that the buffer it is stored in is a multiple of the block size. The following example shows a simple function to calculate the block size for a buffer:

DWORD ToBlockSize(DWORD original) {
    DWORD result = CRYPTPROTECTMEMORY_BLOCK_SIZE;
    if (0 != original) {
        DWORD remainder = original % CRYPTPROTECTMEMORY_BLOCK_SIZE;
        result = original + (0 != remainder ?
            CRYPTPROTECTMEMORY_BLOCK_SIZE - remainder : 0);
    }
    return result;
}

Given this function, it becomes easy to use the CryptProtectMemory function. The following code shows how to read text from a control in an MFC application and encrypt it:

CWnd* pControl = GetDlgItem(IDC_SECRET);
DWORD charCount = pControl->GetWindowTextLength() + 1;

std::vector<BYTE> buffer(ToBlockSize(charCount * sizeof (WCHAR)));
pControl->GetWindowText(reinterpret_cast<PWSTR>(&buffer[0]), charCount);

if (!::CryptProtectMemory(&buffer[0], static_cast<DWORD>(buffer.size()),
           CRYPTPROTECTMEMORY_SAME_PROCESS)) {
    AtlThrowLastWin32();
}

Decrypting the data is a simple matter of calling CryptUnprotectMemory with the exact same set of parameters. The buffer will again contain the plaintext data.

So far, I have talked about encrypting data for long- and short-term storage. What I have yet to discuss is how to protect the data when it is stored in the file system or the registry. The user profile, which includes the user's My Documents folder, provides a secure location in which applications can store user-specific application data as well as documents. The user profile includes a portion of the registry for the current user as well as a branch of the file system, both of which are secured by ACLs that effectively limit access permissions to user information to the specific user and administrators. The two most useful folder locations in the user profile are the application data folder and the personal folder, also known as My Documents. The application data folder is for application data files that are specific to each user but that the user does not normally have to interact with. For example, you might want to store user preferences in a configuration file in this location. The personal folder is where the application should default to when saving documents created for the user.

To use these folders from C++, employ the SHGetFolderPathAndSubDir function. This function, introduced in Windows XP, does the work of calling SHGetFolderPath followed by PathAppend in a single function. It can also create the folder if it does not exist. Here is an example:

std::vector<wchar_t> path(MAX_PATH);
CheckError(::SHGetFolderPathAndSubDir(0 /*no parent window*/,
    CSIDL_FLAG_CREATE | CSIDL_APPDATA, 0 /*no token*/,
    SHGFP_TYPE_CURRENT, L"Kerr\\SecuritySample", &path[0]));
::PathAppend(&path[0], L"settings.xml");
CHandle file(::CreateFile(&path[0], ...));

Keep in mind that many of these Windows Shell functions do not take a buffer size argument so you need to take special care not to introduce a buffer overrun error. To create the file in the user's personal folder, simply replace the CSIDL_APPDATA flag with CSIDL_PERSONAL. The equivalent C# code is slightly less concise, but it's simpler and less error prone:

string path =
  Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData);
path = Path.Combine(path, @"Kerr\SecuritySample");
Directory.CreateDirectory(path);
path = Path.Combine(path, "settings.xml");
FileStream file = new FileStream(path, ...);

Protecting Named Pipe Clients

Named pipes are one of the simplest and most efficient forms of interprocess communication (IPC) on the Windows platform. A named pipe provides a simple one-way or duplex pipe that a client and server can use to communicate messages or byte streams. It also provides an abstraction over sockets and the Security Service Provider Interface (SSPI) to make network communications programming easier. There are, however, some nuances in how named pipes deal with authentication that can be troublesome.

In this section, I will explore named pipes from the perspective of the named pipe client. Even if you do not use named pipes directly, you may still be relying on them for some other API that you are using. For example, the Service Control Manager (SCM) functions, used to manage local and remote Windows services, use named pipes for communication. Access to a remote registry when using the RegConnectRegistry function also relies on named pipes. Having a good understanding of named pipe security will help you understand how many common Windows services work, if not your own applications.

A client connects to a named pipe using the CreateFile function. Internally, it delegates the management of the communication with the named pipe server to the Workstation service. Consider the following example:

CHandle handle(::CreateFile(L"\\\\server\\pipe\\Kerr.SecuritySample",
    FILE_READ_DATA | FILE_WRITE_DATA, 0 /*no sharing*/,
    0 /*default security attributes*/, OPEN_EXISTING,
    SECURITY_SQOS_PRESENT | SECURITY_IDENTIFICATION, 0 /*no template*/));

By requesting read and write access to the pipe, the client is assuming that the server created a duplex, or bi-directional, pipe. If this is not the case, CreateFile will fail and the GetLastError function will return ERROR_ACCESS_DENIED.

The other interesting thing to note is the Security Quality of Service (SQOS) information. When communicating with named pipes, you will typically want to combine the SECURITY_SQOS_PRESENT flag with either SECURITY_IDENTIFICATION or SECURITY_IMPERSONATION. SECURITY_IMPERSONATION is the default if you do not provide any information for SQOS. As a developer of a client application, one of your main goals should be to protect the identity of the client. This means limiting your trust in the servers to which you connect.

The SECURITY_IMPERSONATION flag indicates that the server will be able to access local resources on the client's behalf. For example, the server may want to impersonate the client and call the CreateFile function to allow the file system to perform an access check against the client's identity. If you do not want to let the server masquerade as the client, you can specify the SECURITY_IDENTIFICATION flag instead. This flag indicates that the server will be able to impersonate the client for the purposes of getting authorization information about the client, but will not be able to impersonate the client when accessing other resources.

The way the client is authenticated is also important. The CreateFile function has no provision for accepting credentials, so there must be some other way to authenticate with the named pipe server, considering it is common to require alternate credentials. This is easier to understand if you consider that named pipes are really hosted by the Server service, which, coincidentally, also manages file and print sharing. The Workstation service mentioned earlier is then simply the service that manages sessions to various Server services on different computers. This infrastructure is loosely referred to as the Windows file server.

What makes using the file server a little tricky is that for a given logon session there can be only one client session connected to any single server. If you attempt to establish a second connection with a different set of credentials, it will fail. So managing sessions becomes an important part of writing a named pipe client app.

Sessions are usually created automatically on your behalf when you use Explorer to connect to a share on a remote computer, call CreateFile to open a file residing on a different computer, or open the client end of a named pipe. These sessions are generally short lived and are closed automatically by the Workstation service some time after the client's file handle is closed. They use the identity of the logon session that initiated the connection. Sometimes, however, there is a need to use a different set of credentials.

A set of credentials can be associated with a remote machine to be used in place of any implicit sessions that may or may not be creatable. This is commonly referred to as a use record, which is helpful in creating a session to a file server with an explicit set of credentials, where the user's logon session either does not have network credentials or does not have permission to access the remote server.

To create a use record, call the NetUseAdd function. Figure 5 shows an example of using the PromptForCredentials class introduced earlier. When the use record is no longer needed, call the NetUseDel function to remove it:

CheckError(::NetUseDel(0 /*reserved*/, L"\\\\server\\IPC$", USE_FORCE));

Figure 5 Creating a Use Record (Close Figure 5)
PromptForCredentials prompt;
prompt.Target(L"server");
prompt.Flags(CREDUI_FLAGS_DO_NOT_PERSIST);

if (prompt.ShowDialog()){
    USE_INFO_2 useInfo = { 0 };
    useInfo.ui2_remote = L"\\\\server\\IPC$";
    useInfo.ui2_asg_type = USE_WILDCARD;
    useInfo.ui2_username = const_cast<PWSTR>(prompt.UserName());
    useInfo.ui2_password = const_cast<PWSTR>(prompt.Password());

    CheckError(::NetUseAdd(0 /*reserved*/, 2 /*use USE_INFO_2 structure*/,
               reinterpret_cast<PBYTE>(&useInfo), 0 /*no param index*/));
}

Protecting COM Clients

Detailed coverage of COM architecture and security would fill a book or two, so for this section I will focus on what you should understand about COM security to better protect your COM client applications. COM is still a big part of a developer's life, being at the heart of many Windows services. Some are hidden behind managed wrappers and exposed through elegant interfaces in the .NET Framework, but COM is still there. The point is that COM is going to be around for a long time.

To simplify the terminology, I will refer to code that makes use of COM objects as COM clients, and I'll refer to COM objects as COM servers, regardless of how they are activated. This is consistent with the COM specification.

The first interesting function called by a typical COM client, after selecting the apartment model, is CoCreateInstance—as shown in the following code snippet:

CheckError(::CoCreateInstance(__uuidof(CoLibraryObject),
             0 /*no outer unknown*/, CLSCTX_INPROC_SERVER,
             __uuidof(ILibraryObject),
             reinterpret_cast<PVOID*>(&spLibraryObject)));

This COM server will be created in the client process, indicated by the CLSCTX_INPROC_SERVER flag. The CLSCTX, or class context value, does not determine where the object will be activated (that is controlled by the server). Rather, this value is used to restrict where the object can be created for a given instantiation. If the class context does not match the packaging defined by the server, the call to CoCreateInstance will fail. Because the server in this example is running within the client's logon session, it can do almost anything on the client's behalf. Therefore, for this discussion, I will focus on out-of-process activation, shown here:

CheckError(::CoCreateInstance(...,
             CLSCTX_LOCAL_SERVER | CLSCTX_REMOTE_SERVER, ...));

CLSCTX_LOCAL_SERVER indicates that the COM client directs the COM server to run out of process. You can also specify CLSCTX_ALL, which effectively indicates that the COM client does not care where the COM server will run. This is the default for the CComPtr<T>::CoCreateInstance method. In order to avoid any surprises you should always be explicit about where you expect the COM server to run.

In the previous example, the COM client provided the client's process identity to the COM server for the purposes of authentication and authorization. In this case, the COM server will get a token for the client's logon session. If the COM server were activated on a remote computer, a network logon session would have been created to represent the client.

If the client is impersonating—implying that there is a thread token—and you want to provide the client's thread identity to the COM server, you must enable dynamic cloaking. You can enable it for the entire process using the CoInitializeSecurity function, but this is often undesirable as it affects all the COM clients in the process. A better option is to enable dynamic cloaking for a given COM interface proxy. You can do this with the CoSetProxyBlanket function:

CheckError(::CoSetProxyBlanket(spServerObject, RPC_C_AUTHN_DEFAULT,
               RPC_C_AUTHZ_NONE, COLE_DEFAULT_PRINCIPAL,
               RPC_C_AUTHN_LEVEL_DEFAULT, RPC_C_IMP_LEVEL_DEFAULT,
               COLE_DEFAULT_AUTHINFO, EOAC_DYNAMIC_CLOAKING));

Most of the parameters just tell COM to pick an appropriate default. The parameters of interest are the first parameter, which specifies the proxy to configure, and the last parameter, which specifies the EOAC_DYNAMIC_CLOAKING capability. For the lifetime of the proxy, before every subsequent method call, the proxy will determine the effective token and use it when connecting to the COM server. This might sound inefficient at first, but it isn't. Internally the proxy's channel object holds onto a remote procedure call (RPC) binding handle, since distributed COM uses RPC under the covers. The binding handle manages the authentication information used to connect to a particular server. As long as the effective token does not change, the same binding handle will be reused on subsequent method calls.

Another option is to use a different set of credentials for connecting to a remote computer. This may be useful if the client cannot establish a network logon session on that computer, for example if the computer is not part of a trusted domain. Explicit credentials are provided with a COAUTHIDENTITY structure. This structure can be used during the initial activation request to connect to the remote computer using the CoCreateInstanceEx function. It can also be used when setting the authentication information for a proxy using the CoSetProxyBlanket function. You can provide credentials independently for either or both. If you use activation credentials, it will have no effect on proxies. The proxies will still use the process token under normal conditions or the effective token if you enable dynamic cloaking. Looking at this a different way, regardless of whether you use different credentials for activation and for the proxy, two distinct logon sessions will be created on your behalf.

To protect myself from common buffer-related errors, I wrote the CoAuthIdentity class shown in Figure 6 for populating COAUTHIDENTITY structures. This class takes care of all the error handling when dealing with string buffers and especially passwords. Calling CoCreateInstanceEx is now relatively simple, as shown in Figure 7.

Figure 7 Calling CoCreateInstanceEx (Close Figure 7)
CoAuthIdentity authIdentity(L"principal@authority", L"G&m25!YIp");

COAUTHINFO authInfo = { 0 };
authInfo.dwAuthnSvc = RPC_C_AUTHN_WINNT;
authInfo.dwAuthzSvc = RPC_C_AUTHZ_NONE;
authInfo.dwImpersonationLevel = RPC_C_IMP_LEVEL_IMPERSONATE;
authInfo.dwCapabilities = EOAC_NONE;
authInfo.pAuthIdentityData = &authIdentity;

COSERVERINFO serverInfo = { 0 };
serverInfo.pAuthInfo = &authInfo;
serverInfo.pwszName = L"server";

MULTI_QI multiQI = { &__uuidof(IServerObject) };

CheckError(::CoCreateInstanceEx(__uuidof(CoServerObject),
               0 /*no outer unknown*/, CLSCTX_REMOTE_SERVER,
               &serverInfo, 1 /*one MULTI_QI structure*/, &multiQI));

CComPtr<IServerObject> spServerObject;
spServerObject.Attach(static_cast<IServerObject*>(multiQI.pItf));
Figure 6 CoAuthIdentity Class (Close Figure 6)
class CoAuthIdentity : public COAUTHIDENTITY {
public:

    CoAuthIdentity(PCWSTR pUserName, PCWSTR pPassword) {
        // Zero out the COAUTHIDENTITY fields
        ::ZeroMemory(static_cast<COAUTHIDENTITY*>(this),
                     sizeof (COAUTHIDENTITY));

        // Get the character lengths of the parameters
        size_t userNameLength = 0;
        CheckError(::StringCchLength(pUserName, STRSAFE_MAX_CCH,
                                     &userNameLength));

        size_t passwordLength = 0;
        CheckError(::StringCchLength(pPassword, STRSAFE_MAX_CCH,
                                     &passwordLength));

        // Split the user name into authority-principal parts
        const std::wstring userName(pUserName,
                                    pUserName + userNameLength);
        UserName::Split(userName, &m_principal, &m_authority);

        // Copy the password to the local buffer
        m_password.resize(passwordLength + 1);
        std::copy(pPassword, pPassword + passwordLength,
                  m_password.begin());

        // Populate the COAUTHIDENTITY fields
        Flags = SEC_WINNT_AUTH_IDENTITY_UNICODE;
        User = const_cast<PWSTR>(m_principal.c_str());
        UserLength = static_cast<ULONG>(m_principal.size());
        Domain = const_cast<PWSTR>(m_authority.c_str());
        DomainLength = static_cast<ULONG>(m_authority.size());
        Password = &m_password[0];
        PasswordLength = static_cast<ULONG>(m_password.size() - 1);
    }

    ~CoAuthIdentity() {
        if (!m_password.empty()) {
            ::SecureZeroMemory(&m_password[0], m_password.size() *
                sizeof (std::vector<wchar_t>::value_type));
        }
    }

private:
    std::wstring m_principal;
    std::wstring m_authority;
    std::vector<wchar_t> m_password;
};

Keep in mind that you should never hard-code credentials, and especially passwords, in your code as I've done here. I use literal string passwords in the samples for this article only because it makes the samples simpler and more to the point.

A COSERVERINFO structure is passed to CoCreateInstanceEx, providing the name of the remote computer as well as the authentication information in the form of a COAUTHINFO structure. The COAUTHINFO structure in the example is configured to use Windows NT LAN Manager (NTLM) authentication. This is a safe bet, as you should expect NTLM to always be available. One reason that you may not want to use NTLM, however, is that it does not verify the identity of the server, which I will discuss later.

Using RPC_C_IMP_LEVEL_IMPERSONATE as the impersonation level is necessary because CoCreateInstanceEx defers to the local COM SCM to communicate with the remote SCM for the purposes of remote activation. You need to provide the SCM with this level of trust so that it can establish a remote network logon session on your behalf.

As I mentioned, the credentials used for the activation request do not affect the proxy that will be returned by CoCreateInstanceEx. To have the proxy's underlying binding handle authenticate with the same credentials, simply pass the COAUTHIDENTITY structure or CoAuthIdentity object to the CoSetProxyBlanket method, as shown in the following:

CheckError(::CoSetProxyBlanket(..., &authIdentity, EOAC_DEFAULT));

Keep in mind that the proxy may refer to the COAUTHIDENTITY structure at any time until it is released, so you need to ensure that the memory does not change or become freed before all proxies are released.

So far, I've focused on how COM clients manage authentication with COM servers. The next important aspect of COM security is the level of trust you place in the server. By connecting to the COM server, the COM client allows the server to impersonate the client. What the server is allowed to do with the resulting thread token is limited by what the client allows. Although this can be specified at the machine or process level, I recommend always specifying your preferences for a given proxy. Again, let's turn to the CoSetProxyBlanket function, as shown here:

CheckError(::CoSetProxyBlanket(..., RPC_C_IMP_LEVEL_IMPERSONATE, ...));

The relevant options for the dwImpLevel parameter are RPC_C_IMP_LEVEL_IDENTIFY, RPC_C_IMP_LEVEL_IMPERSONATE, and RPC_C_IMP_LEVEL_DELEGATE.

Let's first tackle the RPC_C_IMP_LEVEL_DELEGATE option. Delegation refers to the ability of a server to take a client's network credentials and use them to establish another network logon session on a different machine. Clearly, this will only work if the network logon session actually caches the client's credentials. NTLM does not support delegation, but Kerberos does. Interestingly, if a client uses the RPC_C_IMP_LEVEL_DELEGATE flag to access a local COM server, delegation will be supported despite the fact that NTLM is used for local authentication. This is possible because when the COM client connects to a local COM server, a network logon session is not created (as is the case for remote COM servers), but rather the client's logon session is used. Because the client's logon session will typically have network credentials, delegation effectively works. Considering the power that a server can wield with a client's delegated credentials, you need to carefully consider its use.

The RPC_C_IMP_LEVEL_IDENTIFY flag has the same effect as the SECURITY_IDENTIFICATION flag used by named pipe clients. The server will be able to query the client's token for information that can be used to identify the client and make authorization decisions. The server will not, however, be able to impersonate the client in order to access other secure resources such as the file system.

RPC_C_IMP_LEVEL_IMPERSONATE has the same effect as the SECURITY_IMPERSONATION flag used by named pipe clients and is a superset of the RPC_C_IMP_LEVEL_ IDENTIFY flag. In addition to being able to query the client's token, the server can impersonate the client while accessing local resources.

Once the client has a proxy and has configured suitable authentication, the client can finally get to work and call the methods on the COM object. The security service provider (SSP) can also provide protection for the data communicated between the client and the server. Again, using only the defaults will not help you get a good night's sleep.

The authentication level controls data protection on the wire and optionally provides two services: integrity and privacy. Although there are a number of authentication levels, the only two that are useful for the defensive security programmer are RPC_C_AUTHN_LEVEL_PKT_INTEGRITY and RPC_C_AUTHN_LEVEL_PKT_PRIVACY. The latter provides complete protection that covers tamper detection and encryption of all data, including protocol headers. RPC_C_AUTHN_LEVEL_PKT_INTEGRITY provides the same degree of tamper detection by signing the data with the session key, but the data is visible to eavesdroppers. You should favor packet privacy, unless the performance overhead of sealing the entire communication is prohibitive. You may find that it does not have a noticeable effect on performance, and you will make it that much harder for hackers to attack your software.

In the discussion about COM authentication, I hinted that NTLM does not authenticate the server. Windows 2000 introduced the Kerberos network authentication protocol, and compared to NTLM, Kerberos is quite modern and sophisticated. One of its many distinguishing characteristics is that Kerberos supports mutual authentication and can ensure server identity. To use Kerberos, not only do you need to use Windows 2000 or later, but the computers involved need to be part of a Kerberos domain, sometimes referred to as a Windows 2000 domain to distinguish it from the vastly different Windows NT® domain model. Assuming these requirements are met and all the planets are aligned correctly, COM makes it relatively simple to request Kerberos authentication with the COAUTHINFO structure. One thing to keep in mind is that if you are trying to get Kerberos to work, make sure the COM server's identity has network credentials. If you launch your COM server under the Local Service account, for example, the server identity will not have any network credentials to authenticate.

Protecting Data on the Server

Frequently, a server application needs to store sensitive configuration information such as database connection strings or, in an extreme case, some cached user credentials. This data needs to be stored securely, and a best practice involves the use of a strong ACL to protect the data on disk or in the registry. An ACL is associated with a file or registry key by means of a security descriptor. Security descriptors are used to store per-object security information to support the object-centric security model that is at the heart of Windows security programming.

A security descriptor stores two things of interest: the first is the owner's security identifier (SID) that identifies the owner of the object, and the second is a discretionary access control list (DACL) that lists principals and groups that are granted or denied access to the object and the permissions assigned to each.

Creating a security descriptor from scratch has traditionally involved writing a lot of code. To create the DACL you need to call the InitializeAcl function to create an initially empty ACL, and then call AddAccessAllowedAceEx and AddAccessDeniedAceEx functions on each SID for which you want to specify permissions. Of course, before you can do this you need to actually retrieve or create the SIDs representing the various principals and groups. Then you need to create the actual security descriptor by calling InitializeSecurityDescriptor followed by SetSecurityDescriptorDacl to set the previously created ACL as the DACL for the security descriptor. This can be an error-prone task. One solution is to create some helpful wrapper functions and classes to hide some of this complexity. An even better solution is to use a helper function introduced with Windows 2000 to create security descriptors from a string. The ConvertStringSecurityDescriptorToSecurityDescriptor function takes a specially formatted string and constructs a complete security descriptor for you. Here is an example:

LocalMemory<PSECURITY_DESCRIPTOR> pSecurityDescriptor;

const std::wstring definition = L"O:BAD:(A;;FA;;;BA)(A;;FR;;;NS)";

CheckError(::ConvertStringSecurityDescriptorToSecurityDescriptor(
    definition.c_str(),SDDL_REVISION_1, &pSecurityDescriptor.m_ptr, 0));

The LocalMemory template class automatically frees the owned memory in its destructor using the LocalFree function. I often use the LocalMemory class when doing security programming because many security functions allocate memory with LocalAlloc. The LocalMemory class is available along with the sample code for this article. The O: and D: tokens in the string denote the owner and DACL sections in the security descriptor string definition. The owner is classified as BA, which represents the built-in Administrators group. The DACL is described by the (A;;FA;;;BA)(A;;FR;;;NS) segment, which represents two access control entries (ACEs). The A in both indicates that they are used to allow access. Denying access is represented by D. FA and FR indicate the permissions granted, where FA represents FILE_ALL_ACCESS and FR represents FILE_GENERIC_READ. BA and NS indicate the SIDs being granted the access. As I mentioned previously, BA indicates the Administrators group and NS represents the Network Service account.

For a detailed description of the string format, consult the Security Descriptor String Format documentation in the MSDN® Library.

Now that you have a basic understanding of how to construct a security descriptor, let's proceed to secure a newly created file. When you call the CreateFile function to create a new file, you can specify a security descriptor as well. The security descriptor is specified in the SECURITY_ATTRIBUTES structure passed to CreateFile. The catch is that if the file already exists, the security descriptor will be ignored. This is a serious risk because an attacker could have created a file with a weak ACL before your application had a chance to create the file. If you do not write code to defend against this sort of attack, you may leave sensitive data unprotected and vulnerable to unauthorized access. To avoid this threat, check the result of the GetLastError function after a successful call to CreateFile. If it returns the ERROR_ALREADY_EXISTS error code, you know that the file already existed and you can then attempt to apply the security descriptor once again. This is illustrated by the code in Figure 8.

Figure 8 Creating a File with a Security Descriptor (Close Figure 8)
SECURITY_ATTRIBUTES sa = { sizeof (SECURITY_ATTRIBUTES) };
sa.lpSecurityDescriptor = pSecurityDescriptor.Get();

CHandle file(::CreateFile(L"security.txt",
                     FILE_WRITE_DATA | READ_CONTROL | WRITE_DAC |
                     WRITE_OWNER,
                     0 /*no sharing*/, &sa, CREATE_ALWAYS,
                     FILE_ATTRIBUTE_NORMAL, 0 /*no template*/));

if (INVALID_HANDLE_VALUE == file.m_h) {
    AtlThrowLastWin32();
}

if (ERROR_ALREADY_EXISTS == GetLastError()) {
    PSID pOwnerSid = 0;
    BOOL ownerDefaulted = FALSE;

    CheckError(::GetSecurityDescriptorOwner(pSecurityDescriptor.Get(),
                                            &pOwnerSid, &ownerDefaulted));

    PACL pDacl = 0;
    BOOL daclPresent = FALSE;
    BOOL daclDefaulted = FALSE;

    CheckError(::GetSecurityDescriptorDacl(pSecurityDescriptor.Get(),
                   &daclPresent, &pDacl, &daclDefaulted));

    CheckError(::SetSecurityInfo(file.m_h, SE_FILE_OBJECT,
                   OWNER_SECURITY_INFORMATION | DACL_SECURITY_INFORMATION,
                   pOwnerSid, 0, pDacl, 0));
}

The SetSecurityInfo function is used to update the object and DACL for the file. You may have noticed that I had to include a few more access rights to allow me to update the security descriptor through the file handle returned by CreateFile. These permissions are only used if the file's security descriptor needs to be updated. You should always limit the permissions you request to the absolute minimum. A better solution that avoids requesting all these permissions is to use the SetNamedSecurityInfo function instead of SetSecurityInfo. Instead of passing a handle to the file as the first parameter, you pass the full path to the file as a string. The function will internally open the file with the appropriate access rights, update the security descriptor, and then close its file handle.

Defending Named Pipe Servers

The file protection techniques I explored in the previous section can also be used to protect named pipe servers as well as other types of servers. A named pipe server is typically implemented as a Windows service. The service's main listening thread calls CreateNamedPipe to create an instance of a named pipe. It then calls the ConnectNamedPipe function to wait for a client process to connect to it. Once a client has connected to the pipe, the server can queue the connection to a worker thread and create a new instance of the named pipe to listen for the next client. Here is a basic use of the CreateNamedPipe function:

CHandle handle(::CreateNamedPipe(L"\\\\.\\pipe\\Kerr.SecuritySample",
                 PIPE_ACCESS_DUPLEX | FILE_FLAG_OVERLAPPED,
                 PIPE_TYPE_BYTE, PIPE_UNLIMITED_INSTANCES,
                 0 /*default output buffer size*/,
                 0 /*default input buffer size*/, 0 /*default time-out*/,
                 0 /*no security attributes*/));

This code will create a new instance of the Kerr.SecuritySample pipe for bidirectional communication and asynchronous input and output on the server end. This is simple enough, but who is allowed to access this pipe? If you call the GetSecurityInfo function on the returned handle to get the pipe's security descriptor, you can then pass it to the ConvertSecurityDescriptorToStringSecurityDescriptor function to get the security descriptor in string format. Using the OWNER_SECURITY_INFORMATION and DACL_SECURITY_INFORMATION flags, you should get something like the following line of code:

O:NSD:(A;;FA;;;SY)(A;;FA;;;BA)(A;;FA;;;NS)(A;;FR;;;WD)(A;;FR;;;AN)

Figure 9 gives you a breakdown of the string. Clearly, this is not a very strong ACL—you should never include the Anonymous and Everyone aliases in an ACL. A better approach is to provide a security descriptor explicitly. The security descriptor is specified in the SECURITY_ATTRIBUTES structure passed as the last parameter to the CreateNamedPipe function.

Figure 9 Security Descriptor String (Close Figure 9)

0: (Owner)
Code | Description
NS | Network Service
D: (DACL)
Code | Description
(A;;FA;;;SY) | Allow FILE_ALL_ACCESS Local System
(A;;FA;;;BA) | Allow FILE_ALL_ACCESS Local Administrators
(A;;FA;;;NS) | Allow FILE_ALL_ACCESS Network Service (Owner)
(A;;FR;;;WD) | Allow FILE_GENERIC_READ Everyone
(A;;FR;;;AN) | Allow FILE_GENERIC_READ Anonymous

Another important step in improving your server defenses is to ensure that you are indeed the creator of the named pipe. The first call to CreateNamedPipe with a unique pipe name is said to be the creator and thus the owner of the pipe. Subsequent calls result in new instances of the existing pipe object. A hacker could determine your pipe name and attempt to create the named pipe object before the application does. Then, being the creator, he or she will have sufficient permissions to access any pipe instances freely.

In order to avoid this type of attack, you can use the FILE_FLAG_FIRST_PIPE_INSTANCE access mode flag. If CreateNamedPipe determines that the named pipe already exists, it will fail.

Depending on your requirements, you may need to create additional named pipes for individual clients. For these pipes, you can create a more restrictive security descriptor that only allows individual clients access to the pipe. For this and other reasons, it is often useful to get the token representing the client's logon session. The ImpersonateNamedPipeClient function places the client's token on the current thread. Assuming the client used the SECURITY_IMPERSONATION flag (discussed under Protecting Named Pipe Clients) when connecting to the client end of the pipe, the server will now be able to access local resources on behalf of the client. The following code snippet is a simple class that you can use to place the client's token on the current thread:

class SafeImpersonateNamedPipeClient {
public:
    explicit SafeImpersonateNamedPipeClient(HANDLE pipe) {
        CheckError(::ImpersonateNamedPipeClient(pipe));
    }
    ~SafeImpersonateNamedPipeClient() {
        BOOL success = ::RevertToSelf();
        ATLASSERT(success);
    }
};

If the client uses the SECURITY_IDENTIFICATION flag instead, the server will not be able to use the thread token for accessing resources, but it will, however, be able to query the token for the client's group membership, privileges, and so on, which can be used for authorization. To get the token, call the OpenThreadToken function. The SafeImpersonateNamedPipeClient destructor will automatically take care of restoring the server's security context. With the help of this class, it becomes simple to obtain the client's token, as shown here:

CHandle token;
{
    SafeImpersonateNamedPipeClient impersonate(handle);
    CheckError(::OpenThreadToken(::GetCurrentThread(),
                                 TOKEN_QUERY, TRUE, &token.m_h));
}

When execution leaves the inner scope, the RevertToSelf function will be called and the token variable will hold a copy of the client's token. A common misconception is that impersonation means that the server can perform tasks acting as the client. This is untrue. In Windows security parlance, impersonation simply refers to the act of placing a token for the client's logon session on the current thread. Whether or not the server can access resources on behalf of the client is dependent on authentication settings defined by the client. If you attempt to access the file system using the CreateFile function while impersonating a client for example, it will fail unless the client has granted you the rights to do so.

Now that you have a token, you can interrogate it using the GetTokenInformation function to perform any authorization of your own. Alternatively, you can manage your own private security descriptors for your application and use the AccessCheck function to take care of implementing a consistent authorization model.

Although named pipes support signing, thus providing a level of data integrity, they do not support privacy. Signing data is not even enabled by default and is controlled by a machine-wide setting. For these reasons, do not use pipes for communicating sensitive information. If you must use a pipe, first establish a shared key using some other form of interprocess communication that does provide data integrity and privacy, and then use a symmetric algorithm to encrypt the data communicated over the pipe.

Defending COM Servers

For COM clients, I suggested being explicit about your security demands and ignoring machine and process defaults. Server applications, on the other hand, are generally better off letting administrators influence their security configuration. It is still important to be conscious of the level of security in use, but rather than simply overriding the declarative security settings, you should start by installing your server with secure defaults. An administrator can then modify those security defaults as needed. In this section, I'm going to focus on those aspects of security that relate to classic COM servers as well as COM+ server applications.

There are two distinct points of access control for a COM server. The first is controlled by launch or activation permissions and the second is controlled by access permissions. Classic COM servers are typically caller activated, meaning that the process that hosts the COM server is started when the client first activates the COM object—for example, using the CoCreateInstanceEx function. The ability for a client to activate a COM server is thus controlled by activation permissions, which are controlled at the COM application level and expressed in the form of a security descriptor stored in the COM application's registry key.

Administrators can manage launch permissions through the Component Services Microsoft® Management Console (MMC) snap-in. If launch permissions are not set at the application level, machine-wide defaults are used. These, too, can be edited using the Component Services snap-in.

The second point at which access control occurs is when a client makes a call to the COM server through a proxy. The proxy's authentication settings are communicated to the stub on the server, which can then determine whether the client has permissions to access the object. This is called access control, which is also managed at the COM application level through the DCOM Config node of the Component Services snap-in. A server can also override access permissions by calling the CoInitializeSecurity function when it first starts. The first parameter to this function takes a security descriptor which, if present, provides the effective ACL that all RPC calls into the server are then measured against.

Although the client specifies the level of authentication information provided to the server, the server can demand a minimum quality of service from the client. This is controlled by the authentication level. In the Protecting COM Clients section, I discussed the authentication level that the client will use since it is a choice the client makes. The server can, however, demand a minimum level of authentication. This is typically used to ensure data integrity and privacy if needed. You should employ the packet integrity and packet privacy levels, favoring packet privacy, as it provides tamper detection as well as encryption of all communication. The authentication level can also be controlled for a given COM application using the Component Services snap-in or using the CoInitializeSecurity function.

Once the client has successfully navigated past launch and access permissions, the server can use the CoImpersonateClient function to place the client's token on the current thread. When you have finished impersonating, call the CoRevertToSelf function to restore the server's security context. Both of these functions are simply wrappers around the IServerSecurity interface, which you can retrieve directly using the CoGetCallContext function. What the server can do with the user token is subject to the impersonation level controlled by the client. Using the client's token is the same as for named pipe servers, using the OpenThreadToken function to get the token from the thread, and so on.

COM+ applications break from the traditional Windows object-centric security model of access control lists to a model where access is granted to objects and methods based on role membership. These roles are distinct from Windows groups and are managed declaratively using the Component Services snap-in. An application developer defines the roles for a given application, and pre-populates them with conservative defaults during installation. The server administrator can then modify these default role assignments as needed. The nice thing about this model is that security is managed declaratively—you do not need to do anything special in the code to take advantage of role-based security. If a client attempts to call a method that it does not have access to, based on its role assignment, the call will be rejected before entering the COM object.

COM introduced the concept of a call context for representing the current call into an object and specifically the direct caller. You can access the call context through the IServerSecurity interface returned by CoGetCallContext. COM+ extended the notion of call context to provide more information about the call context including information about intermediate security contexts in the call chain. It also provides the ability to programmatically check role membership for fine-grained control. This functionality is exposed through the ISecurityCallContext interface, which can also be retrieved using the CoGetCallContext function. The following code snippet illustrates how to determine if the direct caller is in a given role:

bool IsCallerInRole(const CComBSTR& role) {
    CComPtr<ISecurityCallContext> spCallContext;
    CheckError(::CoGetCallContext(IID_ISecurityCallContext,
               reinterpret_cast<PVOID*>(&spCallContext)));

    VARIANT_BOOL inRole = VARIANT_FALSE;
    CheckError(spCallContext->IsCallerInRole(role, &inRole));
    return VARIANT_FALSE != inRole;
}

COM+ services are also exposed through the .NET Framework in the System.EnterpriseServices namespace, though more and more of its features and services are being provided natively by the common language runtime, ASP.NET, and Web services.

Conclusion

The Windows programming landscape covers many technologies that can be combined to provide new and exciting applications. All of these technologies and applications must be secured to provide reliable protection for users and organizations providing services. For every type of client and server technology you employ, you need to understand the security model that is used to authenticate, authorize, and protect users and their data.

Kenny Kerr designs and builds distributed apps for Windows. He also has a particular passion for C++ and security programming. Reach Kenny at http://weblogs.asp.net/kennykerr or visit his Web site at http://www.kennyandkarin.com/kenny.


Cryptography: Employ Strong Encryption in Your Apps with Our CryptoUtility Component

Cryptography

Employ Strong Encryption in Your Apps with Our CryptoUtility Component

Michael Stuart and J Sawyer
This article discusses:
  • A handy cyptographic utility
  • Background on cryptographic algorithms
  • Protecting your keys
  • Potential pitfalls
This article uses the following technologies:
Visual Basic, COM+, .NET
Code download available at: CryptoUtility.exe (1,491KB)
Contents
CryptoUtility Architecture
Cryptographic Decisions
How Symmetric Algorithms Work
Options for Symmetric Algorithms
Creating Strong Keys
Creating Strong Salt
"Hygienic" Cryptography
Storing the Key
Protecting the Key with ACLs
Protecting the Key with DPAPI
Mutual Authentication and Key Obfuscation
Protecting the Crypto Application Itself
COM Security
Putting It All Together
Conclusion

W
e've all been warned not to store secret information on computers. You know that storing passwords, credit card numbers, or Social Security numbers on your system is risky and can open the door for someone to steal that information. However, you may well have the need to store such sensitive information on computer systems.

Let's say you're running a bank payment processing system. Such a system needs to store credit card numbers to process recurring charges, to reverse charges, and to perform account audits. In this scenario, a non-reversible methodology (such as a hash algorithm applied to a password) is simply not appropriate. Additionally, both scenarios require encryption and decryption on multiple, independent machines.

In this article, we will examine some of the issues involved with developing strong encryption components for applications. The components will be usable in scenarios like the one we described. In addition, we have some other goals for a program we created called CryptoUtility. First, we want the highest possible level of security that is practical for server-side Web applications. Second, setup should be relatively easy from an administrator's point of view, allowing deployment to multiple servers without wading through 10 pages of installation instructions. Third, reversible encryption (which we'll get into later) must be available, with the keys protected and stored securely. Of course, since this is a server application, it needs to be highly scalable and run for months without administrative babysitting or rebooting. Finally, this component should be accessible to legacy COM applications (such as classic ASP) as well as .NET-based applications (such as Web services and Web applications running on ASP.NET).

There are potential vulnerabilities of some kind in every application. With something that handles extremely sensitive information, these dangers are magnified, and you'll need to understand them as thoroughly as possible. You must determine how identifiable threats interact with each other and how several independent issues could combine to constitute a vulnerability. With this knowledge, you can identify security hazards, prioritize their potential peril, and decide on acceptable levels of risk.

We'll explore threats using the STRIDE and DREAD models, as outlined by Michael Howard in Writing Secure Code, Second Edition (Microsoft Press®, 2003). STRIDE is a model to categorize threats; it stands for Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Elevation of privilege. DREAD is a model to calculate the total risk associated with a threat; it stands for Damage potential, Reproducibility, Exploitability, Affected users and Discoverability. Threat modeling and analysis are beyond the scope of this article, but we'll refer to terms associated with the STRIDE and DREAD models as we discuss system security measures.

CryptoUtility Architecture

Architectural decisions affect every solution. Our first decision point is whether to implement CryptoUtility as an in-process DLL, a Windows® service, or a Serviced Component using Enterprise Services. Developing the component as an in-process DLL presents several security issues. First, the CryptoUtility runs as the ASP.NET account. Even though we haven't yet decided on a key management strategy (that will come later), we know that we don't want the ASP.NET account to have access to the key. This would create the possibility that an attacker could exploit a vulnerability and obtain confidential information through the Web site. While the odds are against this, the data that we are protecting is such that the damage potential is extremely high. This kind of intrusion could affect all users, so we always try to avoid this option.

The second option, the Windows service, looks promising. Since a Windows service runs under its own identity, we don't have the concern that we had with the in-process DLL. But the most realistic way to communicate with the server would be .NET Remoting. While Remoting is an excellent technology for inter-appdomain communications, in the Microsoft® .NET Framework 1.x secure access control is not built into the Remoting API, nor is it accessible from COM clients. While it is true that you can build your own Remoting channel sink components to add a layer of security, this is somewhat outside our current scope. (See Stephen Toub's MSDN®Magazine article "Secure Your .NET Remoting Traffic by Writing an Asymmetric Encryption Channel Sink". Also, see ".NET Remoting Authentication and Authorization Sample—Part II".)

We could use Code Access Security (CAS) with Remoting, but the lack of support for COM clients makes it a poor choice. We will, however, be examining CAS later in this article.

Using a serviced component in Enterprise Services has the same benefit as using a Windows service—it runs under its own identity with its own credentials. Moreover, Enterprise Services is built on COM+ Component Services, which provides strong access control and easy access to COM clients. Therefore, we made CryptoUtility a serviced component.

Cryptographic Decisions

Nineteenth century Flemish cryptographer Auguste Kerckhoff proposed a simple but powerful rule: the security of a cryptographic system should depend entirely on keeping the key a secret. The lesson to developers is: do not develop your own encryption algorithm (in fact, avoid implementing a well-known algorithm yourself except as an educational exercise). Unless you are a mathematical genius focusing specifically on cryptographic theory, you'll almost certainly make mistakes trying to do it yourself. Commercial implementations of established algorithms have been vetted through extensive certification, such as the Federal Information Processing Standards issued by the National Institute of Standards and Technology (NIST).

There are two types of cryptographic algorithms available: symmetric and asymmetric encryption. Symmetric (or private-key) encryption uses a single key for both encryption and decryption. A compromised key can lead to complete failure of the security of a system. With symmetric algorithms, the security of the key is inversely proportional to the number of people who have the key.

With asymmetric (or public-key) encryption, there are two mathematically related but independent keys, a public key and a private key. Information encrypted with the private key can only be decrypted with the corresponding public key, and information encrypted with the public key can only be decrypted with the corresponding private key.

Since we won't be distributing the key to anyone, we'll use symmetric encryption for our utility. It has the advantage of being significantly faster than an asymmetric algorithm, but we will need to take some precautions to protect the key.

How Symmetric Algorithms Work

Symmetric algorithms of the type discussed in this article are block ciphers. They break cleartext up into blocks of a fixed size (in the case of the Rijndael algorithm, 16, 24, or 32 bytes) and perform iterative rearrangement and substitution on successive blocks. In the .NET Framework, you have a choice of several symmetric algorithms: Data Encryption Standard (DES, the previous Federal standard), 3DES (or Triple-DES), RC2, and Rijndael. The Data Encryption Standard has fallen out of favor as hardware has advanced because, with only a 56-bit key, it is susceptible to brute-force attacks. Triple-DES, which is DES three times over, is still relatively secure and possibly a good choice for encryption as it is widely supported on most platforms.

We have chosen Rijndael for the CryptoUtility because it offers the greatest key length of the algorithms available natively from .NET—256 bits. It is also the new Advanced Encryption Standard accepted by the Federal Government in 2001, and only after a long and thorough review of several algorithms in a contest sponsored by NIST as a replacement for DES.

Options for Symmetric Algorithms

Symmetric algorithms offer a number of options to control their operation. In most cases, you don't have to set these specifically as the chosen defaults are the most secure. However, it helps to understand what they're doing. The options for symmetric algorithms include the following:

Mode sets the cipher mode. For Rijndael this is either Cipher Block Chaining (CBC) or Electronic Code Book (CB):

  • CBC, the .NET default, is the most secure cipher mode. CBC performs an XOR operation on each block of cleartext with the previous cipher block before enciphering it. It also requires an Initialization Vector (IV), a random block of the same length as the algorithm's block size. The IV is used as a stand-in to perform Cipher Block Chaining on the first block of cleartext, since at that point there is no previous block. The IV ensures that repetition in the first block of cleartext does not result in similar repetition of the first block of ciphertext when the same key is used.
  • ECB, the less-secure option, has each block enciphered independently. Repetition in the cleartext may produce patterns in the ciphertext, thereby weakening security.

Padding sets the padding mode. Since a given message's length may not be an exact multiple of the algorithm's block size, you have to pad the end of the message to fill the last block. Padding values are as follows:

  • PKCS7 pads the last block with integers, each of which is the number of bytes used to pad the message. For example, if the message required 2 bytes of padding, the padding would be "0x02, 0x02".
  • None. No padding is added. The message length must be an exact multiple of the block size.
  • Zeros. The last block is padded with zeros.

Key size sets the length of the key used for encryption and decryption. Longer keys are generally more secure because brute-force attacks on them are less likely to be successful. Rijndael allows for keys of sizes 128, 192, and 256 bits. The key size is independent from the block size.

Block size sets the length of each individual block. With Rijndael, the block sizes can be 128, 192, and 256 bits.

Note that Rijndael can be used with all nine permutations of block and key sizes; however, the IV size must be the same as the chosen block size. There is an advantage to using larger block sizes because Rijndael may use from 10 to 14 rounds internally during encryption, with a larger number of rounds for larger block and key sizes. Thus, the more rounds that Rijndael uses internally, the more resistant the cipher is to cryptanalysis. For more information on Rijndael, see James McCaffrey's article on the subject from the November 2003 issue of MSDN Magazine.

I chose to use cipher block chaining and 256-bit keys and block sizes with Rijndael to ensure the greatest level of security for the ciphertext. The CBC and initialization vector are critically important; many credit card numbers start with similar leading digits. If the initial ciphertext was the same for these, it would make it a lot easier for someone to gain access to encrypted credit cards should he manage to steal the database!

Creating Strong Keys

Predictable cryptographic keys, like predictable passwords, badly compromise the security of your application. Don't generate these keys by hand—humans are poor random number generators. Using the System.Random pseudo-random number generator is also not sufficient since its random sequences are deterministic and repeatable (hence the name "pseudo-random").

You can use RNGCryptoServiceProvider to generate strong keys. It generates cryptographically strong random numbers and is suitable for key and salt/IV generation. While RNGCyptoServiceProvider is technically a pseudo-random generator, it is NIST-certified as cryptographically strong. The following code is an example in Visual Basic® .NET that uses the RNGCryptoServiceProvider to get 32 random bytes of data:

Dim rng As RNGCryptoServiceProvider = New RNGCryptoServiceProvider()
Dim key as Byte() = New Byte(31){}
rng.GetBytes(key)
Another approach is to generate a cryptographic key from a hashed password. Starting with a plain-text, easily memorized passphrase, you can use the PasswordDeriveBytes class to generate a cryptographic key. The chief advantage of this method is that the passphrase is easy to remember. However, it is more involved, requires more code, and compounds the key storage problem because you have to also store the salt. This method is mainly useful when a user will input a passphrase, which will immediately perform encryption or decryption and then be discarded. This method won't work for the CryptoUtility since it's a server-side application. It could, however, be used to protect a key stored outside the application, as shown in the following lines of code:
Dim entropy As Byte() = ...
Dim pwd As PasswordDeriveBytes = New PasswordDeriveBytes(txtKey.Text, _
    entropy)
Dim key as Byte() = pwd.GetBytes(32)

Creating Strong Salt

Salt provides additional entropy (degree of disorder) to the cryptographic algorithm. We'll use the same method for generating the IV as we did for the key, the RNGCryptoServiceProvider random number generator, as shown here:

Dim rng As RNGCryptoServiceProvider = New RNGCryptoServiceProvider()
Dim entropy As Byte() = New Byte(15)
rng.GetBytes(entropy)

The encryption algorithm uses this IV as the first block used to perturb subsequent blocks to obfuscate any regularity in the first block. In the CBC mode, each block is fed forward into the next block's encipherment; for the first block, the IV stands in.

With Rijndael, the strength of the cipher is not dependent on the secrecy of the IV. But the fact that the same cleartext encrypted with the same key and IV produce the same ciphertext could be used to guess a message's contents. For example, if your cleartext was limited to the words yes and no, it would be trivial to decode these messages if the same key and IV were used each time, as the ciphertext would repeat. Adding a random IV each time produces different ciphertext, thus foiling this guessing game.

The IV need not be private; in our examples, we prepend the IV to the ciphertext before storing it. On decrypting, we must use the same IV as used during decryption. We can easily grab this from the beginning of the message below. Some have advocated storing the IV separately from the ciphertext. However, CBC mode provides sufficient protection without having to keep the IV secret.

' now append the IV to the beginning of the ciphered text
output = New Byte(IV_SIZE + data.Length - 1)
Buffer.BlockCopy(newIV, 0, output, 0, newIV.Length)
Buffer.BlockCopy(data, 0, output, IV_SIZE, data.Length)

' strip salt (IV) off first 16 bytes of input
Buffer.BlockCopy(cipherBlob, 0, iv, 0, IV_SIZE)
' put actual ciphertext back into cipherText array
Buffer.BlockCopy(cipherBlob, IV_SIZE, cipherText, 0, _
  (cipherBlob.Length - IV_SIZE))

"Hygienic" Cryptography

As strong as the algorithm might be, there is always a window of opportunity where we will know the cleartext— during the entire chain of code leading up to the encryption. During this window, the cleartext is sitting unprotected in memory, waiting to be encrypted. Even after code has been encrypted, the original cleartext may still hang around in memory for an indeterminate time.

If the cleartext was ever a System.String object, it is immutable and will remain in memory until it is overwritten by a garbage collection cycle. TextBoxes, Labels, and a host of other controls store their .Text property as a System.String. This changes somewhat in the .NET Framework 2.0, which introduces the SecureString class specifically for storing strings securely. (See the Visual Studio® 2005 Beta 1 documentation for information about SecureString.) However, that does not solve the problem of cleartext embedded in memory from multiple other sources such as TextBoxes and the like. If the cleartext in any form (byte array, string, char array, and so on) was written to the swap file by the operating system, it is on the hard drive. Furthermore, if the system crashes and produces a memory dump while your cleartext is in memory, the cleartext will be written to disk.

The attack mode for these vulnerabilities ranges from attaching a debugger to your process and reading the cleartext or key (high privilege, complex, and low exploitability) to bribing the driver of the truck that moves backup tapes to the off-site storage facility and reading your dump files (low-tech social hack, but still not highly exploitable). The best approach to these paranoid scenarios is to know the value of the data and then elevate the difficulty of compromising it beyond that value.

When analyzing such threats to your data, a detailed threat tree can show if these represent the culminations of several sub-threats that, on their own, are not highly exploitable, or repeatable. For example, to attach a debugger to a process, you must have debugging privileges on the Web server and must search the memory on the machine to find the pertinent information. While certainly possible, this is so unlikely that were it to actually transpire, it would mean that you have much bigger problems.

Still, you can follow some hygienic code practices to minimize the risks of these attacks. First, minimize the time the key itself is in cleartext in memory by storing it encrypted with the Data Protection API (DPAPI) in a registry key protected by an access control list (ACL). Decrypt the key immediately before use, then clear the byte array with Array.Clear, which zeroes out the bytes. Never store the key in a String object, even Base64-encoded; remember that Base64 is not encryption. Note also that clearing the byte array in this fashion might not completely solve the problem as the garbage collector could move the byte array in memory before it is cleared. A workaround for this scenario involves pinning the byte array at a specific memory location, thus preventing the collector from moving it. For more on ACLs see "Safety in Windows: Manage Access to Windows Objects with ACLs and the .NET Framework" by Mark Pustilnik in this issue.

Second, minimize the number of places and length of time that your cleartext is in memory. It is impossible to avoid this entirely; after all, the code needs to see the cleartext to encrypt it. But you can minimize the number of String objects containing your cleartext by converting it to a byte array and then clearing it when you have finished the encryption. Unfortunately, keeping cleartext out of String objects is very difficult. As mentioned, most controls store their Text property as a System.String, so this is nearly impossible.

Storing the Key

The most important part of cryptography in a real-world application is the storage of the key. Bruce Schneier, a security technologist and the author of Applied Cryptography (John Wiley & Sons, 1996), neatly summarized the pitfalls of relying wholly on strong cryptography for security by comparing it to a picket fence with just one picket a mile high. Since the key is the "secret" in symmetric cryptography, how do you protect the key? From an attacker's perspective, compromising the key is a better avenue than brute-forcing the ciphertext. Why climb the mile-high picket fence when you can just walk around it?

For this project, we're working with a server-based application, probably exposed as a Web site or Web service, which must encrypt and decrypt data on behalf of client applications. This presents different constraints than if you were designing the cryptography for a desktop application. In a desktop application, you should always assume that someone can debug your process; after all, a serious hacker is always the admin on his own box. The Web application may be run in a Web farm across multiple front-end servers, and multiple machines will need to share this secret information. We also want to design cryptography modules that can be used apart from the rest of the application, thus encouraging their reuse.

We could store the key in the database, but then we'd have the circular problem of retrieving the key if we wanted to use an encrypted database connection string. We could store the key in a local file, which is a workable option since we can control access with ACLs. We could store the key in a COM+ constructor string, but this limits us to one key and requires careful, trusted administration. After considering all these options, we chose to store the key in the registry, keyed to the application name. This allows us to set ACLs to control access, store a key for each application using CryptoUtility, and easily set the key from the administration utility. Registry keys also provide auditing capabilities so that we can see exactly who has read and written to the registry keys in the security audit log.

In CryptoUtility there is an option for caching the decrypted key in memory which allows for higher performance at the expense of slightly less secure key storage. If your app does not require the utmost security, you might choose to cache the key as well. Accessing the registry and using DPAPI to decrypt the key on each call is expensive, but not enough to matter unless you anticipate hundreds of hits per second.

Protecting the Key with ACLs

We can protect the key in two ways. First, ACLs limit access to the key in the registry. Setting ACLs is not difficult to do manually, but every manual step is a possible compromise waiting to happen, especially when deploying an application to a Web farm that may consist of dozens of servers. Therefore, we strongly advocate programmatically setting ACLs on the registry where the key is stored.

By developing a code-based utility to set registry key ACLs, we can incorporate key setup into either the installer itself or into a simple command-line utility that automates the setup and securing process. Unfortunately, the .NET Framework 1.x does not expose a way of setting ACLs on registry keys natively. But thanks to the hard work of Renaud Paquet, there is an extremely useful managed library that can be used to set ACLs on just about anything. You can find Renaud's ACLs in .NET library on GotDotNet.

Using his Win32® Security library, we were able to restrict access to the key in the registry (see Figure 1). Now, retrieving the key requires the attacker to either assume the ACLed identity or become administrator on the machine.

Figure 1 Securing a Registry Key (Close Figure 1)
' Create the dacl.
Dim regDacl As Microsoft.Win32.Security.Dacl = _
    New Microsoft.Win32.Security.Dacl()

regDacl.AddAce(New AceAccessAllowed(Sids.Admins, _
    AccessType.SPECIFIC_RIGHTS_ALL Or AccessType.STANDARD_RIGHTS_ALL, _
    AceFlags.CONTAINER_INHERIT_ACE))

regDacl.AddAce(New AceAccessAllowed(New Sid(_userName), _
    AccessType.GENERIC_READ, AceFlags.CONTAINER_INHERIT_ACE))

' Open the registry key
Dim hKey As IntPtr
Dim rc As Integer = Win32.RegOpenKey( _
    New IntPtr((int)Microsoft.Win32.RegistryHive.LocalMachine), _
    regKey, hKey)

' Set the DACL for the registry key
If rc = Win32.SUCCESS Then
    Dim sd As SecurityDescriptor = _
        SecurityDescriptor.GetRegistryKeySecurity(hKey, _
            SECURITY_INFORMATION.DACL_SECURITY_INFORMATION Or _
            SECURITY_INFORMATION.GROUP_SECURITY_INFORMATION Or _
            SECURITY_INFORMATION.OWNER_SECURITY_INFORMATION)
    sd.SetDacl(regDacl, false)
    sd.SetRegistryKeySecurity (hKey, _
        SECURITY_INFORMATION.DACL_SECURITY_INFORMATION)
    Win32.RegCloseKey(hKey)
End If

Protecting the Key with DPAPI

While we have limited access to the key with an ACL, we still have a cleartext encryption key stored in the registry and hence somewhere on the file system. This brings us to the task of encrypting the encryption key itself using DPAPI. Essentially, DPAPI provides "keyless" symmetric encryption by using parts of the currently loaded user or machine profile as a key. (For the full source of a DPAPI wrapper class, see the DpapiCryptographer.vb file in the sample code.)

In USER mode, DPAPI uses a loaded user profile to generate the key. With anything encrypted in DPAPI's user mode, only that particular user account can decrypt the encrypted data. This requires a profile to be loaded for a particular account, something that is not the default with a component in COM+. Additionally, to allow for encryption and decryption across multiple machines, roaming profiles must be enabled and used by all clients, which can be a management challenge.

Figure 2 CryptoUtility in Context
Figure 2 CryptoUtility in Context

MACHINE mode DPAPI means that any code running on the machine has access to DPAPI's key and therefore can decrypt any secret encrypted in MACHINE mode. We want to limit access to the key, and DPAPI allows us to pass a unique salt to inject additional entropy into the key generation process. We will store this salt in the registry with the encrypted key, limited by ACL to the same user identity as the key. This is not a perfect solution; you may want to store the DPAPI salt elsewhere. The attack for this particular solution involves mounting code on the machine, gaining access to the ACLed registry key, and being able to run DPAPI with the stolen salt to decrypt the key. Considering the low exploitability of this attack, this seemed to be an acceptable risk. Figure 2 shows CryptoUtility in context with COM+, the client application, and the surrounding subsystems DPAPI and Registry.

Mutual Authentication and Key Obfuscation

You will notice that certain overloads of the Crypto class accept a partial key and others do not. In another application, we wanted to use mutual authentication, and we provide these overloads to satisfy both requirements.

On the one hand, storing the entire key encrypted and ACLed in the registry is a good thing: you know where it is, how it is protected, and what it will take to steal it. On the other hand, it is easily discoverable, and once an attacker has leaped the hurdles necessary to steal the key, he has complete access to the encrypted data.

For one of the applications using the CryptoUtility, we'll add another layer to the key storage: the client application itself. That application uses one of the overloads that accepts a partial key. The application stores half the key, and CryptoUtility stores the rest in the registry. The application obfuscates its half of the key by hiding it in multiple places: innocuous-sounding method names and enumerations, plainly named resource file strings, simple mathematical operations, and the like.

We use Mutual Authentication because the application is passing half of the key; it is not completely anonymous and we know, to some degree, who is calling us. To decrypt information, the application has to know its half of the key.

Obfuscation's "security through obscurity" has been rightly reviled as a poor practice; it violates Kerckhoff's Law. Bluntly put, obscurity is not security. However, it takes time to attack a system and discover everything about it. If we assume an attacker works over the network, then he must spend some time downloading our assembly and decompiling it to understand how we derived our half of the key. In that time, we hope that the intrusion detection systems will alert us to unusual activity. Security by obscurity, if it is the only defense, is woefully inefficient. It is, however, a smart part of an overall defense-in-depth strategy when it supplements stronger practices.

The worst-case scenario for the partial-key approach is an inside attacker who already knows the system (one of its creators, for instance) and who has network access to the machine. If this worst-case attacker is also an administrator on the machine, all bets are off—he will now own your system.

However, if your organization is following good practices and does not allow developers administrative access to production servers, the hurdle is again quite high. The knowledgeable inside attacker must gain access to the encrypted, ACLed registry key to obtain the half of the cryptographic key he doesn't know, raising the bar to the same level as the full key-in-the-registry scenario. In addition, with auditing in place, you may well be able to track back to the internal miscreant and catch him in the act. Remember that even the most secure systems are vulnerable to inside attacks. Technology must be supplemented by the right people and processes to ensure true security.

Another objection to the partial-key scenario is that, if the attacker elucidates the code-obfuscated partial key, the remaining key is only half-strength, making a brute-force attack exponentially easier. But since we are using the full 256-bit Rijndael, the 128-bit half-key still provides very high security. If storing half the key in the client application does not make sense for your scenario, then it's perfectly reasonable to store the whole key in the registry.

Protecting the Crypto Application Itself

We've discussed two layers of protection: protecting the private data itself with strong cryptography via Rijndael, and protecting the key to that data using ACLs and DPAPI. But there is a third layer that needs protection: the CryptoUtility itself!

Storing the entire key in the registry also means that any assembly with the right to call the Crypto assembly can use its encrypt/decrypt methods without supplying additional credentials, so the previous two layers are useless if an anonymous caller can get CryptoUtility to decrypt a key for them.

To help prevent this, we've used a feature of CAS called the StrongNameIdentityPermission (SNIP) LinkDemand. This declarative CAS feature allows us to decorate a method or class with an attribute containing the public key of the assemblies allowed to call our assembly. For example:

<StrongNameIdentityPermission(SecurityAction.LinkDemand, PublicKey:= _
  "002400000480000094000000060200000024000052534131000400000100010" & _
  "085525e9438e9fae122f71ec7124443bf2f9f57f5f3760b3704df168493004b" & _
  "9ef68413f500d54fa9fa3869b42b1e2365204826e54b618d56e7e575f27f675" & _
  "f0eae3ea8458a8ee1e92dc3f4bfc34fbe23851afa9d2c28fc8cd5b124f60a03" & _
  "a06bfb598bc3acbd8c4380aef02cc58bdf955d140390f740a7e115c59e3b3b5" & _
  "758ca")>

At run time, the .NET Framework confirms that the assembly calling our assembly has been signed with the exact Strong Name demanded by the attribute. If it has not, the call will fail. This applies to attempts to reflect over the assembly as well.

There are some weaknesses in this approach, however. The main weakness of the SNIP LinkDemand is that a sufficiently privileged user can simply bypass all such security checking by, for example, using the Code Access Security Policy (CASPOL) tool to modify the security settings in the .NET Framework. Keith Brown discusses such issues in his April 2004 Security Briefs column. This approach also fails if your SNK file is compromised; this could result from, say, an insider attack by someone on or near the team with access to the Strong Name key files.

Last, but not least, this particular approach fails in a predictable but slightly surprising place: COM. Since COM has no concept of Strong Names, it also has no understanding of StrongNameIdentityPermission LinkDemand.

COM Security

COM does not obey .NET CAS rules. If we choose to mount CryptoUtility as a COM+ application, it is accessible to COM to the same degree as any other COM component on the machine. This means that every bit of our carefully planned security system can be compromised by a simple, four-line VBScript snippet:

set crypto = CreateObject( "CryptoUtility.Crypto" )
original = "this is the secret message."
cipher = crypto.Encrypt("ourApp", original)
clear = crypto.Decrypt("ourApp", cipher)

Figure 3 Require Authorization
Figure 3 Require Authorization

But all is not lost. COM provides strong, time-tested protections against illicit access. In order to limit access from COM callers, you should enable the COM+ authorization for the package, as shown in Figure 3 and Figure 4.

Keep in mind that these security checks apply to .NET callers as well, so this offers yet another level of security for .NET clients.

Putting It All Together

The sample code for this article includes CryptoUtility and its administrative application, CryptoAdminUtility. We covered some aspects of CryptoAdminUtility, but the full source code does much more, including adding Windows users, setting their group membership, creating and configuring COM+ applications, and placing DPAPI-encrypted keys in the registry.

Figure 4 Set Security Level on COM+
Figure 4 Set Security Level on COM+

CryptoUtility consists of six classes. The main class, Crypto, provides the only externally visible API through its interface ICrypto. Since CryptoUtility is a COM+ ServicedComponent, we followed best practices with its design, including explicitly implementing an interface. The Crypto class makes use of several internal helper classes which perform the actual work of encrypting. These include SymmetricCryptographer, DpapiCryptographer, Hasher (for hashing and comparing hashes), and a utility class for reading strings from a resource file.

The Crypto class manages key caching and allows client applications to specify whether to use caching. Internally, Crypto uses a contained class called CachedKey to hold cached keys. Crypto accesses these cached keys through an internal collection. Lastly, Crypto contains internal methods to fetch data from the registry, and in turn uses DpapiCryptographer to decrypt those keys.

Figure 5 illustrates a static diagram depicting the CryptoUtility classes. The encryption and decryption sequences through CryptoUtility are very similar. The sequence diagram in Figure 6 shows the encryption process, but the decryption process functions exactly the same way, just performing decryption where the diagram depicts encryption.

Figure 6 Encryption with CryptoUtility (Close Figure 6)

Figure 5 The CryptoUtility Classes (Close Figure 5)

Conclusion

While there is no such thing as perfect security, it is possible to use cryptography in a secure manner in a real-world application. The .NET Framework provides excellent support for building such secure applications out of the box. There are trade-offs in every architecture and design; we've described the ones we made so that you can make better choices for yourself. A large part of successful architecture and software design lies in understanding these trade-offs and then choosing wisely.

Michael Stuart is a senior consultant with Microsoft Consulting Services. He reminds you that Halo 2 is coming in November.

J Sawyer is a Developer Evangelist with the Microsoft Central Region Developer & Platform Evangelism team working in Houston. Previously, J was a consultant with MCS and has been developing solutions on the Microsoft platform for over 10 years.


Trustworthy Code: Exchange Data More Securely with XML Signatures and Encryption

Trustworthy Code

Exchange Data More Securely with XML Signatures and Encryption

Mike Downen and Shawn Farkas
Parts of this article are based on a prerelease version of the .NET Framework 2.0. All information pertaining to those sections is subject to change.
This article discusses:
  • XML Signature and XML Encryption standards
  • Digital signing and encryption features in the .NET Framework 1.x and 2.0
  • X.509 certificate integration
This article uses the following technologies:
XML, .NET Framework, C#, Security
Code download available at: XMLSignatures.exe (241KB)
Contents
Digital Signing
XML Signature Basics
References
Transforms
Canonicalization
Key Management
Signature Profiles
New Features in the .NET Framework 2.0
EncryptedData Elements
XML Encryption Example
X.509 Support
Conclusion

T
he XML Signature and XML Encryption standards are being used extensively as building-block technologies. Microsoft® Office InfoPath™ uses XML signatures to sign partial or whole forms. Web services use XML signatures to sign SOAP messages and XML encryption to encrypt them. The XML manifests for ClickOnce®-based applications, new in Visual Studio® 2005, also use XML signatures. The .NET Framework 1.x includes an object model for the XML Signature standard, and the .NET Framework 2.0 adds additional support, while adding an object model for XML encryption as well. This article explains the XML Signature and XML Encryption standards and shows you how to use them with .NET. For the actual XML Signature specification, see the W3C standard at XML-Signature Syntax and Processing.

Digital Signing

Before we dive into the XML Signature standard, let's review the basics of digital signing. Because it's important to prevent a malicious user from altering a message during transmission, digital signing protects the integrity of data and can detect any changes made to it while it's en route to its recipient. Because it's also important to be able to identify a sender, a message is typically signed using the private (secret) key of the sender and verified with the corresponding public key, allowing recipients to confirm the identity of the sender when they know the sender's public key. This prevents a malicious user from pretending to be a known sender, either by trying to send messages as the known sender or by intercepting messages from the known sender and replacing them with their own—a form of the man-in-the-middle attack.

To create a digital signature, first hash the message you are signing using a cryptographic hash function. For any length of input, a cryptographic hash function returns a fixed-length set of bits called the hash value. This hash value cannot be easily converted back to the input. The hash value changes unpredictably when even a single bit changes in the input, so someone cannot find an input that is similar to the original just by finding a similar hash value. One commonly used hash function is SHA-1, which produces a 160-bit hash value.

The next step is to sign the hash value using a signing algorithm and your private key to produce a signature value. You create the signature with your private key so others who have your public key can verify it (more on that later in the article). RSA is a popular cryptographic algorithm to use with signatures.

After you've sent the message and this signature to the recipient, the verification process begins. The received message is hashed with the same hash function used when signing. Then the signature value is verified by passing it, along with the public key and the computed hash, to the signing algorithm. If the computed hash and the signature hash match, then the signature is valid. If the hashes don't match, either the data or the signature has been changed and therefore the integrity of the data is not assured.

You can also sign and verify data using a keyed hash algorithm, but that's beyond the scope of this article. The .NET Framework already includes a rich set of classes for all kinds of hashing, encryption/decryption, and signing/verification algorithms. For more information on these, see the .NET Framework SDK documentation and .NET Framework Security, by Brian LaMacchia, et al (Addison-Wesley, 2002).

XML Signature Basics

You can sign any kind of data using XML Signature, including part of an XML document, other XML documents, or other data of any format. However, in practice, XML signatures are most frequently used to sign other data represented in XML. The XML Signature standard is also very flexible, allowing you to filter and transform your data before you sign it and letting you choose exactly what to sign and how.

Let's take a look at a simple XML Signature scenario: signing an entire document and including the signature in it (see Figure 1). Notice the Signature element that has been added to the document. This element contains the XML signature. Let's look at what each element contains.

Figure 1 Simple XML Document Before and After Signing (Close Figure 1)

A simple document:
<docRoot>
  <a>Hello</a>
  <b>World</b>
</docRoot>
The document when signed:
<docRoot>
  <a>Hello</a>
  <b>World</b>
  <Signature xmlns="http://www.w3c.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod
        Algorithm="http://www.w3c.org/TR/2001/REC-xml-c14n-20010315"/>
      <SignatureMethod
        Algorithm="http://www.w3c.org/2000/09/xmldsig#rsa-sha1"/>
      <Reference URI="">
        <Transforms>
          <Transform
          Algorithm=
          "http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
        </Transforms>
        <DigestMethod
          Algorithm="http://www.w3c.org/2000/09/xmldsig#sha1"/>
        <DigestValue>cbPT0951Ghb2G3UjpVjWw+7q0Bc=</DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue>IoEwS(3 lines of Base64 text)XSo=</SignatureValue>
  </Signature>
</docRoot>

SignedInfo The children of this element contain all the information about what is signed and how it was signed. The signature algorithm is actually applied to this element and all of its children to generate the signature.

CanonicalizationMethod This specifies the canonicalization (C14N) algorithm used on the SignedInfo element to normalize the XML. We'll talk more about C14N later in the article.

SignatureMethod This element specifies the signature algorithm for this signature. In this example, the signature algorithm is SHA-1 (for hashing) with RSA (for signing the resulting hash value).

Reference These elements specify the data that will be signed and how it should be processed before hashing. The URI attribute, representing the Uniform Resource Identifier, identifies the data to be signed, while the Transforms element (described later) specifies how to process the data before hashing. In this example, we're using a special URI, the empty string, which specifies that the document containing the signature is the data to be included in the signature.

The XML Signature standard uses an indirect signing mechanism for Reference data. Rather than hashing and then encrypting the hash value for all of the data from the Reference, the standard hashes the data for each Reference, using the algorithm specified by the Reference's DigestMethod element, and then stores the hash value in the DigestValue element of the Reference. Then the SignedInfo element and all of its children (including the Reference elements) are hashed; that hash value is encrypted to generate the signature. So you're really signing the hash of the hash of data referred to in Reference elements, but this scheme still protects the integrity of the data. Figure 2 and Figure 3 show the signing and verification process alongside the matching XML.

Transforms Each Reference element can have zero or more transforms specified for it. These transforms are applied to the data for the Reference in the order they are listed in the XML. Transforms allow you to filter or modify the data for the Reference before the data is hashed. In this example, we're using the enveloped signature transform, which selects all of the XML in the containing document except for the Signature element. We must remove the Signature element from the data to be signed; otherwise we would be modifying the data we're trying to sign when we store the signature value. We'll talk more about transforms later in the article.

SignatureValue This element contains the signature value computed by signing the SignedInfo element and all of its children.

Figure 2 Signing Process
Figure 2 Signing Process

Now let's discuss the processing model for creating a signature (see Figure 2). First, for each Reference element in the signature:

  • Apply each transform algorithm specified in the Transform elements to the data for the Reference, in the order the transforms appear under the Transforms element.
  • Hash the transformed data using the hashing algorithm specified by the DigestMethod element of the Reference.
  • Store the resulting hash value in the DigestValue element of the Reference.

The next step is to canonicalize the SignedInfo element and its children using the algorithm specified in the CanonicalizationMethod element of the signature. Then you sign the SignedInfo element and its children using the algorithm specified in the SignatureMethod element of the signature. The signature value is stored in the eponymous SignatureValue element.

Figure 3 Verification Process
Figure 3 Verification Process

Signature verification is the inverse of the process just described (see Figure 3). First, you must canonicalize the SignedInfo element and its children using the C14N algorithm specified in the CanonicalizationMethod element. Then you must verify the signature value stored in the SignatureValue element against the SignedInfo element and its children.

Finally, for each reference element in the signature:

  • Apply each transform algorithm specified in the Transform elements of the Reference to the data for the Reference, in the order the transforms appear under the Transforms element.
  • Hash the transformed data for the reference using the hashing algorithm that is specified by the DigestMethod element of the reference.
  • Compare the hash value computed with the value stored in the DigestValue element.

If the signature verification succeeds and the hash value for each reference is equal to the one stored in the signature, the XML signature is valid. Otherwise, either the data referred to by one of the reference elements has changed, or the signature element has been changed.

A signature embedded in the document it signs is called an enveloped signature. The code to create this kind of signature is shown in Figure 4. The code to verify the signature is shown in Figure 5.

Figure 5 Verifying an Enveloped Signature (Close Figure 5)
using System.Security.Cryptography;
using System.Security.Cryptography.Xml;
// Also, add a reference to System.Security.dll

// Load the signed data
XmlDocument doc = new XmlDocument();
doc.PreserveWhitespace = true;
doc.Load("data-signed.xml");

// Find the Signature element in the document
XmlNamespaceManager nsm = new XmlNamespaceManager(new NameTable());
nsm.AddNamespace("dsig", SignedXml.XmlDsigNamespaceUrl);
XmlElement sigElt = (XmlElement)doc.SelectSingleNode(
  "//dsig:Signature", nsm);

// Load the signature for verification
SignedXml sig = new SignedXml(doc);
sig.LoadXml(sigElt);

// Verify the signature, assume the public key part of the
// signing key is in the key variable
if (sig.CheckSignature(key))
    Console.WriteLine("Signature verified");
else
    Console.WriteLine("Signature not valid");
Figure 4 Creating an Enveloped Signature (Close Figure 4)
using System.Security.Cryptography;
using System.Security.Cryptography.Xml;
// Also, add a reference to System.Security.dll

// Assume the data to sign is in the data.xml file, load it, and
// set up the signature object.
XmlDocument doc = new XmlDocument();
doc.Load("data.xml");
SignedXml sig = new SignedXml(doc);

// Make a random RSA key, and set it on the signature for signing.
RSA key = new RSACryptoServiceProvider();
sig.SigningKey = key;

// Create a Reference to the containing document, add the enveloped
// transform, and then add the Reference to the signature
Reference refr = new Reference("");
refr.AddTransform(new XmlDsigEnvelopedSignatureTransform());
sig.AddReference(refr);

// Compute the signature, add it to the XML document, and save
sig.ComputeSignature();
doc.DocumentElement.AppendChild(sig.GetXml());
doc.Save("data-signed.xml");

You've seen how to create and verify enveloped signatures. While they are commonly used and handy for signing entire XML documents, the XML Signatures standard lets you sign other data as well by specifying different URIs in the Reference element. So let's take a look at different types of references next.

References

In addition to enveloped references (a Reference element that has an empty string for the URI attribute), there are two other broad types of references defined in the XML Signature standard: references to detached data and references to XML data by ID. Detached data is outside of the XML document containing the signature. These references can point to another XML document or to any other type of resource. You would typically use this type of reference when signing multiple resources in one signature, such as an XML document and some other files that are referred to by the document. The following XML code fragment shows an example of a detached reference:

<Reference URI="http://www.example.com/foo.jpg">
  <DigestMethod
    Algorithm="http://www.w3c.org/2000/09/xmldsig#sha1" />
  <DigestValue>cbPT0951Ghb2G3UjpVjWw+7q0Bc=</DigestValue>
</Reference>

In order to create a detached reference using the Framework classes, you simply set the URI of the reference to the URI of the resource when creating the reference object, and then add the reference to the signature, as shown here:

// Create a Reference to detached data, assume a SignedXml object in sig
Reference refr = new Reference("http://www.example.com/foo.jpg");
sig.AddReference(refr);

References to XML data by ID point to XML within the document containing the signature or within the signature itself. For these references, the signature engine looks for an element with an Id attribute that matches the URI in the reference, not including the #. Here's an example of an ID-based reference:

<Reference URI="#myData">
  <DigestMethod
    Algorithm="http://www.w3c.org/2000/09/xmldsig#sha1" />
  <DigestValue>cbPT0951Ghb2G3UjpVjWw+7q0Bc=</DigestValue>
</Reference>

If the element with the Id of "myData" is in the document containing the signature, this reference is complete and the signature engine will find it when processing the signature. You would typically use this type of reference to limit the scope of a signature to a specific part of an example document. For example, in a document processing application, a reviewer would only sign the portions of the XML document she reviewed, not the entire document.

The standard also allows you to add arbitrary data to the signature in an Object element. The XML for an Object element looks like the following line of code:

<Object Id="myData">Your XML goes here</Object>
To create an ID-based reference to an element that is already in the document containing the signature, add code like this:
// Create a Reference to XML data in the containing document,
// assume a SignedXml object in sig
Reference refr = new Reference("#myData");
sig.AddReference(refr);

If the ID-based reference is to an Object element in the signature, you must also add a DataObject to the signature in addition to adding the reference, as in the code that was just shown. The data object can contain any XML that you pass in.

// Adds a DataObject with an Id of "#myData" to the signature, assume a
// SignedXml object in sig, and xml data of type XmlNodeList in data
DataObject dobj = new DataObject();
dobj.Id = "myData";  // Note: no #
dobj.Data = data;    // XML Data of the Object
sig.AddObject(dobj);
You would typically use a reference to an Object element for signing metadata about the signature, such as a unique identifier for the signer or some other information about the signature.

Transforms

Transforms give you even more control over the content signed by allowing you to modify the data for a reference before the hash value for that data is generated. The enveloped signature transform, for example, removes the Signature node from the XML document before signing it. A reference can specify any number of transforms, which are applied in the order specified in the Transforms element. The .NET Framework classes support the following transforms in addition to the enveloped signature transform that we mentioned earlier:

  • Any canonicalization algorithm can be used as a transform.
  • The Base64 transform allows you to decode data in the Base64 encoding.
  • The XSLT transform allows you to apply an XSLT stylesheet to XML data before signing it. The XSLT stylesheet to be applied is specified as XML under the Transform element.
  • The XPath transform allows you to filter XML data with an XPath expression.

The XPath expression is specified as the text content of an XPath element under the Transform element. It's important to note that the XPath transform acts as a filter, not as a means of selecting nodes in the XML being passed as input. The transform evaluates the XPath expression against each node passed as input to the transform, and the result is converted to a Boolean. An input node is considered to pass the evaluation and will be included in the output of the transform if the result of the evaluation is true. Consider the following XML input to the transform:

<a>
  <b>Some data</b>
  <c>More data</c>
</a>
<d>
  <b>Even more data</b>
</d>
Let's say we only want to select "b" nodes for signing. An XPath transform with the XPath expression of "ancestor-or-self::b" will return the following nodeset, which is what we want:
<b>Some data</b>
<b>Even more data</b>
The Transform element for this XPath expression would look like the following code snippet:
<Transform
  Algorithm="http://www.w3c.org/TR/1999/REC-xpath-19991116">
  <XPath>ancestor-or-self::b</XPath>
</Transform>

To create transforms programmatically when signing, create an instance of a transform object, set its properties appropriately, and add it to the reference to which it applies. The following example adds the XPath transform used in the previous example to a reference to create a transform:

// Add an XPath transform to a reference.
// Assume a Reference object in refr
XmlDocument doc = new XmlDocument();
doc.LoadXml("<XPath>ancestor-or-self::b</XPath>");
XmlDsigXPathTransform xptrans = new XmlDsigXPathTransform();
xptrans.LoadInnerXml(doc.ChildNodes);
refr.AddTransform(xptrans);

Canonicalization

The purpose of canonicalization is to produce the same XML data for two XML fragments that are logically the same, but may not be represented by the same text. For example, take a look at the following two code fragments. They are logically the same; they differ only in text representation. But if you were to hash both of them as is, the hash values would be different:

<root>
  <a   >Some text</a>
  <b attr1="yes" attr2="no"></b>
  <c    Id="foo">More text</c>
</root>

<root>
  <a>Some text</a>
  <b attr2="no" attr1="yes" />
  <c Id="foo">More text</c>
</root>
To avoid this problem, the default canonicalization algorithm specified in the standard performs a number of tasks including the removal of white space in start and end tags and the conversion of empty elements to start/end tag pairs. It does not, however, change any white space in the content of elements. The complete list of operations performed is available in the Canonical XML recommendation, located at Canonical XML.

The signature engine automatically canonicalizes data when necessary to comply with the W3C standard. Specifically, the signature engine canonicalizes XML data anytime it needs to convert it to binary data for hashing. This happens when it prepares to sign the SignedInfo element and its children, for instance. It also may happen when preparing a reference or the output of a transform for signing. For example, if you use an ID-based reference to other XML data in the document containing the signature, and the reference has no transforms associated with it, the signature engine will canonicalize the XML data for that reference before hashing the data.

Key Management

The XML Signature standard provides the KeyInfo element to help with key management. This element can store a key name, key value, key retrieval method, or certificate information to help receivers verify the signature. The standard does not specify how or if any information in the KeyInfo element should be trusted.

The KeyInfo element can be useful when the sender and recipient share a list of trusted keys or if you come up with some other method for mapping key names to keys, for example. The .NET Framework 1.x has some support for key name, value, and retrieval method. The .NET Framework 2.0 also includes support for X.509 certificates.

Let's say the sender and recipient share a list of keys. The recipient has a public key for each sender from whom she expects to receive messages. The signing application can add the following code to add a KeyInfo element to the signature:

// Adds an KeyInfo element with RSA public key information to the
// signature.
// Assumes a SignedXml object in sig, and an RSA object in key.
KeyInfo ki = new KeyInfo();
ki.AddClause(new RSAKeyValue(key));
sig.KeyInfo = ki;
This code should be added before ComputeSignature is called. It will produce a KeyInfo element in the signature that looks something like the following code:
<KeyInfo><KeyValue><RSAKeyValue>
  <Modulus>4LfG(2 lines of Base64 text)2Fr=</Modulus>
  <Exponent>AQAB</Exponent>
</RSAKeyValue></KeyValue></KeyInfo>
This represents the RSA public key used to sign the XML document. The receiving application should compare this key to a list of trusted keys and should not trust the document if the public key is not on the list. Otherwise an attacker could replace a signed document while in transit, signing it with another key. If a signature includes an RSAKeyValue like this, the verification code can call the CheckSignature method of the SignedXml class that takes no parameters, and the .NET Framework will figure out the key from the RSAKeyValue element. Here's an example:
// Verify a signature that includes RSAKeyInfo or DSAKeyInfo.
// Assume a SignedXml object in sig.
bool verified = sig.CheckSignature();

Signature Profiles

Along with the flexibility of XML signatures comes some amount of risk. In particular, because transforms are so flexible, it can be difficult to figure out exactly what data was covered by the signature, which can lead to unexpected or insecure results. These signature profiles can help in this regard by specifying the form of signature your application supports. While there is no standard for signature profiles a signature profile should at the least specify references and transforms that the application expects for a signature, so you can be sure that the data you expect to be signed is indeed signed. Signature profiles could also contain additional data, such as signing algorithms or key sizes expected for signed data. Your application should check and enforce that those signatures it creates and verifies comply with the signature profile that your application supports.

To better understand why profiles are important, consider Figure 1. Suppose you are writing an application that accepts XML signed data, but your application only expects signatures that use the enveloped signature transform and no other transforms. Now someone sends you a signed document with an extra XPath transform, as shown in Figure 6.

Figure 6 Signed Document with an Extra Transform (Close Figure 6)
<docRoot>
  <a>Hello</a>
  <b>World</b>
  <Signature xmlns="http://www.w3c.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod
        Algorithm="http://www.w3c.org/TR/2001/REC-xml-c14n-20010315"/>
      <SignatureMethod
        Algorithm="http://www.w3c.org/2000/09/xmldsig#rsa-sha1"/>
      <Reference URI="">
        <Transforms>
          <Transform
            Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-
              signature"/>
          <Transform
            Algorithm="http://www.w3c.org/TR/1999/REC-xpath-19991116">
              <XPath>ancestor-or-self::a</XPath>
         </Transform>
       </Transforms>
       <DigestMethod
         Algorithm="http://www.w3c.org/2000/09/xmldsig#sha1"/>
       <DigestValue>mN4R0653F4ethOiTBeAu+7q0Be=</DigestValue>
     </Reference>
    </SignedInfo>
    <SignatureValue>X4Ie(3 lines of Base64 text)nP3=</SignatureValue>
  </Signature>
</docRoot>

In the example in Figure 6, the signature only covers the "a" element in the sample document. If you just loaded the document and called the CheckSignature method of the SignedXml class, the signature may still verify even though the "b" element is not covered by the signature, because the signature engine will apply the transform specified in the signature. If your application relies on the "b" element being covered in the signature, the integrity of your data has been compromised. The application should verify the signature profile it expects by checking that there is just one reference that has an empty string as the URI and that the reference has one transform: the enveloped signature. It would reject any other signature profile when verifying a signature. Some sample code that checks this signature profile is shown in Figure 7.

Figure 7 Checking a Signature Profile (Close Figure 7)
// This method checks the signature profile for the signature
// in the supplied document. It ensures there is only one
// Signature element and only one enveloped reference with only
// one enveloped signature transform
public bool CheckSignatureProfile(XmlDocument doc)
{
    // Make sure there is only one Signature element
    XmlNamespaceManager nsm = new XmlNamespaceManager(new NameTable());
    nsm.AddNamespace("dsig", SignedXml.XmlDsigNamespaceUrl);
    XmlNodeList sigList = doc.SelectNodes("//dsig:Signature", nsm);
    if (sigList.Count > 1)
        return false; //Wrong number of Signature elements

    //Make sure the Signature element has only one Reference
    XmlElement sigElt = (XmlElement)sigList[0];
    XmlNodeList refList = sigElt.SelectNodes(
        "dsig:SignedInfo/dsig:Reference", nsm);
    if (refList.Count > 1)
        return false; //Wrong number of Reference elements

    // Make sure the Reference URI is ""
    XmlElement refElt = (XmlElement)refList[0];
    XmlAttributeCollection refAttrs = refElt.Attributes;
    XmlNode uriAttr = refAttrs.GetNamedItem("URI");
    if ((uriAttr == null) || (uriAttr.Value != ""))
        return false; // Wrong type of reference

    // Make sure the only tranform is the enveloped signature transform
    XmlNodeList transList = refElt.SelectNodes(
        "dsig:Transforms/dsig:Transform", nsm);
    if (transList.Count != 1)
        return false; //Wrong number of Transform elements
    XmlElement transElt = (XmlElement)transList[0];
        string transAlg = transElt.GetAttribute("Algorithm");
    if (transAlg != SignedXml.XmlDsigEnvelopedSignatureTransformUrl)
        return false; //Wrong type of transform

    return true;
}

So far, we've looked at some different aspects of the XML Signature standard and the support for it in the .NET Framework. Let's put some of these features together into a more complete example. Suppose you're writing an application to exchange messages in XML and you want to sign the entire content of the message. You also want to add some XML data about the signer to the signature element as an object, only signing the signerID element of this data. Your application has access to a well-known list of keys, so you will also store the public key information in the signature and check to make sure the key maps to a well-known key during verification. The code for signing and verifying a message is included in the complete code download for this article. Signing a message with this code will produce an XML signature that looks something like Figure 8.

Figure 8 A More Complex Signed Document (Close Figure 8)
<root>
  <myData1>Some Data</myData1>
  <myData2>More data</myData2>
  <Signature xmlns="http://www.w3c.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod
        Algorithm="http://www.w3c.org/TR/2001/REC-xml-c14n-20010315"/>
      <SignatureMethod
        Algorithm="http://www.w3c.org/2000/09/xmldsig#rsa-sha1"/>
      <Reference URI="">
        <Transforms>
          <Transform
           Algorithm=
           "http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
        </Transforms>
        <DigestMethod
          Algorithm="http://www.w3c.org/2000/09/xmldsig#sha1"/>
        <DigestValue>cbPT0951Ghb2G3UjpVjWw+7q0Bc=</DigestValue>
      </Reference>
      <Reference URI="#signer">
        <Transforms>
          <Transform
          Algorithm=
          "http://www.w3c.org/TR/1999/REC-xpath-19991116">
            <XPath xmlns:my="http://example">
              ancestor-or-self::my:SignerID
            </XPath>
          </Transform>
        </Transforms>
        <DigestMethod
          Algorithm="http://www.w3c.org/2000/09/xmldsig#sha1"/>
        <DigestValue>mN4R0653F4ethOiTBeAu+7q0Be</DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue>IoEwS...</SignatureValue>
    <KeyInfo>
      <KeyValue>
        <RSAKeyValue>
          <Modulus>4LfG(2 lines of Base64 text)2Fr=</Modulus>
          <Exponent>AQAB</Exponent>
        </RSAKeyValue>
      </KeyValue>
    </KeyInfo>
    <Object Id="signer">
      <my:SignerData xmlns:my="http://example">
        <my:SignerName>Mike</my:SignerName>
        <my:SignerID>4815</my:SignerID>
      </my:SignerData>
    </Object>
  </Signature>
</root>

New Features in the .NET Framework 2.0

All the XML Signature features discussed so far are available in both the .NET Framework 1.x and 2.0. XML Encryption and X.509 certificate integration are new to version 2.0. X.509 certificate integration makes it easier to use X.509 certificates with XML Signature. The new X509CertificateEx class and related classes make it easier to manipulate and use certificates, and the XML signing object model uses these classes when appropriate. We'll talk more about X.509 integration at the end of this article.

XML Encryption is another W3C standard. Just as XML Signature specifies a format and processing model for creating digital signatures in XML, XML Encryption standardizes how to encrypt data in XML. XML digital signatures are driven through the SignedXml class, and XML encryption is performed using the new EncryptedXml class.

Although XML encryption can be used to encrypt arbitrary data, you would most frequently use it to encrypt other XML. When used in this way, you have a lot of flexibility in how documents are encrypted. For instance, different nodes of an XML document could be encrypted with different keys, while some nodes are left in plain text. And since encrypting something with the EncryptedXml class results in XML, you can even encrypt the encrypted results, a process known as super encryption.

Let's take a look at some encrypted XML (see Figure 9). One interesting thing to note right away is that the XML Encryption standard uses the XML Signature namespace for some elements, including the KeyInfo element.

Figure 9 XML Document Before and After Encryption (Close Figure 9)

A simple document:
<docRoot>
  <a>Hello</a>
  <b>World</b>
</docRoot>
The document when encrypted:
<docRoot>
  <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
  xmlns="http://www.w3.org/2001/04/xmlenc#">
    <EncryptionMethod
    Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc"/>
    <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
      <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
        <EncryptionMethod
        Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5"/>
        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
          <KeyName>recipient_public_key</KeyName>
        </KeyInfo>
        <CipherData>
          <CipherValue>PrI6(3 lines of Base64 text)Dwy4=</CipherValue>
        </CipherData>
      </EncryptedKey>
    </KeyInfo>
    <CipherData>
      <CipherValue>awcH(3 lines of Base64 text)NNqQ=</CipherValue>
    </CipherData>
  </EncryptedData>
</docRoot>

EncryptedData Elements

EncryptedData is the root element generated by encrypting the XML, and it contains all the information about the data. EncryptedData contains three main subelements: EncryptionMethod specifies which algorithm was used to encrypt the data, KeyInfo element provides information about which key to use to decrypt the data, and the CipherData or CipherReference element contains the actual encrypted information. The EncryptionMethod element specifies the algorithm used to encrypt and decrypt the cipher data. The algorithms are specified by a URI, like the XML Signature standard. EncryptionMethod is used for both encrypted data and encrypted keys, but not every algorithm can be used in both places. Figure 10 shows where each algorithm can be used. Note that Advanced Encryption Standard (AES) is also available with 192- and 128-bit keys; you just change the key size in the URI.

Figure 10 Where to Use Encrypting Algorithms (Close Figure 10)

 | URI Properties of EncryptedXml Class | Encrypting Data | Encrypting Keys
AES | XmlEncAES256Url |  |
 | XmlEncAES256KeyWrapUrl |  |
DES | XmlEncDESUrl |  |
TripleDES | XmlEncTripleDESUrl |  |
 | XmlEncTripleDESKeyWrapUrl |  |
RSA | XmlEncRSA1_5Url |  |

Before any data can be encrypted or decrypted, the encryption engine needs to know which key should be used for encryption and decryption. Keys can be identified in two ways, the easiest of which is to assign the key a name and place a KeyName element within the KeyInfo element. The application decrypting the document will be responsible for taking the KeyName tag and providing the key that matches the name given.

Instead of providing only the key's name, a key can also be embedded directly into the KeyInfo element as an EncryptedKey. EncryptedKey contains the same elements as EncryptedData: the encryption method, the key used to decrypt this key, and the cipher data that makes up the encrypted key.

Encrypted keys are often used in combination with named keys as random session keys. First a random session key is generated and used to encrypt the XML. Then the session key is itself encrypted with a named key that is known by anyone who needs to decrypt the document. Finally, the named key would be inserted into the key info element of the encrypted session key, and the encrypted session key attached to the encrypted data. The sample in Figure 9 shows this. The encryption method for the XML data is 256-bit AES, and the key for the AES algorithm has been encrypted. The EncryptedKey element contains information about how the key for the AES algorithm was encrypted. In this sample, it was encrypted using an RSA key named recipients_public_key. The applications using XML Encryption would have to map this name to an actual key.

The actual encrypted data may either be embedded into the EncryptedData element or placed in a separate location and then referenced from EncryptedData. If the cipher data is to be placed directly into EncryptedData, it is placed in a CipherData element as Base64-encoded binary. The example shown in Figure 9 uses a CipherData element.

Another scenario involves placing the encrypted data outside the EncryptedData element. The cipher text might be placed anywhere from another element in the document to a remote Web site. In both cases, a CipherReference is used instead of a CipherData element (see Figure 11).

Figure 11 CipherReferences (Close Figure 11)

CipherReference Location | URI Format | Cipher Text Format
Same document | #order | Base64 string
Remote Web site | http://www.example.com/order.bin | Binary

CipherReferences to remote Web sites present an interesting security scenario. In the process of decrypting the XML, the decryption engine will have to go to an arbitrary Web site and download the cipher text. Since the document being decrypted may not be trusted as much as the code that's doing the decryption, this operation should be done in a sandbox. This is accomplished by providing evidence to the EncryptedXml object that it will use when resolving any CipherReferences.

Sandboxing allows for safer code execution since the decrypting application may not have the same permissions as the site that supplies the encrypted data. For instance, if an application attempts to decrypt an untrusted site, and the untrusted site does not have access to some trusted data on a secure, trusted site, it could cause the decrypting application to access that file for it by including a cipher reference.

Since the .NET security policy revolves around evidence, generally the evidence supplied should include at least Site, Zone, and Url objects, as shown here:

// Create evidence based on the referring document
Evidence evidence = new Evidence();
evidence.AddHost(new Zone(SecurityZone.Internet));
evidence.AddHost(new Site("untrustedsite"));
evidence.AddHost(new Url("untrustedsite/encrypted.xml"));

EncryptedXml exml = new EncryptedXml(untrustedDoc, evidence);

XML Encryption Example

Now let's look at how to use the classes in .NET Framework 2.0. This example shows a Web site that sells CDs. Each purchase is filed as an XML document that contains details on what was ordered, shipping information, and a credit card number. Figure 12 shows some sample XML for an order.

Figure 12 XML of a CD Order (Close Figure 12)
<order>
   <purchase>
     <item quantity="1">Def Leppard: Pyromania</item>
     <item quantity="1">Ozzy Osbourne: Goodbye to Romance</item>
   </purchase>
   <shipping>
     <to>Shawn Farkas</to>
     <street>One Microsoft Way</street>
     <zip>98052</zip>
   </shipping>
   <payment>
     <card type="visa">0000-0000-0000-0000</card>
   </payment>
</order>

In this instance, it is acceptable for anyone inside the company to see the items in the order, but you might want to secure some of the more sensitive data such as the shipping address and credit card information. In fact, you might even want to encrypt them separately, so that only the billing department has access to the credit card information, and only the shipping department has access to the shipping address.

To accomplish this, the portion of the XML under the payment element would be encrypted so that only the billing department has access to it. A separate key that is only available to the shipping department would be used to encrypt the shipping element. Finally, the entire order would be encrypted with a key available to anyone in the company.

The first step is to create an EncryptedXml object. This is done by passing in the document that has the data to be encrypted or decrypted, as shown here:

// Assumes the order is in the order.xml file.
XmlDocument doc = new XmlDocument();
doc.Load("order.xml");
EncryptedXml exml = new EncryptedXml(doc);
Before the XML can be encrypted, the keys to be used have to be mapped to their corresponding names. These are the names that will appear in KeyName elements. You use the AddKeyMapping method of the EncryptedXml for this:
// Set up the key mapping. Assumes a method called GetBillingKey
// that returns the RSA key for the billing department.
RSA billingKey = GetBillingKey();
exml.AddKeyNameMapping("billing", billingKey);

Once the key-to-name mapping is set up, encrypting the XML is easy. The first step is to call the Encrypt method, which does the actual encryption and returns an EncryptedData object representing the encrypted portion of the document. After that, you call a utility method to swap the unencrypted portion of the original XML document for the new encrypted data:

// Find the element to encrypt.
XmlElement paymentElement =
  doc.SelectSingleNode("//order/payment") as XmlElement;
// Encrypt the payment element, passing in the key name.
EncryptedData encryptedPayment =
  exml.Encrypt(paymentElement, "billing");
// Swap the encrypted element for the unencrypted element.
EncryptedXml.ReplaceElement(paymentElement, encryptedPayment, true);
This will result in the encrypted XML shown in Figure 13.

Figure 13 CD Order with Payment Information Encrypted (Close Figure 13)
<order>
  <purchase>
    <item quantity="1">Def Leppard: Pyromania</item>
    <item quantity="1">Ozzy Osbourne: Goodbye to Romance</item>
  </purchase>
  <shipping>
    <to>Shawn Farkas</to>
    <street>One Microsoft Way</street>
    <zip>98052</zip>
  </shipping>
    <payment>
      <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
           xmlns="http://www.w3.org/2001/04/xmlenc#">
        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/
             xmlenc#aes256-cbc" />
        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
          <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
            <EncryptionMethod Algorithm="http://www.w3.org/2001/04/
                 xmlenc#kw-aes256" />
            <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
              <KeyName>billing</KeyName>
            </KeyInfo>
            <CipherData>
              <CipherValue>Sce6lLD+u2f8HzPFyuGxTF32z4mb2ugql3JuJIPAqIP98iYs+Muhqg==
              </CipherValue>
            </CipherData>
          </EncryptedKey>
        </KeyInfo>
        <CipherData>
          <CipherValue>FXKC(3 lines of Base64 text)ApqQt</CipherValue>
        </CipherData>
      </EncryptedData>
    </payment>
</order>

Looking at the encrypted data, you can see the various parts that we've described. First, the EncryptionMethod element shows the URI for AES-256, meaning that the document was encrypted with the AES algorithm (as implemented by the RijndaelManaged class) with a 256-bit key.

The Encrypt method generates a random session key for you; this key is encrypted in the KeyInfo element. From looking at the EncryptedKey element you can see that it was encrypted with the RSA algorithm by a key named "billing." The CipherData element holds the encrypted value of the key. Following the KeyInfo is the CipherData that contains the encrypted contents of what used to be the payment element.

Decrypting the document just shown is easy, thanks to the DecryptDocument method of EncryptedXml. First, you load the document with encrypted content:

// Assumes the encrypted order is in encrypted.xml
XmlDocument doc = new XmlDocument("encrypted.xml");
EncryptedXml exml = new EncryptedXml(doc, documentEvidence);
Next, you set up the key name mappings, like this:
// Set up the key mapping. Assumes a method called GetBillingKey
// that returns the RSA key for the billing department.
RSA billingKey = GetBillingKey();
exml.AddKeyNameMapping("billing", billingKey);
Finally, you call DecryptDocument. This method will take care of decrypting the cipher text and replacing the encrypted XML with its decrypted contents.
// Decrypt the encrypted XML in the document
exml.DecryptDocument();

Behind the scenes, the encryption engine will look for any EncryptedData elements when DecryptDocument is called. In this case, it finds only the one below the payment element. Once it finds this element, it will look at the KeyInfo clause and see that it holds an encrypted 256-bit AES key. The KeyInfo for the encrypted key will show that it was encrypted with an RSA key named billing. It then checks the key mapping table for a key named billing. We added a key mapping for a key named billing, so the engine then decrypts the encrypted key. Now that it has the key, the engine decrypts the CipherData. The results of the decrypted CipherData are then swapped for the EncryptedData element. The engine will follow this process for each EncryptedData element it finds in the document when DecryptDocument is called.

X.509 Support

One of the challenges in security is trust. Securely distributing and storing trusted public keys on a network is not easy. Windows® provides lots of infrastructure for solving this problem, however. The Cryptographic API (CAPI) provides support for distributing, manipulating, and storing X.509 certificates. The support that CAPI provides is also known as Public Key Infrastructure (PKI).

An X.509 certificate provides additional information about a key. Certificates specify who issued the key, to whom the key was issued, and when the certificate is valid, along with the actual key information. Certificates are issued by Certificate Authorities (CAs). A CA is an entity (an external company or your company's IT department, for example) that you trust to vouch for the identity of a certificate holder. If you have a valid certificate containing your billing department's public key that is issued by a CA you trust, you can be sure the public key is really the billing department's.

The Windows PKI support allows you to securely store certificates in a certificate store on a computer, add and remove certificates in certificate stores on machines across the network, add and remove trusted CAs on machines across the network, and obtain and verify information about individual certificates, among other things. This allows you, for example, to distribute the certificate containing the public key for your company's billing department to all the computers on your company's network. In the .NET Framework 1.x, you had to call unmanaged APIs to take advantage of most of this support. In the .NET Framework 2.0, most of these APIs are available in managed code, via the X509CertificateEx class and related classes. The XML Signature classes support the X509CertificateEx class directly.

To create an XML signature with a certificate, you simply get the private key from the certificate's PrivateKey property and use it for the SignedXml object's signing key.

// Use the private key from the certificate. Assumes a SignedXml
// object in sig and an X509CertificateEx object in cert.
sig.SigningKey = cert.PrivateKey;
Of course, the certificate must have a private key associated with it, or the PrivateKey property returns null.

You can also add the X.509 certificate information to the KeyInfo element of the signature by using the KeyInfoX509Data class, as shown in the following code:

// Add X.509 certificate info to the KeyInfo element. Assumes a
// SignedXml object in sig and an X509CertificateEx in cert.
KeyInfoX509Data keyInfoX509 =
  new KeyInfoX509Data(cert, X509IncludeOption.EndCertOnly);
sig.KeyInfo.AddClause(keyInfoX509);

To verify an XML signature signed with a certificate's private key, you can call a new overload of the CheckSignature method of the SignedXml class. This new overload takes an X509CertificateEx object and a Boolean. If the Boolean is set to true, the method will verify the signature against the public key in the certificate and verify the certificate by checking key usage and building a chain to a trusted root issuer.

// Check the signature against the cert and verify the cert. Assumes a
// SignedXml object in sig and an X509CertificateEx object in cert.
bool verified = sig.CheckSignature(cert, true);

You can also use the CheckSignature method that takes no parameters to verify a signature, but it will not also verify an X.509 certificate if one was used to sign the XML. You would have to recreate the X509CertificateEx object from the KeyInfo and verify it as a separate step.

For more information about creating X.509 certificates for testing purposes, see the Certificate Creation Tool (Makecert.exe) topic in the .NET Framework SDK documentation.

The .NET Framework 2.0 adds a few new transforms and canonicalization algorithms to the signature engine. These include the Exclusive C14N canonicalization algorithm, XML Decryption transform, and the LTA transform. Figure 14 contains a complete table of the transforms and canonicalization algorithms available in the .NET Framework 2.0 signature engine.

Figure 14 Transforms and Canonicalization Algorithms (Close Figure 14)

Class | Description
XmlDecryptionTransform | Decrypts encrypted XML
XmlDsigBase64Transform | Decodes base64 encoded data
XmlDsigC14NTransform | Performs C14N canonicalization (see http://www.w3.org/TR/xml-c14n for more information)
XmlDsigEnvelopedSignatureTransform | Removes an enveloped signature from a document
XmlDsigExcC14NTransform | Performs exclusive C14N canonicalization (see http://www.w3.org/TR/2002/REC-xml-exc-c14n-20020718 for more information)
XmlDsigXPathTransform | Applies an XPath filter to the input XML
XmlDsigXsltTransform | Applies an XSLT transform to the input XML
XmlLicenseTransform | Implements the LTA transform
XmlDsigC14NWithCommentsTransform | Performs C14N canonicalization, but leaves comments in the canonicalized XML
XmlDsigExcC14NWithCommentsTransform | Performs exclusive C14N canonicalization, but leaves comments in the canonicalized XML

Conclusion

We've discussed the basics of the XML Signature standard and how it is implemented in the .NET Framework 1.x. We've also covered some new features in the .NET Framework 2.0, including support for the XML Encryption standard and X.509 certificate integration for XML signing. These building-block technologies allow you to interoperate with other applications using the standards and build standards support into your own applications.

Mike Downen is a program manager on the CLR security team at Microsoft, working on code access security, cryptography, and the new ClickOnce security model. You can reach Mike at mdownen@microsoft.com.

Shawn Farkas is a software design engineer in test on the CLR security team at Microsoft. He works on ClickOnce, cryptography, and IL verification. See his blog at http://blogs.msdn.com/shawnfa.


Safety in Windows: Manage Access to Windows Objects with ACLs and the .NET Framework

Safety in Windows

Manage Access to Windows Objects with ACLs and the .NET Framework

Mark Novak
This article is based on a prerelease version of the .NET Framework 2.0, formerly code-named "Whidbey." All information contained herein is subject to change.
This article discusses:
  • .NET Framework classes for securing files, directories, registry keys, and other objects
  • How allow and deny rules work
  • Inheritance and propagation of security settings
  • Auditing rules and object ownership
This article uses the following technologies:
Security, C#, .NET Framework
Contents
The Care and Feeding of Security Settings
To Allow and Deny
Got Inheritance?
Parents and Children
A Brief Treatise on Owners
Audit Rules
Taking Snapshots of Security Settings
What Happened to Security Descriptors?
Conclusion

T
hese days, it's not uncommon for a computer system to be accessed by a variety of users, both locally and over a network. Any data these users generate must be kept separate and private. For quite some time, designers of multiuser operating systems have recognized the need for controlling access to files, directories, synchronization objects, network shares, and the like. For instance, in Windows®, each user who logs onto a particular computer is usually assigned her own separate My Documents folder and is prevented from reading and writing to other users' files, unless explicit provisions are made. Student users may be required to submit assignments into a directory while restricted from altering its contents or reading other students' work. A publishing company may have a policy that allows only employees in the desktop publishing department to access the color printer.

Object security is not limited to individual objects—sometimes it applies to entire hierarchies. For example, per company policy the Windows source code on my development machine is accessible only to developers in the Windows organization. The list of applications for object security is virtually inexhaustible.

Support for securing objects has been provided by the Windows family of operating systems for some time. The functionality first appeared in Windows NT®, and evolved only slightly in subsequent releases (Windows 2000, Windows XP, and Windows Server™ 2003). Today, you can secure NTFS files and directories, printers, network shares, Active Directory® objects, and kernel objects, to name a few. The list of securable object types grows all the time. Most often, new security schemes invented by application developers are based on the same set of techniques as the security of existing objects. For example, access control lists (ACLs) that define the security of files and Microsoft® Exchange Server mailboxes are very similar.

Veteran developers using Windows are very familiar with the Windows API support for securing objects, and many articles have been written about access control entries (ACE), ACLs, Security Descriptors, and the Security Descriptor Definition Language (SDDL). However, despite the prominence of object security, manipulating security settings remains a rather difficult development task, even with the gradual introduction of new helper functions by the Windows SDK.

Fortunately, the .NET Framework 2.0 provides an easy, uniform, and extensible way of manipulating the security settings of files, registry keys, directory service containers, and other objects. This article presents the new functionality planned for the Framework and illustrates it with code samples.

The Care and Feeding of Security Settings

Until now, Microsoft did not provide explicit support in the .NET Framework for manipulating security settings. With the .NET Framework 1.x, access can only be granted to users via a series of cumbersome platform invocation (P/Invoke) calls. To make matters worse, not only does this require familiarity with many intricate details pertaining to Windows ACLs and security descriptors, but the resulting code may be prone to a variety of programming errors, some with serious security implications. This is the area where you can find some of the most serious security holes in any application. In perhaps the most famous example of a pitfall of this sort, many readers will recall that setting a NULL discretionary access control list (DACL) on an object grants everyone full access, while an empty DACL does not grant anyone any access.

The .NET Framework 2.0 allows developers to manipulate security settings of objects in a few easy steps using managed code. It does so by introducing the concepts of security objects and rules. The framework classes I will review in this article populate the new System.Security.AccessControl namespace.

Most of my examples will use the file system for illustration. Other objects are secured in much the same way as files, but the file system is the best subject of illustration because it consists of both leaf objects (files) and container objects (directories). Other types of resource managers often have only one or the other type of objects. For example, the registry is purely hierarchical, while mutexes have no hierarchy.

Let's start by looking at how to grant access to a single file. Under the new scheme, access to files is controlled by adding and removing file access control rules from file security objects (see Figure 1). As you can see, changing a file's permissions is accomplished in several steps. First, a file is opened. Then, the file's security object (of type FileSecurity) is retrieved by a call to the GetAccessControl method. This object contains, among other things, an ordered set of access rules which collectively determine the rights of various users and groups to the file.

Figure 1 Add File Access Control (Close Figure 1)
// Start by opening a file
using(FileStream file = new FileStream(
    @"M:\temp\sample.txt", FileMode.Open, FileAccess.ReadWrite))
{
    // Retrieve the file's security settings
    FileSecurity security = file.GetAccessControl();

    // Create a new rule granting Alice read access
    FileSystemAccessRule rule = new FileSystemAccessRule(
        new NTAccount(@"FABRIKAM\Alice"), FileSystemRights.Read,
        AccessControlType.Allow);

    // Add the rule to the existing rules
    security.AddAccessRule(rule);

    // Persist the changes
    file.SetAccessControl(security);
}

In the example, a new access rule is added to the FileSecurity object, granting a user named Alice from the Fabrikam domain read access to the file. Before the change can take effect, it must be persisted to storage. This final step is accomplished by calling the SetAccessControl method.

The previous example illustrated how you can dole out access to users of existing objects. Another equally important scenario is associating security settings with an object at the time of its creation. There is an important security reason for associating access rules with brand new objects. Securable objects are always created with some default security semantics. By default, objects in a hierarchical resource manager (such as file system or registry) inherit their security settings from their parent. Files inherit their security settings from their parent directory, registry keys inherit their security setting from their parent key, and so on. In contrast, nonhierarchical objects, such as mutexes or semaphores, have their default rights assigned to them using system defaults.

The default rights depend on the type of object being created and may not be what you want. For example, rarely would you deliberately create objects to which everyone has full access, yet that may be exactly what the default security settings specify.

The reason you can't simply create an object with default security settings and modify them later is that securing an object after it has been created leaves open a window of opportunity (between creation and modification) when it can be hijacked. Hijacking may lead to a situation where the creator may lose control over a freshly created object, with disastrous consequences. Figure 2 illustrates how to secure a brand new object.

Figure 2 Securing a New Object (Close Figure 2)
// Create a FileSecurity object
FileSecurity security = new FileSecurity();

// Create a new rule granting Alice read access
FileSystemAccessRule rule = new FileSystemAccessRule(
    new NTAccount(@"FABRIKAM\Alice"), FileSystemRights.Read,
    AccessControlType.Allow );

// Add the rule
security.AddAccessRule(rule);

// Create the file passing in the security settings
FileStream file = new FileStream(
    @"M:\temp\sample.txt", FileMode.CreateNew, FileAccess.ReadWrite,
    FileShare.None, 4096, FileOptions.None, security);

The steps should look familiar by now. The same basic operations are taking place, but the order is different and there is no need to persist the changes (because the object is brand new). Before a file is created, a FileSecurity object is created and populated with the desired access rules. Afterwards, the FileSecurity instance is passed to the file's constructor, and the file is properly secured from the beginning.

To Allow and Deny

There are two types of access rules: "allow" and "deny." You can always determine which type of rule you are looking at by examining the rule's AccessControlType property. By convention, deny rules always take precedence over allow rules. Thus, if you add two rules to an object: "grant everyone read and write access" and "deny Bob write access," Bob will be denied write access despite being a member of a group (everyone) that has that right. He will still be allowed read access, though. Figure 3 shows how you can list an object's access rules.

Figure 3 Listing an Object's Access Rules (Close Figure 3)
// Start by opening a file
using(FileStream file = new FileStream(
    @"M:\temp\sample.txt", FileMode.Open, FileAccess.ReadWrite))
{
    // Retrieve the file's security settings
    FileSecurity security = file.GetAccessControl();

    // List each rule in order
    foreach(FileSystemAccessRule rule in
        security.GetAccessRules(true, true, typeof(NTAccount)))
    {
        Console.WriteLine("Rule {0} {1} access to {2}",
            rule.AccessControlType == AccessControlType.Allow ?
                "grants" : "denies",
            rule.FileSystemRights,
            rule.IdentityReference.ToString());
    }
}

The code in Figure 3 will generate output similar to this:

Rule denies Write, Synchronize access to FABRIKAM\Bob
Rule grants FullControl access to BUILTIN\Administrators
Rule grants FullControl access to NT AUTHORITY\SYSTEM
Rule grants FullControl access to FABRIKAM\Alice
Rule grants ReadAndExecute, Synchronize access to BUILTIN\Everyone

Conveniently, access rules are exposed as a collection. This collection is read-only so all modifications to its rules must be performed through dedicated methods of the FileSecurity object, such as AddAccessRule, SetAccessRule, and RemoveAccessRule. Rule objects inside a collection are also immutable.

To understand why deny rules take precedence over allow rules, you have to know how the access check algorithm works. When an access check is performed, rules are evaluated in the order in which they appear inside the access control list. In the example shown in Figure 3, when Bob's access is checked, the rule denying Bob read access is evaluated before the rule granting BUILTIN\Everyone read and execute access. Once a decision to either allow or deny is made, evaluation stops. This is why deny rules "work." If placed after an allow rule, they would not always perform their desired function.

Just as new access rules can be added, existing ones may be removed. Be aware, though, that there is a difference between removing a right from a user and outright denying that right. For example, suppose that Alice is a member of the "full-time employees" group, and that the security settings are: "Full-time employees can read the file" and "Alice has read and write access." Under this arrangement, removing Alice's read right is going to produce the following rules: "Full-time employees can read the file" and "Alice has write access." This is hardly what you would expect since removal of Alice's read access produced no net effect: Alice can still get read access as a full-time employee. If your goal is to be assured that access is not granted to Alice, the only way to accomplish it is by adding a deny rule.

As an aside, if the object contains no access rules at all, everyone will be denied all access to the object.

Got Inheritance?

The examples we've seen so far were pretty straightforward, in part because they have only dealt with simple leaf objects that have no children. Things get a little more complicated once you move from leaf objects such as files, semaphores, and mutexes to container objects such as directories, registry keys, and active directory containers. The additional complexity stems from the fact that the access rules of a container may be configured to apply not just to the object itself, but to its child objects, child containers, or both. Enter the world of inheritance and propagation settings.

Before we go any further in this discussion, a bit of background is in order. Every access rule is either explicit or inherited (the IsInherited property is used to determine which is which). Explicit rules are those that have been added to an object via an explicit action performed on that object. In contrast, inherited rules come from a parent container. When working with an object, you can only manipulate its explicit rules. When adding a new explicit rule to a container, you can specify two sets of flags: one flag for inheritance and one flag for propagation.

There are two inheritance flags: container inherit (CI) and object inherit (OI). Rules that specify container inherit will apply to children of the present object that are containers. Object inherit rules will apply to child leaf objects. With propagation flags set to None, these relationships are transitive: they will span the entire subtree of the hierarchy below the current container and apply to its children, grandchildren, and so on.

The code sample shown in Figure 4 will grant Alice read rights to all subdirectories of the parent directory. Bob will get write access to all files inside the parent directory. Finally, Carol will get both read and write access to both children directories and files. All access rights granted in this example will apply to the directory object itself as well as to the specified subobjects.

Figure 4 Granting Access Rights (Close Figure 4)
// Retrieve current security settings for the target directory
DirectorySecurity security = Directory.GetAccessControl( @"M:\temp" );

// Create new rules
FileSystemAccessRule ruleAlice = new FileSystemAccessRule(
    new NTAccount(@"FABRIKAM\Alice"), FileSystemRights.Read,
    InheritanceFlags.ContainerInherit, PropagationFlags.None,
    AccessControlType.Allow );

FileSystemAccessRule ruleBob = new FileSystemAccessRule(
    new NTAccount(@"FABRIKAM\Bob"), FileSystemRights.Write,
    InheritanceFlags.ObjectInherit, PropagationFlags.None,
    AccessControlType.Allow );

FileSystemAccessRule ruleCarol = new FileSystemAccessRule(
    new NTAccount(@"FABRIKAM\Carol"),
    FileSystemRights.Read | FileSystemRights.Write,
    InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit,
    PropagationFlags.None, AccessControlType.Allow );

// Add the rules to the existing security settings
security.AddAccessRule( ruleAlice );
security.AddAccessRule( ruleBob );
security.AddAccessRule( ruleCarol );

// Persist the changes
Directory.SetAccessControl( @"M:\temp", security );

Now that we understand inheritance flags, we can turn our attention to propagation flags. Again, there are two kinds: inherit only (IO) and no-propagate inherit (NP). The inherit only flag simply means that the rule applies to child objects only and not to the object itself. Thus, you can grant rights just to files and directories inside a parent directory without affecting the security semantics of the parent itself. The no-propagate inherit flag is a little trickier and fortunately not very common. Its presence indicates that the rule is not transitive; it will affect children, but not grandchildren. It is worth noting that propagation flags are meaningless without at least one inheritance flag. It is indeed an error to try to specify them in that way.

The diagram in Figure 5 illustrates how various combinations of inheritance and propagation flags affect the container (C), child containers (CC), child objects (CO), grandchild containers (GC), and grandchild objects (GO). For each set of inheritance and propagation flags, Figure 5 shows the affected objects in yellow and unaffected the objects in blue. As you can see, the possible range of options available to you is quite rich.

Figure 5 How Inheritance and Propagation Flags Affect Containers (Close Figure 5)

Figure 6 shows how various combinations of inheritance and propagation flags help address common scenarios. It is interesting to note how the explicit and inherited rules are laid out. When you list access rules of a container, they will form several sections (see Figure 7).

Figure 6 Inheritance and Propagation Flags in Common Scenarios (Close Figure 6)

CI | OI | IO | NP | Rights (FileSystemRights enum values) | Explanation
 |  |  |  | DeleteSubdirectoriesAndFiles | The caller will be able to delete any file or directory within a parent directory, but will not be able to delete the directory itself
 |  |  |  | CreateFiles | The caller will be able to create files under this directory or any of its subdirectories
 |  |  |  | CreateFiles | The caller will be able to create files under any subdirectory of this directory, but not inside the directory itself
 |  |  |  | ReadData | The caller will be able to read all files under this directory and under any of its subdirectories
 |  |  |  | ReadData | The caller will be able to read all files under this directory, but not under any of its subdirectories
Figure 7 Container Access Rules
Figure 7 Container Access Rules

As mentioned already, the order of the rules is important since it determines the precedence and ultimately affects how the object can be accessed. While you cannot change the default order, understanding it is important in decoding what type of access is going to be granted by a set of rules. Most importantly, all inherited rules always follow the explicit rules. This way, explicit rules always take precedence over the inherited ones. In turn, parent rules take precedence over grandparent rules, and so on.

Parents and Children

What happens if you want to shield your object from the security semantics bestowed upon it by its parent? A mechanism does, in fact, exist that allows you to say "the security settings of my parent will apply to me no more." At that point, you can even specify whether you want to keep the inherited rules as they exist at that instance, but refuse to "listen" when the parent's settings change, or you can wipe out all inherited rules outright. This is accomplished by access control protection, as Figure 8 illustrates.

Figure 8 Access Control Protection (Close Figure 8)
// Start by opening a file
using(FileStream file = new FileStream(
    @"M:\temp\sample.txt", FileMode.Open, FileAccess.ReadWrite))
{
    // Retrieve the file's security settings
    FileSecurity security = file.GetAccessControl();

    // Protect the rules from inheriting the parent's security settings
    security.SetAccessRuleProtection(
        true,    // changes from parent won't propagate
        false ); // do not keep current inheritance settings

    // Persist the changes
    file.SetAccessControl(security);
}

Note that while you can use this technique to shield yourself from receiving inheritance from your parent, there is no way to receive inheritance that your parent did not intend for you. Propagation only takes place to objects whose ACLs are not protected. The only thing you can do is take a snapshot of the inheritance settings before the parent changes its mind because once your access rules are protected, they remain that way and the parent cannot override that.

A Brief Treatise on Owners

The concept of "owner" is special to object security. An owner is endowed with special powers; even if the rules associated with the object prevent a user from accessing it, if that user is the owner, he or she can override the existing rules and get back the control of the object. The process for doing so is no different from regular manipulation of access rules.

The security object also allows you to change the owner, but the operating system will prevent others from doing this. Generally, in order to change the owner you must either have the TakeOwnership permission to the object or have the special "Take Ownership" privilege. The following code illustrates how you change the owner (provided you have the right to do so):

// Retrieve the file's security settings
FileSecurity security = file.GetAccessControl();

security.SetOwner(new NTAccount(@"FABRIKAM\Dave"));

// Persist the changes
file.SetAccessControl(security);

You can also look to see who the current owner of an object is or ask for the owner to be returned as a security identifier or a Windows NT account object:

// Request the object's owner as a SID
SecurityIdentifier sid =
    (SecurityIdentifier)security.GetOwner(typeof(SecurityIdentifier));
// Will display something like "S-1-5-21-8-9-10-10000"
Console.WriteLine(sid.ToString());

// Request the object's owner as a Windows NT Account:
NTAccount nta = (NTAccount)security.GetOwner(typeof(NTAccount));
// Will display something like "FABRIKAM\Alice"
Console.WriteLine(nta.ToString());

Audit Rules

So far, I've talked only about access control rules. In Windows lingo, these form the DACL of the object. The DACL can be changed at the discretion of the owner of the object, hence the term "discretionary." It can also be changed by anyone upon whom the owner has bestowed the right to change the DACL.

The object's security descriptor contains another list of rules called the system access control list (SACL). This list controls what type of auditing will be performed on the object by the system. Auditing is a security-sensitive operation. In Windows, audits can only be generated by the Local Security Authority (LSA), because the LSA is the only component allowed to write entries into the security event log, where audits are stored. Security audits are a very serious business. Audits are intended to be used in computer forensics when trying to analyze, after the fact, who did what and who attempted to do what on the system. Many organizations keep their audit logs for years.

Needless to say, the settings dictating what gets audited are usually kept under strict administrative control. If you try the code examples in this section and run into an UnauthorizedAccessException message, it is likely because the account you're running under does not contain the "Security Privilege." This powerful privilege must be assigned to your account by the local machine policy in order for you to be able to modify or even examine SACLs. Good luck trying to ask your friendly local system administrator to grant you this privilege!

Despite these dire warnings, reading and manipulating audit settings of an object, once you have the necessary privilege, is similar in all respects to modifying access control settings. If you paid close attention to code examples in the beginning of this article, you should experience an odd sensation of déjà vu when you look at Figure 9.

Figure 9 Manipulating Audit Rules (Close Figure 9)
// Start by opening a file
using(FileStream file = new FileStream(
    @"M:\temp\sample.txt",FileMode.Open, FileAccess.ReadWrite))
{
    // Retrieve the file's security settings
    FileSecurity security = file.GetAccessControl(
        AccessControlSections.Audit);

    // Create a new rule to generate audits any time that a full-time
    // employee is denied write access to the file
    FileSystemAuditRule rule = new FileSystemAuditRule(
        new NTAccount( @"FABRIKAM\Full_Time_Employees"),
        FileSystemRights.Write, AuditFlags.Failure);

    // Add the rule to the existing rules
    security.AddAuditRule(rule);

    // Persist the changes
    file.SetAccessControl(security);
}

Auditing settings are expressed as audit rules. You can specify the name of a security principal (user or group) that you want to audit, what types of access you are interested in (such as read, write, and so on), and whether you want audits to be generated on access being granted, denied, or both. For instance, in the example shown in Figure 9, an audit will be generated by the system any time that a full-time employee is denied write access to a file or a directory under a given parent directory. Inheritance flags, propagation flags, and protection settings all work exactly the same with audit rules as they do with access control rules.

Taking Snapshots of Security Settings

The Windows security system keeps security settings of every object alongside that object in its security descriptor. You can access the security settings when you want to either display it or make changes, but generally they stay with the object (such as the file or mutex) being protected.

Sometimes the need arises to take a snapshot of the security settings. This can be done for a variety of reasons. You might want to take an object's security settings and apply them to another object. Or, you might want to persist them inside of an XML file. For these sorts of applications, a special textual representation was created called the Security Descriptor Definition Language (SDDL). This is not so much a language as a gobbledygook string representation that very few people know how to read and interpret. Even at Microsoft, few engineers will be able to make sense of something like the following:

D:P(A;;GA;;;SY)(A;;GRGWGX;;;BA)(A;;GR;;;WD)(A;;GR;;;RC)
Cryptic though it is, SDDL does work and is an excellent way of representing a security descriptor in a compact and quasi-human-readable form. To read and interpret SDDL, you should consult its documentation in MSDN®.

Armed with a FileSecurity object, we can obtain either the binary or SDDL form of its security settings by using the appropriate GetSecurityDescriptorXxxForm method call:

// Retrieve owner and DACL portions as SDDL
string sddl = security.GetSecurityDescriptorSddlForm(
    AccessControlSections.Owner | AccessControlSections.Access);

// Get SACL and DACL portions in binary form
byte[] binary = security.GetSecurityDescriptorBinaryForm(
    AccessControlSections.Audit | AccessControlSections.Access);

What Happened to Security Descriptors?

After reading the preceding pages, any seasoned developer using Windows is probably left wondering where the .NET Framework support is for security descriptors, considering that they have been a fundamental building block of Windows security. Indeed, the Microsoft engineering team tasked with implementing object security for the .NET Framework 2.0 felt that security descriptors were too difficult for an average or even above-average developer to understand. As a result, the team formulated the concepts of security objects and rules.

The security descriptors did not go away entirely, however. The engineering team recognized that advanced developers may require access to security descriptors as well as the ACLs, and ACEs of which they're composed. Indeed, objects such as FileSecurity and FileSystemAccessRule are internally backed by a whole family of public classes implementing security descriptors, ACLs, and ACEs. That entire class hierarchy is too complex to be documented here, but a few things are worth highlighting.

ACEs are represented by a family of classes deriving from a base "GenericAce" class. ACEs are immutable objects providing properties such as access mask, security identifier, and various flags (whether the ACE is inherited, propagation flags, and so on).

ACLs also form a tree of classes, rooted at GenericAcl. An ACL is essentially an ICollection of ACEs, but it provides additional functionality such as converting to and from binary form, which is useful for interoperability with the Windows API. Notable subclasses of GenericAcl include an advanced and powerful RawAcl, allowing arbitrary ordering of all ACE types, and CommonAcl, reflecting properly ordered ACLs containing a well-defined set of ACEs. CommonAcl is what you would see on a garden-variety file in your file system.

Finally, security descriptors, rooted at GenericSecurityDescriptor, aggregate owner and group SIDs, a DACL, SACL, and a set of security descriptor control flags. Security descriptor classes provide methods for converting them to and from SDDL and binary form, among other features. A RawSecurityDescriptor subclass of GenericSecurityDescriptor is a general and powerful tool that allows developers to create arbitrary security descriptors, while CommonSecurityDescriptor mirrors commonly encountered security descriptors, with ACLs in their "canonical order."

All in all, this class library should be suitable for both novice and advanced developers, as it places no practical limitations on the type of manipulations you might want to perform on security descriptors, while retaining a simple and attractive rules-based interface for the most commonly performed operations.

Conclusion

This article illustrated the manner in which objects can be secured in the upcoming release of the .NET Framework 2.0. Developers with experience in implementing access control using the Win32® API will find welcome relief from dangerous and difficult manipulations of low-level data structures that plagued "pure" Windows-based programming of these features. The ease with which the new library of classes allows you to secure objects should lead to improved application security. Low-level functionality was retained behind a simplified veneer so that advanced developers will be able to continue to use features familiar to them and to create security descriptors of arbitrary complexity.

Mark Novak is a Software Design Engineer at Microsoft. He worked on several versions of Microsoft Exchange Server before joining the Windows Core Security Team. Mark can be contacted at markpu@windows.microsoft.com.


Intrusion Prevention: Build Security Into Your Web Services with WSE 2.0 and ISA Server 2004

Intrusion Prevention

Build Security Into Your Web Services with WSE 2.0 and ISA Server 2004

Dino Esposito
This article discusses:
  • Authentication techniques for .NET Web services
  • Security and policy in WSE 2.0
  • The use of firewalls in protecting Web services
  • ISA Server 2004 and WSDL
This article uses the following technologies:
WSE 2.0, Web services, XML, C#, Security
Contents
Desperately Seeking Standards
The Moral of the Story
Putting WSE 2.0 into Perspective
Authenticating Users
Integrity and Confidentiality
Policy and Assertions
ISA Server 2004 At a Glance
Web Service-Specific Firewalls
Call to Action

T
here are aspects of software design and development many of us would like to overlook because they're just not fun. The work we do in security, performance tuning, and optimization may not be glamorous, but it is essential. And while security has always been important, it is especially critical to today's Web applications because of the great financial risk that is posed by insecure code.

Software security is a team effort. An application cannot be secure if the network on which it runs is insecure. Likewise, proper administration is ineffective if the code is wide open to malicious attack. While proper administrative policy may be at the foundation of security, even the most restrictive policy won't be able to keep inherently insecure code safe.

To achieve security for any piece of software exposed over the Web, you must enforce network and application security. The primary network threats to guard against include denial of service (DoS), spoofing, and eavesdropping. A more complete list of threats and a description of various threat models can be found at http://msdn.microsoft.com/security. Such attacks can result in the theft of passwords and other critical data, as well as request floods. They also provide attackers with an understanding of your network topology. Examples of threats aimed at the application include the notorious SQL injection and cross-site scripting (XSS) attacks.

In this article, I'll focus on some of the issues surrounding .NET Web services security. I'll review authorization, authentication, confidentiality, encryption, integrity, and policy, and spend a good deal of my time talking about what you can do in and around your code to fend off attacks. In addition to the hacker-resistant programming techniques I will discuss, I'll will explain the benefits of using a SOAP-aware firewall to stop potentially injurious calls from reaching your Web services. One of these smart Web service-specific firewalls is available with Internet Security and Acceleration (ISA) Server 2004. I'll take a look at it later on and explain the ad-vantages that XML intrusion prevention can bring to your system.

Desperately Seeking Standards

Security for Web services comes at three levels. First, you must ensure that the message has not been tampered with. Second, you must verify that the user sending the message provides valid credentials and is authorized to have the message processed. Third, you need to check that any assertions made for processing the message are fulfilled (including assertions defined using regular expressions on the content of the message). The implementation of these features is left to the developer, but the industry is coalescing around standards (and related developer's toolkits) that should be taken into careful account.

Imagine you need to build a .NET Web service that implements authentication and ensures that only authorized users can invoke methods. The interaction between a client and a .NET Web service consists of sending a packet (typically an HTTP request) to a public URL (typically an ASMX resource). Any request that hits IIS is first processed by the Web server. If the request is considered valid, control of the request passes to the ASP.NET application. If the ASP.NET application is configured for Windows® authentication, the security token generated by IIS is set as the identity of the authenticated user. Next, ASP.NET looks at whether that user is authorized to access the requested ASMX resource.

You don't need to be a security guru to see that such a built-in security mechanism available in the Microsoft® .NET Framework for Web services is somehow limiting in the real world. To allow or deny specific users access to the Web service, either you use the ASP.NET configuration system, providing a list of users and rules that are authorized to access the specified resource (such as that done in the Web.config snippet shown in Figure 1), or you set access control lists (ACLs) on the .asmx file itself. In the end, ASP.NET Windows authentication is mostly appropriate for intranet applications that authenticate users in the current domain. Windows authentication is simply too unwieldy for Internet applications where you may have to manage thousands or even millions of users. What can you do at this point?

Figure 1 Allow and Deny Specific User Access (Close Figure 1)
<configuration>
  <system.web>
    <authentication mode="Windows" />
  </system.web>

  <location path="myservice.asmx">
    <system.web>
      <authorization>
        <allow users="Administrator"/>
        <allow users="DOMAIN\Davolio"/>
        <deny roles="BUILTIN\Power Users"/>
      </authorization>
    </system.web>
  </location>
</configuration>

To authenticate a user who is attempting to execute a WebMethod, credentials must be passed along with the request. The format you use for the credentials is custom and service specific. In general, valid credentials are any sequence of values that the Web service can validate and successfully transform into a user identity. At a minimum, valid credentials include a user ID and a password. Some real-world services like Amazon.com also employ user tokens, such as alphanumeric strings unequivocally linked to an identity record in a number of ways.

How can you pass these user tokens to the service? You could, for example, wrap the user token in a custom SOAP header or have it passed as an extra argument to each and every method that requires authorized users. It would then perhaps be wise to use an HTTP module to filter incoming requests for authenticity before the request is served and the method is executed. This allows you to factor the authentication out of the individual service methods.

A SOAP message can optionally contain user-defined headers that solely serve to transmit metadata that is not strictly related to the semantics of a Web service call. This data can include security credentials. A Web service that intends to implement a custom authentication layer based on SOAP headers must do a couple of things. First, it must indicate that the SOAP header is expected to contain the authentication credentials. Second, it must process the credentials and authorize or deny the request.

In the .NET Framework, a custom SOAP header is a class that inherits from SoapHeader, like the simple one shown here:

public class CredentialsHeader : SoapHeader
{
    public string UserName;
    public string Password;
}

This class is bound to the Web service through the SoapHeader attribute, as shown in Figure 2. The Web service, in turn, defines a property to import the values carried by the header. In light of the custom header, the SOAP envelope for each secured method looks slightly different (see Figure 3).

Figure 3 The Soap Envelope (Close Figure 3)
<soap:Envelope ...>
  <soap:Header>
    <CredentialsHeader xmlns="...">
      <UserName>string</UserName>
      <Password>string</Password>
    </CredentialsHeader>
  </soap:Header>
  <soap:Body>
    <GetCustomersOrders xmlns="...">
      <custID>string</custID>
      <year>int</year>
    </GetCustomersOrders>
  </soap:Body>
</soap:Envelope>
Figure 2 Binding the SoapHeader Class (Close Figure 2)
public class CredentialsHeader : SoapHeader
{
    public string UserName;
    public string Password;
}

public class MyService : WebService
{
    // Custom authentication header
    public CredentialsHeader UserToken;

    [WebMethod]
    [SoapHeader("UserToken", Required=true)]
    public DataSet GetCustomersOrders(string custID, int year)
    {
       // Forward a message to the logic

       // Collect data and return
    }

    •••
}

The values in the SOAP header are used to populate the corresponding Web service property (the UserToken property of Figure 2). On the client side, the proxy C# or Visual Basic® .NET file generated for the Web service contains another [SoapHeader] attribute for each secured method. The attribute guarantees that any SOAP packet issued for that method will contain the extra header.

Unfortunately, this approach will result in passwords sent to the server as clear text. One thing you can do to work around this problem is hash the password before sending it to the server. The server can then compare the hashed password to a similarly hashed string stored in a server database. In this scenario, the code to hash the password must be placed within the client application making the call. In addition, the hash algorithms used to hash the password on the client and on the server (for storing in the database) must match. Note, however, that sending hashed passwords doesn't protect you against replay and dictionary attacks. In the end, use of a username token is not a great idea unless the underlying transport is inherently secure, such as secure sockets layer (SSL), or unless you encrypt the username token, for example. I'll return to this point later in this article when discussing the .NET security framework for Web services—Web Services Enhancements (WSE) 2.0 (now at Service Pack 1).

Let's consider where the authentication takes place. Certainly it doesn't happen in the WebMethod since it would be improperly factored. If all WebMethods are secured, which is definitely a good practice, at the very least you can use a SOAP extension or an HTTP module to keep authentication code out of the WebMethod. Note that in an HTTP module you can intercept any events that characterize the lifecycle of an HTTP request. In a SOAP extension, you can inspect and modify parts of the SOAP message before and after its serialization and deserialization on the client and server.

Authorization doesn't require much more work once the user's been authenticated, though one more step is needed. You need to parse the SOAP source, retrieve the name of the invoked method, and check if the user is in the list of those who can execute the requested method. Typically, the method/user information is found in a separate data store, such as a SQL database.

SOAP headers are central to the implementation of an authentication and authorization layer for Web services. As the SOAP specification dictates, SOAP messages can include any number of custom headers for carrying any sort of custom information.

The Moral of the Story

Web services are growing quickly, and more and more super-smart people are taking up the challenge. As a result, good ideas are emerging at an amazing pace, but without much control over their implementations. For example, suppose two very smart people have the same raw idea. It's likely that the two ideas differ on some details and require some adjustments to really express the same concept. These adjustments define common rules and naming conventions so that the same concept is expressed using the same words. This is, of course, the purpose of having standards.

Let's say you want to use SOAP headers that contain scrambled passwords. How do you name the additional SOAP headers? How do you hash or encrypt the password? How do you validate parameters before execution? How do you define assertions? The list could go on and on.

Figure 4 Web Service Security Standards
Figure 4 Web Service Security Standards

Web services standards (especially standards that relate to security) have been evolving so rapidly in the past couple of years that software toolkits have had a hard time keeping up with them. Today, the situation is more stable, and a bunch of WS-* specifications explain how Web services security should be implemented in a loosely coupled and platform-independent manner. WS-* specifications include directives about establishing secured communications, defining policies for services interaction, and defining rules of trust between domains of services. Figure 4 shows the stack of emerging security standards and Figure 5 provides a brief description of the main specifications.

Figure 5 Main Web Security Specifications (Close Figure 5)

Specification | Description
WS-Security | Describes enhancements to SOAP messaging to provide protection through message integrity, confidentiality, and authentication. WS-Security also provides a mechanism for associating security tokens with messages and describes how to encode binary security tokens (X.509 certificates and Kerberos tickets) as well as how to include encrypted keys.
WS-Policy | Provides a model and corresponding syntax to describe and communicate the policies of a Web service. It doesn't define any policy assertions itself; it simply defines the framework for packaging and processing assertions in a standard way.
WS-PolicyAssertions | Specifies a set of common message policy assertions that can be specified within a policy. For example, it contains an assertion that validates user-defined expressions in a generalized way.
WS-PolicyAttachment | Specifies mechanisms for associating policy expressions with WSDL type definitions and UDDI entities.
WS-SecurityPolicy | Extends the WS-Security specification and indicates the policy assertions from the WS-Policy specification that apply to security.
WS-Trust | Built on WS-Security, it defines how to request and issue security tokens and manage trust relationships.
WS-SecureConversation | Built on WS-Security and WS-Trust, it defines how to provide secure communication. It also defines mechanisms for establishing and sharing security contexts.
WS-Federation | Defines the mechanisms used to enable identity, authentication, and authorization federation across different trust realms.
WS-Addressing | Defines elements to render in a transport-neutral format (information typically provided by transport protocols and messaging systems). The specification formalizes the receiver of a message (Web service endpoint) and message information headers.
WS-ReliableMessaging | Describes a protocol that allows messages to be delivered reliably between distributed applications in the presence of software component or network failures.

A roadmap to Web services security can be found at Security in a Web Services World: A Proposed Architecture and Roadmap. It presents a broad set of specifications that cover security technologies including authentication, authorization, privacy, trust, integrity, confidentiality, and secure communications channels across a wide range of application and business topologies.

Standards are very important, and as a result, libraries have been developed that incorporate them. Most of the WS-Security and related specifications, such as WS-Policy, are now integrated in Web Services Enhancements 2.0.

Actually, there are far more standards out than are integrated into WSE. All of the WS-Transaction and related standards are not there, for example. As mentioned, the primary standards implemented by WSE are WS-Security and WS-Policy, the first of which is probably the most mature of all the WS-* specifications.

Putting WSE 2.0 into Perspective

WSE 2.0 helps Web service developers implement security best practices using accepted standards. Note, though, that the WSE property dialogs and wizards don't replace a good understanding of Web service security topics, nor are they supposed to.

WSE 2.0 enables developers and administrators to apply security policies for .NET Web services more easily. WSE 2.0 allows you to create signed and encrypted channels for Web services communication using Kerberos tickets, X.509 certificates, user credentials, and other custom binary or XML-based security tokens. In addition, WSE 2.0 supports policies and assertions to validate Web services calls and provides the ability to secure long-running communication via secure conversations. You can download WSE 2.0 from Web Services Enhancements. Aaron Skonnard provides a programmer's perspective of WSE 2.0 in the August 2004 installment of The XML Files column.

Let's see how WS-* specifications standardize and simplify the application of some common security measures such as authentication, authorization, encryption, and validation assertions. I'll use WSE 2.0 to explain these concepts.

Authenticating Users

WS-Security provides a general-purpose mechanism for associating security tokens with Web service method requests. Security tokens can be username and password pairings, X.509 certificates, or Kerberos tickets. WS-Security standardizes the header for credentials using <wsse:Security>. Here's an example:

<wsse:Security soap:mustUnderstand="1">
  <wsse:UserNameToken xmlns:wsu="...">
    <wsse:UserName>user</wsse:UserName>
    <wsse:Password type="wsse:PasswordDigest">R4eU...</wsse:Password>
  </wsse:UserNameToken>
</wsse:Security>
This fragment is an excerpt from a SOAP message that contains username/password credentials. The UserNameToken node maps to a WSE 2.0 class (with the same name) that stores the credentials in two properties: Username and Password. A management class is then expected to process this token. The class is named UsernameTokenManager in WSE 2.0.

Interestingly enough, the UsernameTokenManager class is not abstract (nor is it sealed). Its default implementation authenticates the security token in the received message against a Windows account or an Active Directory® user. If it succeeds, a Windows principal is assigned to the Principal property of the UsernameToken. In most cases, though, you don't want Windows authentication because it requires that a user have an account on the server. This is acceptable for a small set of users (mostly intranets), but not for the thousands of users of an e-commerce Internet application. To implement a custom-authentication algorithm, you derive a class from UsernameTokenManager and override the AuthenticateToken method. For example, if you want to compare the incoming credentials with those stored in a SQL database, you can do so with a derived class as follows:

public class SqlUsernameTokenManager : UsernameTokenManager
{
   protected override string AuthenticateToken(UsernameToken t)
   {
      // Find a match in the SQL database
      // Return the stored password for the specified user
   }
}

Note that AuthenticateToken must return a string intended to be the password read from the data store. The password you return from the method will be compared to the password supplied with the SOAP message, and the authentication succeeds if the two match. If the user can't be authenticated, an exception is thrown.

In most production systems the AuthenticateToken method typically consults an external users database where the "password" is often in actuality a derivative of the real password, such as a hash of it. Most secure applications, in fact, store passwords in the database using a one-way hashing algorithm. This means that it is theoretically impossible to revert the hashed string back to the clear password for comparison. As a result, for authentication to work, the client must be able to send an equally hashed version of the password so that the comparison can occur on hashed strings. As mentioned, though, this is still subject to dictionary attacks. The advantage is that you can check the password's validity without manipulating the actual password for the user.

WSE 2.0 allows you to choose from three options when it comes to sending the password, as shown in Figure 6. The sidebar "Building Password-Secure Authentication Models" summarizes the feasible approaches you can take to build password-secure authentication models for WSE-enabled Web services.

Figure 6 Sending a Password in WSE 2.0 (Close Figure 6)

Value | Description
SendHashed | An SHA-1 hash of the password is sent in the SOAP message.
SendNone | No password is sent in the SOAP message. The option makes sense if a digital signature is attached to the message. The signature must be built from a valid security token (such as credentials, Kerberos tickets, or certificates). The solution makes sense in distributed applications where a single sign-on architecture exists.
SendPlainText | The password is sent in clear text. When using this option, a secure transport channel such as SSL should always be used. However, for a stricter security implementation, SSL should be used with SendHashed too.

The custom security token class must be registered with the Web service application. You can accomplish this in either of two ways. You can manually edit the Web.config file, which is the new section added by WSE 2.0, or you can resort to the WSE 2.0 tabbed dialog. You open the Security tab and add a new security token manager type. A lot of practical information on WSE 2.0 programming can be found in the following article: Securing .NET Web Services with the WS-Security Protocol. If you're particularly interested in WS-Security, be sure to take a look.Building Password-Secure Authentication Mode

If you choose Windows authentication with WSE 2.0, you can only send the password as clear text. In this case, WSE 2.0 gets the authentication token through a call to the LogonUser Win32® API function, which requires the password as plain clear text. It goes without saying that you should be sending unencrypted passwords only over a secure channel (SSL or TLS) or within a SOAP packet that has been previously encrypted using some of the other WSE 2.0 facilities.

If you go for custom user authentication and implement a custom user token class, you can take advantage of the other two options provided for sending passwords: sending a password hashed or sending no password whatsoever. If you choose the former option, WSE 2.0 transmits a hash of your password calculated using the SHA-1 algorithm. On the server, WSE 2.0 retrieves the stored password for the user, applies the same SHA-1 algorithm and compares the result against the transmitted hash. That's why the AuthenticateToken overridable method on the UsernameTokenManager class requires you to return the stored password instead of a simpler Boolean value.

This is a hazard from a general security standpoint. If the server is ever hacked, the attacker can just sit and see clear passwords pass by. This approach can produce severe damage in the case of a compromised server and lays the server open to replay attacks. A replay attack is when someone sees a valid token pass and just inserts the same token in another message.

A way to alleviate such risks consists of using (and storing) pre-hashed versions of the password created using an algorithm such as SHA-1 on which both the sender and the receiver agree. Basically, you transmit the hashed password (either further hashed or clear), rather than the original, and make AuthenticateToken return the hashed password stored in the database. Of course, the hash algorithm must be the same. This trick alone doesn't make the Web service call inherently more secure. Once the system is hacked, it doesn't make any difference if the attacker replays the call using the original or the hashed password. However, at least the hacker doesn't know the original password and can't cause more severe damage to the system.

If custom user authentication is your choice, then an even more secure approach involves adding a signature to the SOAP message along with the user token. If you choose to indicate a digital signature, you can avoid sending the password altogether. The signature built on the client from user name and password (the user token) represents your proof of possession. By doing so, you instruct Web Services Enhancements 2.0 to retrieve the password from the attached signature and compare it to the password that is returned by AuthenticateToken.

The WSE 2.0 user security token object features a property named Principal. The property gets and sets the IPrincipal object for the security token. The principal is set in the AuthenticateToken method immediately after retrieving the password for the specified user. In this case, the principal is a WindowsPrincipal object. Its IsInRole method will look up the user in the list of Windows accounts. If you're using a custom user name token class, you must add some code to your override of AuthenticateToken to generate a new GenericPrincipal object with its own list of roles:

// First, retrieve roles for the authenticated user
this.Principal = new GenericPrincipal(token.Username, roles);

In the WebMethod code, you ensure that the user is a member of the roles authorized to call:

if (token == null || !token.Principal.IsInRole("PowerUsers"))
    throw new UnauthorizedAccessException();
The preceding code ensures that a UsernameToken was used to sign the SOAP message and that the user is, in fact, a member of the PowerUsers role.

Integrity and Confidentiality

Just adding credentials to a SOAP message doesn't make it inherently more secure. At a minimum, you need to add some encryption to ensure that the contents of the SOAP message are not intelligible as they travel over the network. SOAP messages, in fact, are by default plain text and thus can be read by any recipient. An encrypted SOAP message is cryptographically encoded, so that only the owner of the proper decryption key can read the contents of the message.

You might be asking why you should care about SOAP encryption when you can simply use network-level secure protocols, such as SSL and Transport Layer Security (TLS). SSL is easy to set up and works well; however, it has been designed mainly for Web sites and is not always optimal for Web services.

The first aspect to consider is that Web services are often implemented over HTTP, but that is neither necessary nor is it the only option. Microsoft Message Queuing (MSMQ) and SMTP, for example, are valid transport layers as well. WS-Security decouples Web services from HTTP security. Second, SSL encrypts the entire SOAP message and was not designed to protect only selected portions of messages.

This might not seem like a significant issue but consider what happens when a message is routed through multiple locations. Should you permit sensitive data to be visible to everybody who has something to process in the message? It's a good rule to make critical data available to the fewest number of people, so an all-or-nothing encryption scheme may pose a severe security issue. Another point to consider might be performance. Data confidentiality doesn't come for free and encrypting and decrypting the whole message is, computationally speaking, more costly than protecting only portions of it. WS-Security provides you with a flexibility that you just can't obtain out of SSL or similar protocols.

WSE 2.0 supports both asymmetric and symmetric encryption. Asymmetric encryption allows the client of a Web service to encrypt the message using the public key of an X.509 certificate. Subsequently, only the owner of the private key can decrypt the message. The WSE 2.0 Settings Tool contains a certificate wizard to let you secure request and response of a service application.

Symmetric encryption requires that a Web service and client already share a secret key. The sender encrypts SOAP messages using the shared key after which the recipient decrypts the SOAP messages using that same secret key.

The information collected through the WSE 2.0 Settings Tool wizard is recorded in a policy configuration file that contains assertions about the required integrity and/or confidentiality of the message. When you don't specify the portions of the SOAP message to encrypt, WSE 2.0 encrypts the entire contents of the <Body> element, but all of the SOAP headers are left as clear text. As for digital signing, by default WSE 2.0 signs the body, timestamp, and all addressing headers, such as To, ReplyTo, and RelatesTo. You can change these default settings programmatically or by editing the assertions in the policy file. If you want to edit the policy file, locate the <wssp:MessageParts> node below <wssp:Integrity> (for digital signing), or <wssp:Confidentiality> (for encryption).

Policy and Assertions

In summary, the security of Web services has three main facets: encryption/signing, authentication/authorization, and assertions. The first relates to the secure sending of the bytes over the network. While encryption guarantees that the message is not viewed, you also need to ensure that nothing got modified along the way. Digital signatures are one way of providing such integrity as well as providing the identity of the sender. The second aspect relates to the binding between the sender and the requested function. Authentication checks the credentials of the sender (whom he or she claims to be) while authorization verifies whether the sender has rights to perform the operation. The third level of security gets closer to the usage and logic of the Web service and, among other things, is aimed at ensuring that methods are invoked according to accepted predicates. Web services policies are extremely important to this level of security.

WS-Policy provides a flexible grammar for expressing the capabilities and requirements of WebMethods. A policy is a collection of one or more policy assertions. Some assertions specify requirements related to the wire (such as the authentication scheme or transport protocol) while others specify requirements critical to the service usage (such as the privacy policy). WS-Policy provides a unified grammar to allow assertions to be defined consistently.

The WS-PolicyAssertions standard provides an initial set of assertions to address some common needs of Web services. Built-in assertions include the required character set, language, and specification version. A fourth type of assertion is MessagePredicate.

MessagePredicate lets you assert arbitrary expressions to implement predicates and preconditions to which each message must conform. For example, these assertions can implement business rules. You can express assertions in either of two languages: XPath or SOAP message parts. You can find more information and an example in the March 2004 installment of The XML Files column.

The three aspects of Web services security I've examined form a defense perimeter only within the Web service application, which is where most of today's attacks occur. Nowadays, a traditional firewall is not necessarily an effective barrier because attacks tend to look like legitimate traffic to firewalls, which often only inspect the outermost envelope such as the headers of communication packets. To be effective, firewalls must start looking into the actual payload to figure out the intent of a request.

ISA Server 2004 belongs to this new breed of firewalls and has the ability to do deep content inspection and analysis for many popular Internet protocols.

ISA Server 2004 At a Glance

ISA Server 2004 is the next version of the Internet firewall, offering improved network security and performance. Designed from the ground up to provide efficient application-layer filtering, ISA Server 2004 is a rule-based firewall and a powerful HTTP proxy and caching server. Of the several enhancements that the newest version of ISA brings to the table, the one that is most relevant here is the extensible application-layer filtering. In Figure 7, you can see the overall architecture of ISA Server where the custom application filter components stand out.

The important thing going on here is that unlike some other firewall software, ISA Server 2004 looks into the actual content of an HTTP request in addition to inspecting the IP and TCP headers. In this way, ISA is capable of blocking any non-HTTP traffic, any contents brought by packets that look suspicious, and anything else that violates rules and policies that are set administratively. A fine-grained set of security policies give administrators the means to configure the level of security as they want.

Application filters play a key role in the architecture shown in Figure 7. They act like protocol handlers by filtering both outgoing (access) and incoming (published) traffic. ISA Server comes with built-in filters for most common protocols including HTTP, SMTP, and RPC.

Figure 7 ISA Server 2004 Architecture
Figure 7 ISA Server 2004 Architecture

The HTTP filter provides for comprehensive syntax analysis, and the SMTP filter analyzes commands and filters messages for potentially injurious data. Other monitored protocols are DNS and POP3. As I mentioned earlier in this article, this portion of the architecture is extensible and based on a plug-in model. In other words, third-party vendors can extend ISA with ad hoc filters performing particular tasks. Yigal Edery discussed the application filter extensibility mechanism of ISA Server in the March 2004 issue of MSDN®Magazine (see Developing an Application Filter for Microsoft Internet Security and Acceleration Server 2004).

Several companies offer application filters for ISA Server that enhance security and interoperability for different protocols and types of traffic (a list is available at Partner Application Filters). I'd like to focus on one of these products in particular—Forum XWall for ISA Server—because of its unique Web service-specific features to protect against intrusions.

Web Service-Specific Firewalls

Forum XWall (an ISA Server 2004 plug-in) works as a kind of smart firewall that implements certain security features specifically targeted at Web services. Usually calls to Web services pass through ports that firewalls leave open, like port 80 and 443. This allows SOAP and XML messages to flow undetected into the company's internal network. While ISA Server introduces rich content filtering at the HTTP level, Forum XWall complements ISA Server by ensuring the integrity of Web services traffic as it flows through the organization. Basically, XWall is a sentinel against XML intrusions. In Figure 8 you see XWall installed within ISA Server 2004 and configured to monitor the request/response traffic for a few Web Services Description Language (WSDL) hosted on the server.

Figure 8 XWall Installed in ISA Server 2004 (Close Figure 8)

All of the policies in the figure are created by navigating the Publish Web Service or Publish Secure Service menu items in the task list. This is similar to the Publish Web Server and Publish Secure Server tasks that exist in the traditional firewall policies. Each Web services firewall policy is created as a result of a WSDL file being consumed from the local file system or from a Uniform Resource Identifier (URI) or Universal Description, Discovery, and Integration (UDDI) directory.

XWall detects and controls XML and SOAP traffic based on predefined criteria. In doing so, it can check the conformance of the call to the WSDL of the service and prevent parameters tampering, buffer overflows, replay attacks, command injection, and more. XWall can decide whether or not individual XML/SOAP messages, users, and URLs may gain access (such as read, write, execute privileges) to Web services behind the wall.

Figure 9 IDP Rule
Figure 9 IDP Rule

It also enforces data integrity by checking raw XML against approved schemas, user-defined data validation rules, and a set of Intrusion Detection and Prevention (IDP) rules such as giving packets a maximum length and a maximum number of nested tags. Figure 9 shows an IDP rule where the maximum document size attribute is used to limit the SOAP/XML payload size that flows through the firewall.

In some cases, you can also use XWall to virtualize the WSDL based on ACLs so that any call that the authenticated user attempts to direct at a certain WebMethod is blocked at the gate if the user is not authorized. In other words, XWall can take the WSDL of the Web service and map portions of it to individual users, such as those using Active Directory.

Figure 10 Controlling Access to Web Service Methods
Figure 10 Controlling Access to Web Service Methods

Access to Web service methods can be controlled either by mapping each method to a group of users for authorization or just by turning off access to every method. In Figure 10, the Add method can only be invoked by users who are members of the group EU-Partners. The Divide method doesn't require any authorization and thus anyone can invoke it. Finally, access to Multiply and Subtract are simply turned off so no one can access these operations.

Call to Action

As an increasingly essential piece of the enterprise architecture, Web services shouldn't simply be considered a sort of Web site accessed by software rather than humans. To deliver appropriate security to a Web service-based environment, you need to remember that Web services are first and foremost "services" and not necessarily tied to HTTP. Hence, secure transportation layers like SSL are certainly an option, but not the only one, and not necessarily the best one.

Securing the physical packet is, however, the first out of three protection levels. The second level is authentication and authorization where you ensure who's making the call and why they are making the call. Finally, the third level consists of verifying that conditions necessary for the call to take place exist.

To secure the packet, authenticate the caller, and validate the call, you can employ a variety of techniques. WS-* specifications for security have been defined and implemented to standardize these techniques across platforms. Adhering to standards and employing related toolkits like WSE 2.0 means developing on best practices, using common solutions, and in the long run, possibly integrating your services with smarter software and hardware system tools and products like firewalls and network filters.

The focus on security for Web applications must be on the application layer and the quality of checks must be raised to touch the actual content being sent and not just the outer envelope. For Web services, this means looking into the actual contents of the XML/SOAP payload itself.

You can build an effective perimeter outside the realm of the service using a smart firewall like Forum XWall or similar products. The advantage you can get is double: on one end, you have the certainty that badly formed calls are stopped at the gate and never reach the service. On the other end, you can keep your service more lean and mean by limiting code-level security checks to a minimum. For all its advantages, code-level security does unfortunately come at the cost of performance.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the new book Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


Editor's Note: Vote Early and Often for MSDN Magazine

Editor's Note
Vote Early and Often for MSDN Magazine

As this issue of MSDN Magazine goes to press, Election Day is drawing near. Like many candidates for office, this magazine sports some bold cover lines and bright, appealing pictures. Unlike those same politicians, however, we will keep working for you month after month, year after year, even after you elect to purchase the magazine. Why should you vote for MSDN Magazine and security?

Our platform Each month, we bring you the finest articles relating to tools and technologies for the Windows operating system.

Foreign relations MSDN Magazine is leading the way in promoting international harmony, with native-language editions now available in Brazil, China, Russia, and brand new English language versions that specifically serve India, Australia, and Southeast Asia. Although we are based in the U.S., we believe it is important to reach developers globally.

If you live outside the United States and would like to get a local version of MSDN Magazine, you can go to the appropriate URL to find out more information.

Russia: http://www.microsoft.com/rus/msdn/magazine

Brazil: http://www.devmedia.com.br/msdn/

China: http://www.microsoft.com/china/msdn

Asia (English): http://www.charltonmedia.com/magazine.asp

MSDN Magazine is expansionist, yet pro-business. If you are located in one of the over 250 countries that is not currently served by a local edition, and you're interested in finding out why not, please drop us a line at mmeditor@microsoft.com.

Security Never before has our nation of developers faced such a stark choice as they do this fall. You can go back down the old road—the road of Blaster, of Sasser, of all those other worms and viruses that sound like Santa's reindeer—or you can choose a new path. A path that helps you reduce your code's attack surface. A path that lets you use ACLs from the .NET Framework. A path that uses cryptography and XML signatures and ISA Server 2004 to keep your code more secure.

But we're not just talking about security for short-term gain here. We've published over 70 articles on security for developers, and you can access them all on our Web site. Visit the MSDN Magazine virtual security center to experience our long-term commitment to making your code safer.

Color-coded alert system When you see a red cover, you know that we're running a special security issue. When we return to cover-level green, you'll know that it is safe to leave your terminal to grab some Red Bull and chips.

No desktop left behind If you've signed up for automatic updates, you should have received Windows XP SP2 recently. SP2 is, as service packs go, a pretty big deal—not just for end users but for developers as well. Changes in how security is handled will affect the way you'll need to code in the future, so you should be prepared now. We've included some late-breaking news on SP2 for the developer on page 58 of this issue.

The economy It costs less for a year's subscription to MSDN Magazine than it does for most programming books. That leaves you with more money to buy durable goods like refrigerators and boats. If you're an MSDN Universal subscriber, you are also eligible to receive the magazine at no charge or at a reduced cost. For more details, go to the MSDN subscriber download page.

Harmony When was the last time you saw a sea of protestors at a conference that we cosponsored? Think about it.

Voting for security won't change the course of the country, but it could very well determine the future path of your business. And even though the tens of millions of adult non-voters might not make a difference in November (although we do wish they would vote), it only takes one script kiddie with broadband to have a huge effect on your life if you're not careful.

We are the editorial staff members of MSDN Magazine, and we heartily approve this message.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Chris Brumme, Susan Chory, Jamie Cool, Robert Green, Cindy Gross, Martin Gudgin, Shawn Hernan, Chris Hollander, Michael Howard, Charlie Kaufman, Raja Krishnaswamy, Ronald Laeremans, John Lambert, Gerrard Lindsay, Steve Lipner, Arun Moorthy, Satyajit Nath, Matt Powell, Matt Thomlinson, Anson Tsao, Dan Vallejo, Maura Van Der Linden, and Ting-Hao Yang.

Active Directory, ActiveX, Microsoft, Microsoft Press, MSDN, MS-DOS, Outlook, OneNote, Visual Basic, Visual C++, Visual SourceSafe, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. InfoPath and Windows Server are trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Nancy Michell

Contents
Build Hot Swapping Into Your App
ASP.NET Charting Control
Architected-RAD Environment for .NET
Access Visual SourceSafe Files Over the Web
Create Patch Packages Easily
Dev Tool for .NET Environments
Automate Any PC Task
Create Custom Processes with New Productivity Tool
Ultra Software Protection

Build Hot Swapping Into Your App

Jungo Software Technologies has announced the release of Go-HotSwap 6.22, a complete off-the-shelf software package offering a solution for hardware vendors, system integrators, and operating system vendors, aiming to provide hot-swap capabilities to their users.

Using the Go-HotSwap architecture reduces time to market, enabling developers to focus on core technologies instead of spending their valuable time developing in-house hot-swap software.

Go-HotSwap supports Windows® 2000, Windows XP, Windows Server™ 2003, and Solaris 8 and 9, utilizing the native plug and play capabilities of these operating systems. The Go-HotSwap software infrastructure consists of a low-level operating system extension, the Hot Swap Engine, which has the following capabilities: identifying the insertion or removal of CompactPCI boards, and notifying system services, relevant drivers, and applications of a hot-swap event.

The Go-HotSwap license includes Jungo's WinDriver driver development tool as an integrated component.

http://www.jungo.com

ASP.NET Charting Control

Telerik is offering a new member of the r.a.d.controls suite called r.a.d. chart. The new component is a business-oriented charting control designed to simplify the display of graphical data in ASP.NET. The product offers four basic chart types—Pie, Bar, Line, and Area, each of which can be substantially customized. Packed with a Quick-Start framework, numerous running examples, and templates, r.a.d.chart can help you represent data in a way that is very easy to understand.

r.a.d.chart supports all the various browsers capable of displaying PNG files. In order to blend the component into your application, you can customize the appearance and behavior of all the chart elements including gridlines, axes, labels, background, borders, titles, and legends.

http://www.Telerik.com

Architected-RAD Environment for .NET

Modularis Inc. has announced the release of Modularis Accelerator 6, an Architected-Rapid Application Development (Architected-RAD) Environment for the Microsoft® .NET Framework. By combining a service-oriented architecture (SOA) with software automation technology, Modularis helps organizations address the fundamental shift toward service-oriented enterprise application development and helps to create .NET-compliant software faster.

Modularis Accelerator 6 consists of a variety of components, including Architect, a sophisticated Architected-RAD tool designed specifically for use by experienced software architects to automate the creation of tedious and repetitive infrastructure code.

Other components include powerful productivity tools designed to aid developers in customizing and maintaining applications generated with Architect. Many of these tools are integrated into the Visual Studio® .NET IDE.

Automated code templating that adds customizable header blocks, structured exception handling, and performance profiling to new or existing code, automated code documentation and sophisticated application performance analysis, and bottleneck detection are just some of the various tasks the package performs.

Accelerator also provides a number of administration and monitoring tools that are designed to help developers or system administrators configure, manage, and monitor the performance of enterprise applications.

Finally, Accelerator includes the Accelerator Libraries—a set of proven, tested, and freely distributable .NET assemblies that provide a number of services critical to all enterprise applications. They build upon the .NET Framework to offer a number of high-level features including an application layer, a role-based security model, advanced object state management services, a centralized, asynchronous audit trail, and lots more.

http://www.accelerating.net

Access Visual SourceSafe Files Over the Web

Acorden Software has just released SourceXT 2.0, a technology for accessing Visual SourceSafe® files across various kinds of boundaries. Based on advanced features of the .NET Framework, SourceXT provides a fast and reliable connection over the Internet to any Visual SourceSafe database.

SourceXT 2.0 features a Windows-based client with the familiar look of the Visual SourceSafe explorer, full integration with Visual Studio .NET, data compression, and support for all major Visual SourceSafe functions.

SourceXT performs very well across a Virtual Private Network, is easy to set up and administer, and works over the HTTP/HTTPS protocol with virtually no firewall issues.

Some of the key SourceSafe operations supported are get, check out, check in, undo check out, create/delete project, recover deleted project or file, search for checked out files, and lots more.

http://www.acorden.com

Create Patch Packages Easily

AgenSoft has released PatchFactory 2.1 for Windows, a program that can help developers create instant patch packages for software and file updating.

Enhanced with a variety of features such as an easy-to-use interface with Wizard mode, a powerful patch engine, integrated compression technology, and adjustable multilingual interface, PatchFactory is an efficient software patching and updating tool for both individual developers and software companies.

With Patch Factory 2.1, even a beginner can build an update package in minutes with the help of the step-by-step Patch Building Wizard. It is an easy and effective way to make patches to meet the frequent demand of software updating.

Generated patch packages are small, self-extracting executable update programs with an adjustable user-friendly interface (available in English, German, Italian, French, Polish, Spanish, Swedish, Hungarian, and Czech). Patches can modify or replace installed software files to add new features, fix bugs, or add support for new hardware.

http://www.agensoft.com

Dev Tool for .NET Environments

Sybase Inc. has just announced the availability of Sybase DataWindow .NET, a powerful new development tool for any .NET environment that both simplifies and accelerates the creation of data-driven enterprise applications—including rich-client and smart-client applications.

Based on proven technology that has been awarded six patents, Sybase DataWindow .NET gives developers a single tool to incorporate data access, complex logic, and data presentation into .NET-based applications both quickly and easily.

A user-friendly point-and-click environment enables easy data retrieval and modification, allowing developers to quickly incorporate complex business logic into applications and to modify it when their business needs change.

Sybase DataWindow .NET is engineered for developers using Visual Studio .NET, Borland C#Builder, or any other integrated development environment that supports the .NET Framework.

http://www.sybase.com/datawindow.net

Automate Any PC Task

NeoWise Software has released RoboTask 2.1, a Windows-based application that makes it easy to automate any series of tasks that you can perform on your PC.

Unlike simple programs that capture and play back keystrokes and mouse movements, RoboTask lets you select actions to automate. Without writing any code, you can test Windows conditions, create conditional IF/ELSE statements, perform loops, create custom variables, and quickly automate repetitive tasks. RoboTask's jobs can be launched manually, scheduled to run automatically, or can be designed to be triggered by a wide variety of changes in your computer.

With RoboTask, it's easy to ping your Web host every few minutes to determine if your site is available, check for e-mail every few hours, or perform incremental file backups on a regular basis.

In just minutes, you can set up simple tasks such as launching applications, opening documents, or seeing if the files in a particular folder have changed. You can send letters and files by e-mail, or schedule and run complex FTP operations.

RoboTask can replace a stack of Windows utilities with its own simple tasks. You don't need to buy an alarm clock utility, low-battery checker, Windows auto-shutdown application, or file backup program. RoboTask is capable of easily performing all of these functions.

WIth RoboTask you'll have access to a growing library of free add-ons. You can download a CD/DVD writer, as well as a BASIC language interpreter that makes it easy for programmers to add virtually unlimited functionality to their RoboTask jobs.

RoboTask 2.1 runs under versions of Windows from Windows 98 up to Windows Server 2003.

http://www.neowise.com

Create Custom Processes with New Productivity Tool

SharpDeveloper has released Sharp Code Foundry, a developer productivity tool that integrates with the Visual Studio .NET 2003 IDE as an add-in. The tool can be used to generate SQL Server™ stored procedures (for SQL Server 7.0 or SQL Server 2000) or Visual Basic .NET and C# data services based on user-defined templates. With templates, the user can create custom processes and tailor them to meet various coding standards.

The Sharp Code Foundry inserts code directly into the currently active document within the Visual Studio .NET 2003 IDE, enabling it to integrate within the user's workflow rather than forcing workflow on the user.

http://www.sharpdeveloper.ca

Ultra Software Protection

Risco Software has released ACProtect 1.35A, an application that allows developers on all Windows systems to protect software with special anti-cracking techniques.

ACProtect can embed software protection codes inside your program. Important code of protected programs can be decrypted before running and encrypted again after it has finished running. This code can even be encrypted such that only registered users can unlock it before it begins executing.

ACProtect can also generate ShareIt.com-compatible key-generation programs. The Key-generation program can generate registration codes, and ShareIt.com will send it to your registered customer.

http://www.ultraprotect.com

Send your New Stuff to newstuff@microsoft.com.

Nancy Michell is a developmental editor at MSDN Magazine where she spends her time manipulating bits of string internals for the good of mankind.


Web Q&amp;A: ADO.NET Joins, HTML to XHTML, ASP.NET ViewState, and More

Web Q&A
ADO.NET Joins, HTML to XHTML, ASP.NET ViewState, and More
Edited by Nancy Michell

Q Back in ADO, Remote Data Service (RDS) allowed a developer to write a query to join tables, make changes to the data which was returned, and then send the changes back to the database with a simple call. In other words, it figured out which columns came from which table and made the changes to the right table. Does ADO.NET only support "smart" updates for queries of single tables, not queries involving joins?

A Yes, that's the case. DataAdapter can send changes from a DataSet/DataTable to a back-end source if the DataTable is associated with a single base table. In all other cases, the user has to write custom Update, Delete, and Insert commands.

Keep in mind that the ADO cursor engine's updating logic for join queries was extremely simplistic. You had little control over the behavior, which was a problem for inserts and deletes against a joined recordset. In the ADO.NET world, breaking up the data into separate queries, storing the results in separate DataTables, and relating the data through a DataRelation gives you much more control over your updates as well as better performance.

Q How can I use a contentEditable DIV to capture user input on a Web page and then do a client-side XSLT transformation to convert the DIV's innerHTML into a Microsoft® Word XML document? The problem is that a contentEditable DIV exposes deprecated HTML (which is not valid XML) via its innerHTML property. So XSLT transformations are not possible without first converting the HTML. My preference would be to do this using client-side script, but I'd consider using server-side C# code, too.

A TidyCOM is a third-party library that you can call from client-side script or put on a server to use in your ASP.NET page.

Also, check out the following document for information about converting HTML to XHTML: Converting HTML documents to XHTML. InfoPath has a custom implementation that even cleans the HTML generated by Microsoft Office into something much cleaner.

Another option is to use Chris Lovett's SgmlReader 1.4 or the InfoPath™ SDK's html2XHTML COM DLL. (For more information on its usage, see Using the HTML to XHTML Conversion Tool.)

Q I have implemented a Web site that makes heavy use of ASP.NET ViewState to which my developers have added custom hidden variables. Now I'd like to move the view state from a hidden variable in the Web page to either a file or database on the server by using the SaveStateToPersistenceMedium and LoadPageStateFromPersistenceMedium methods. Are there any known performance issues with these methods?

A Remember that view state operates on a per-page basis; Session is per user and Cache is per application. So, if a user opens a new window on a form from the file menu (File | New Window), he has two view states (one in each browser window). Each view state will behave independently, but there is only one session, which can lead to some problems. For instance, if you store view state on the server, you still need to have a reference to it on the client in a hidden field so that you can get back to the state of the controls when the form is posted back to the server. In this respect it would be similar to storing a session cookie on the client.

The fact is that with view state on the client there are as many view states as there are browser windows open. Assuming you use the session object to hold on to the view state for a page on the server, there is an implicit assumption that the user has only one copy of the page at a time in a session. Thus, with view state on the server, you get only one copy, which can potentially lead to problems. Since the view state may not correspond to what the user was seeing on their screen, all of the actions that depend on view state will start to fail. Change events may not fire correctly, the controls may be drawn differently, and so on.

The net result of this is that to do what you want, you'll still have to store a unique identifier in the page rendered to the client so that you can later associate the view state stored on the server with the same page on post back.

Q On a Web page served over HTTPS, the referenced style sheets and JavaScript files don't work. What am I doing wrong?

A The Knowledge Base article 825057 ("Internet Explorer Incorrectly Displays Web Pages") addresses the problem when the following conditions are true: the Advanced setting "Do not save encrypted pages to disk" is selected, the Web page uses the HTTPS protocol, and you have turned on HTTP compression for cascading style sheet (CSS) and JavaScript files on the Web server.

Generally, you should be able to hit CSS files and JavaScript files from HTTPS requests using either HTTP or HTTPS (as appropriate to your particular situation). You should turn off the compression to alleviate the problem.

Of course, if you have frames or popups that are trying to interact, then you probably have cross-site scripting errors as a result of the page having the document.domain of beta.dev.ms.com and the JavaScript files having a document.domain of dev.ms.com. The following article tells you how to get them to communicate by setting their domain properties to the same value: domain Property .

Q I have a main page with three sections: header, left menu, and content section. I'm developing a portal that will show content from multiple third-party sites. That content appears along with its own menus and headers. How can I pull in the content from those sites (ASPX pages) and use them within my framework without using frames or iframes?

A You should have the third-party sites do one of two things: either provide the content in a manner that is easy for you to consume, such as through a Web service or even RSS Syndication, or ensure that the layout and structure of the pages won't change so you can use a regular expressions to screen scrape and parse out the desired content. With either a Web service or an RSS feed, the third-party can even use these distribution mechanisms internally on their own sites, further encouragement for them to provide the content in a nice easy format for everyone to consume.

Q I have a lot of mission-critical applications written in ASP and Visual Basic® 6.0 and I'd like to have those apps consume ASP.NET Web services. How can these legacy applications interface with ASP.NET Web services?

A Have a look at the following documents for help: "Integrating Web Services and COM Components", "SOAP-Toolkit 3.0" (deprecated by the .NET Framework), and "PocketSOAP" (OpenSource—MPL).

The SOAP Toolkit makes consuming Web services very easy even without relying on the .NET Framework. However, all you really need is the ability to build an XML string that conforms to the SOAP spec and send it over HTTP to the client. You can even do this using client-side script using the MSXML parser which has been built into Microsoft Internet Explorer for several versions. See Knowledge Base article 555057 ("How to Call a .NET Web Service from Client Side JavaScript Using Internet Explorer and MSXML"). If you need to support additional browsers, you can even cover Netscape 6.0 and up using its native support (though you need to do a couple of overrides of the supplied behavior to make your code look less ugly). If you can build a string and send it over HTTP, then you can consume a Web service. The easiest solution will depend on where you need to consume them. You could arguably just create an assembly to encapsulate the Web service call and expose it via COM interop. It all depends on your priorities and constraints.

Another available option is to use the Web service behavior that is explained at WebService Behavior . The page states that the WebService behavior enables methods that are implemented on Web services to be called from client-side script in a Web page using Internet Explorer 5.0 and later. In order to attach a WebService behavior to a page, you must first download the WebService HTC file and put a copy in the same directory as the pages that use the behavior.

Q I'm developing an ASP.NET Web tool for my team. I am using Server.Transfer to handle page reposts (using a new query). However, I get a view state corruption error if I leave the page idle for more than 20 minutes and try to resubmit. As I understand, view state should be kept on the client side and shouldn't have a timeout, right? I used Response.Redirect too, but the problem doesn't go away. I have set the default of "shutdown worker processes after being idle for 20 minutes" on IIS.

A Do you happen to have an idle timeout set on your worker process? You are most likely autogenerating your validation key, which means that when you create a ViewStateMac, append it to the end of your view state, and send it to your client, it will only be valid for that instance of the AppDomain. Once the AppDomain restarts, a new validation key is generated and your POST will result in ASP.NET believing the view state is invalid since the ViewStateMACs do not hash the same. To mitigate the problem, remove the idle shutdown and hardcode your validation key.

Q How can I disable the Back button so my Web visitors don't get the page-expired warning?

A Unless you are hosting the WebBrowser control, you can't effectively stop the use of the Back button. See Knowledge Base article 811603 ("Use the Forward Button and the Back Button for WebBrowser Control in Visual Basic .NET") if you need more info on using the control.

You can implement an Internet Explorer restriction if you have total control of the client base and if Internet Explorer is your only browser, of course. See Knowledge Base article 823057 ("The Restrictions that are Available to Internet Explorer 6.0 SP1") for more information. If you launch your page in a new window each time, the Back button on that page will be unavailable. You can also script the new window to have no toolbars.

Q What are the best practices for designing a browser-based interface for use with a number of commercial business solutions exposed by Web services, all of which share a common architecture and processing model? Do I create specific interfaces for each solution or just one? How do I address the need to authenticate against the same Web site using Windows® integrated security and forms-based authentication for the public domain?

A The single request method approach is not a great idea, although there are plenty of instances where it has been successfully used. At a minimum it would be better to have one or more methods per logical grouping (customer info, account info, and so forth).

In building sites with the same requirement, one of two scenarios is usually the case. The easy one is where there is a separate IIS-based intranet for internal users, and a DMZ server running IIS for external users. There you have separate servers and separate authentication models, so it's straightforward.

It sounds like you're involved in the more complex case in which you have the same server for both groups of users. A successful approach in this scenario is to set up the site to require domain credentials, but then change the 401 error handling in IIS to point to your page of choice (since the 401 is one of the few errors that you apparently can't trap programmatically), and have that page redirect to the forms login page (since the page at which you point the 401 error must be HTML, not ASP).

This allows the internal users to access the site with their domain credentials and not require a login, while the external users are shown a login page, all from the same site. The Active Directory® Application Mode (ADAM) can be used for the public domain authentication store, which will work quite well .

If you are writing an ASP.NET Web app (not a Web service), then creating a solution that combines forms-based authentication and Windows integrated authentication isn't that hard. To do so, you'll need to write an HttpModule that uses the supplied Forms and Windows modules and then set it all up in the config file.

When you are finished, Thread.CurrentPrincipal will be either a GenericPrincipal or a WindowsPrincipal (depending on how the user was authenticated), and the current identity's AuthenticationType lets you know how the user was authenticated.

Q I have an ASP.NET Web site and I'm getting script errors when I have cache breakers in my ASPX files. Sample cache-breaker code looks like the following:

<script language="javascript" type="text/javascript"
    src="blah.js?<%=System.DateTime.Now %>">
</script>
I run into this script error when the ASPX page is requested repeatedly on the same browser session. How can I include such cache-breaking code in order to avoid proxies caching my pages over an extended period of time?

A If you have an idea of how often this file will change, or has changed over the last month, you could just tell IIS to set the expiration header on the file for a predetermined amount of time. If the file does not change on a regular schedule this may not work.

If you want to prevent proxies from caching this file for an extended period of time, you can configure IIS to set the expiration date for a relative time in the future (such as three days from the time being requested). Proxies (and clients) should not rerequest the file until that three-day period has elapsed (or the user explicitly requests a fresh copy).

When using just the date of the file, some proxies will maintain a history of how many times they requested the file and how many times it has changed during that period. They will then apply that knowledge and only request the file when their best guess has told them that it may have new content.

Other clients will request the file with a "If-Modified-Since" header, but they will send a request every time. The server will either send the file or send a "304 Not Modified" response, telling the client to used their cached copy. You could also change the JavaScript file to an ASPX file and return a "304 Not Modified" from the ASPX code when you don't have any changes to send to the client. Before you proceed, however, you should note that manually setting the Cache-Control header is a bit harder than letting IIS do it, which is what it does when you manipulate the "Enable Content Expiration" setting on the dialog shown in Figure 1.

Figure 1 Manipulating the Expiration Setting
Figure 1 Manipulating the Expiration Setting

Incidentally, this is the same dialog you'd use to manually add the Cache-Control header. For more information on this, see Caching ASP.NET Pages.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Jay Allen, Kawarjit Bedi, Andrew Blair-Stanek, Harold Campos (Intl Vendor), Todd Carter, Ulrich Dallmann, Doug DeFonzo, Nathan Dolly, Russ Helfand, Ken Henderson, CJ Huang, Mark Ingalls, Chris Jackson, David Koronthaly, Vijay Kurup (Infosys Technologies Ltd), Karlen Lie, Brian MacKay, John Mann, Alan Melia, David Sceppa, Willy-Peter Schaub, Aaron Sethi, Norman Sequeira, Vittal Setty (Infosys Technologies Ltd), Ranjiv Sharma, Ken Stanfield, Devdatta Waghdhare, Dennis Wang, Phil Winstanley (Business Guest), Victus Wong, and Tsan Zheng.


Data Points: Updating Data in Linked Servers, Information Schema Views, and More

Data Points
Updating Data in Linked Servers, Information Schema Views, and More
John Papa

Code download available at: DataPoints0411.exe (115KB)

Every day a developer somewhere needs to write code to iterate through SQL Server™ system objects, query and update tables in linked servers, handle optimistic concurrency, and retrieve column and stored procedure metadata. In this month's column, I will address these and other T-SQL development scenarios based on some of the questions I most frequently receive from readers.

Q The app I am working on requires that any time a value is modified in our sales table, a copy of the old row is inserted into a history table. I created an AFTER UPDATE trigger to handle this, but how do I get it to work if the history table is on another machine?

A A trigger is a great tool for tracking changes made to rows in a table, but it needs a little help to communicate with another instance of SQL Server. When a trigger tries to modify a history table on another machine running SQL Server, it is necessary to link the originating server to the remote server using the system stored procedure sp_addlinkedserver. SQL Server handles security at the server level, which means that the bounds of the security context of the transaction in the originating server ends at that server. Thus, when the trigger in the originating server tries to insert a record into a table in the remote server, the security context must either exist on the remote server or alternative rules must be established ahead of time to handle how the transaction should log into the remote (linked) server.

A linked server is an agreement between an originating SQL Server and a remote SQL Server made so that the originating SQL Server can talk to the remote one. This agreement can be set up using the SQL Enterprise Manager interface or by using the sp_addlinkedserver stored proc. Once a linked server has been set up, it can be removed through Enterprise Manager or by calling the sp_dropserver stored proc. For example, to add a linked server called SQLRemote to an originating SQL Server instance called SQLOrigin, this statement could be executed from a connection to SQLOrigin:

sp_addlinkedserver 'SQLRemote'

Once the link has been established, it is important to make sure that the local SQL Server has proper security permissions to communicate with the remote instance. This can be done using the Enterprise Manager interface or through the sp_addlinkedsrvlogin stored procedure. This stored procedure can be used to map the SQL login from the originating server to a SQL login on the remote server:

sp_addlinkedsrvlogin 'SQLRemote', 'false', NULL, 'SQLTestUser',
    'MyPassword'

This code snippet maps all users from the originating server to the remote server SQLRemote's SQL Server login called SQLTestUser. The first parameter indicates the remote server's name. Setting the second parameter to false indicates that the remote server login parameters should be used for the specified local login. The third parameter is used to indicate what local login to map from. If you want all local logins to map to a single remote login, then you should pass a NULL value. This means that no matter who logs in on the local server, they will be mapped to the remote login specified with the remote user and remote password parameters. The last two parameters specify the remote SQL login and password that should be used.

To map a specific local login to the remote login called SQLTestUser, this statement could be modified by changing the third parameter from NULL to a specific login name such as JoeSomebody. You can also map the current login to a specific remote login by setting the second parameter to true. This tells SQL Server that authenticated logins should use their credentials to communicate with the remote SQL Server. Many organizations either create specific logins for server mappings or use the authenticated login to map to the remote server. As you can see, there are several possible solutions so it is important to create a mapping that follows the security patterns of your organization.

The interface in SQL Enterprise Manager is pretty straightforward and can be used in place of executing sp_addlinkedsrvlogin. Figure 1 shows that if the local SQL Server login is Test2, then it will map to the remote SQL Server instance's SQLTestUser2 login. Otherwise, any other local login will map to the remote login called SQLTestUser. One word of caution: if you create a mapping using the sp_addlinkedsrvlogin stored procedure and then go to SQL Enterprise Manager to look at the mapping, be sure to refresh the linked server node. SQL Enterprise Manager caches its settings, so it is essential to refresh the node to be sure you are seeing the most accurate and up-to-date information.

Figure 1 Mapping Security for a Linked Server
Figure 1 Mapping Security for a Linked Server

Once the mapping is established, the problem can be solved as a trigger could be created to insert a record into a remote server's table. All you have to do to specify a remote server is prefix the fully qualified table name with the SQL Server instance name (sqlserverinstance.databasename.owner.tablename). The following trigger will insert all of the inserted or updated rows from the Customers table in the local server's Northwind database into the remote server's Northwind.dbo.Customers table:

CREATE TRIGGER trCustomers_UpdateToOtherServer ON Customers AFTER UPDATE
AS
    INSERT    SQLRemote.Northwind.dbo.Customers
    SELECT    CustomerID, CompanyName, ContactName, ContactTitle,
              Address, City, Region, PostalCode, Country, Phone, Fax
              FROM inserted
GO
It will not track deleted rows.

It should be noted that when using a trigger to do a linked server query, if the remote server is down nothing that fires that trigger can work on the local server. This introduces an additional point of failure into the system. You may want to consider modifying this solution slightly to use a local staging table (populated by the trigger) and a job that periodically moves the data from the local staging table to the remote server.

Q I want to implement optimistic concurrency in my database queries by including a versioning column in each table that will signify when a row was last modified. Can you tell me about the various advantages and disadvantages to using a column of type TIMESTAMP or ROWVERSION versus a type DATETIME as the versioning column?

A Both of these approaches involve adding a column to a table. If the tracking column I add is called LastUpdate, then I must first add the column to the table using the ALTER TABLE data definition language (DDL) statement with one of the following two statements shown here:

-- Add a DATETIME tracking column
ALTER TABLE Customers ADD LastUpdate DATETIME

-- Add a TIMESTAMP tracking column
ALTER TABLE Customers ADD LastUpdate ROWVERSION

If the LastUpdate column is created as a ROWVERSION column (ROWVERSION is a synonym for the TIMESTAMP data type), then no additional code is required to keep its value updated. Columns defined as TIMESTAMP or ROWVERSION are built to automatically update their values every time the row they are associated with is updated.

In addition, the initial values of ROWVERSION columns are also set automatically. However, if the LastUpdate column is created as a DATETIME column, then some additional code is required to initialize its values, otherwise it will be NULL. It is important that the tracking column not be NULL so that its non-NULL value can be retrieved and then compared in an UPDATE statement's WHERE clause. The following code could be executed immediately after the DATETIME column is created to initialize the tracking column's value for all rows:

UPDATE Customers SET LastUpdate = GETDATE()

The main advantage to using a column of type DATETIME to track the version of a row is that the value is easily readable because it is a date and time value. If you want to display this value via an application, it makes much more sense to display a date and time to a user than a binary value such as is stored in a ROWVERSION column. However, creating a versioning column as a DATETIME data type not only requires additional code to initialize its values, but it also requires additional code to update its value whenever the row is modified. The value does not get updated automatically, so a trigger could be written to keep it updated. For example, a trigger called trCustomerTest could be written to update its value to the current date and time whenever a row is updated or inserted into the Customers table using the built-in SQL Server GETDATE function, as shown in the following code:

CREATE TRIGGER trCustomerTest ON Customers AFTER UPDATE, INSERT
AS
    UPDATE    Customers SET LastUpdated = GETDATE()
    FROM      inserted i INNER JOIN Customers c ON i.CustomerID =
              c.CustomerID
GO

This technique requires more code and thus more maintenance. If the trigger were to be dropped, then any application expecting the value of the LastUpdate column to be a versioning column would subsequently be in trouble. The LastUpdate column would not be updated and all UPDATE statements would simply overwrite the last person's changes in a situation where two users grabbed and updated the same row. However, if this trigger is maintained well, then this technique allows an easily readable and usable value to display the date and time the row was modified.

Since the SQL Server ROWVERSION data type automatically updates its column with a new value every time a value in its row is modified, using it to help implement optimistic concurrency is very simple. It requires no additional code since it automatically initializes its value and regenerates it each time its row is modified. Furthermore, the DATETIME data type is only accurate to approximately 3.33 milliseconds, so multiple updates in fast succession could receive the same value. Thus, I prefer the ROWVERSION (TIMESTAMP) column since it doesn't involve any additional code and has fewer potential points of failure.

Q I am attempting to write a query that will retrieve a list of columns (including their data types) for a given table. How do I query the SQL Server system tables for such column info?

A Both the system tables and the information schema views can be queried to retrieve schema information directly through T-SQL. Additionally, if you are looking for specific information you can retrieve system information through some of the system stored procedures. While you can query the system tables (such as sysobjects) directly, Microsoft discourages this, offering the more flexible and documented information schema views. Using the information schema views offers a layer of protection in case system tables are changed in future releases of SQL Server.

The information schema views are contained in each database within the INFORMATION_SCHEMA schema. The INFORMATION_SCHEMA.COLUMNS view exposes the columns from all objects (such as tables and views) within the current database. One of the main advantages of using the information schema views is that the column names are easily readable and mostly self-documenting, which is not the case when you query against the system tables directly.

Retrieving information from the information schema views is often easier than querying the system tables. Sometimes a single view can be queried versus having to join multiple system tables together linking columns with very nebulous names. The following SQL statement will retrieve a list of all columns and their data types from the [Order Details] table in the Northwind database using an information schema view:

-- Get all Columns
SELECT   c.TABLE_NAME, c.COLUMN_NAME, c.DATA_TYPE
FROM     INFORMATION_SCHEMA.COLUMNS c
WHERE    c.TABLE_NAME = 'Order Details'
ORDER BY c.ORDINAL_POSITION

Not only are the columns easy to identify in these views, but they can be sorted in their ordinal position and the data types can be retrieved without joining to another view or table. The same query could be written against the system tables using the code that is shown in the following sample:

SELECT     so.name AS TableName, sc.name AS ColumnName,
           st.name AS DataType
FROM       sysobjects so
    INNER JOIN  syscolumns sc ON so.id = sc.id
    INNER JOIN  systypes st on st.xusertype = sc.xusertype
WHERE      so.name = 'Order Details'
ORDER BY   colid
This code queries the sysobjects table which contains all system objects in the current database such as tables, views, and triggers. It then joins this table to the syscolumns table, which contains a list of all columns in the database. These tables are joined on the id field, which in the context of these two tables represents the link between the object and the column. Then these tables are joined to the systypes table using the xusertype column from the syscolumns table. This can take a little digging as the columns that the tables need to be joined on are not always obvious. Thus, it is easy to think that the systypes table might need to be joined on the column syscolumns.type or even the syscolumns.xtype. Once you experiment with both the system tables and the information schema views, it is easy to see that the views are often the quicker to develop and more maintainable (or self-documenting) solution.

The information schema views can also help identify the primary key fields in a given table. To achieve this, you can join the INFORMATION_SCHEMA.COLUMNS view to the INFORMATION_SCHEMA.KEY_COLUMN_USAGE view and then on to the INFORMATION_SCHEMA.TABLE_CONSTRAINTS view. The KEY_COLUMN_USAGE view exposes the list of columns for each table that contains keys. For example, the Order Details table has four rows in this view, two for the primary key fields of OrderID and ProductID, and two more for each foreign key field of OrderID and ProductID. This makes this view ideal to be used as a many-to-many table between the COLUMNS view and the TABLE_CONSTRAINTS view in order to show the primary key columns (or foreign key columns) for a table. The TABLE_CONSTRAINTS view shows information about the constraints, such as the type of constraint. When joined together, these three views can display a list of primary key fields for a given table using the query shown in Figure 2. One alternate way to retrieve the list of keys contained within a table is to use the system stored procedure sp_MStablekeys shown in the following line of code:

sp_MStablekeys 'Order Details'
However, I prefer to use the information schema views over this stored procedure to retrieve column information since I can more easily specify which fields to return without running through the myriad of options that the system stored procedure offers.

Figure 2 Querying Primary Keys (Close Figure 2)
-- Get all PK columns
SELECT      c.TABLE_NAME, c.COLUMN_NAME, c.DATA_TYPE
FROM        INFORMATION_SCHEMA.COLUMNS c
    INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE kcu
            ON c.TABLE_NAME = kcu.TABLE_NAME AND c.COLUMN_NAME =
               kcu.COLUMN_NAME
    INNER JOIN INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc
            ON tc.TABLE_NAME = kcu.TABLE_NAME AND tc.CONSTRAINT_NAME =
               kcu.CONSTRAINT_NAME AND tc.CONSTRAINT_TYPE = 'PRIMARY KEY'
WHERE       c.TABLE_NAME = 'Order Details'
ORDER BY    c.ORDINAL_POSITION

There are several other information schema views that can be used to access the metadata for a database schema, as well. Some of my favorites are shown in Figure 3.

Figure 3 Schema Views (Close Figure 3)

Information Schema View | Description
INFORMATION_SCHEMA.TABLES | All tables in the database
INFORMATION_SCHEMA.TABLE_PRIVILEGES | Users and the permissions to each table
INFORMATION_SCHEMA.COLUMNS | All columns in the database
INFORMATION_SCHEMA.KEY_COLUMN_USAGE | Key columns, their tables, and their constraint name
INFORMATION_SCHEMA.TABLE_CONSTRAINTS | Information about constraints in the tables
INFORMATION_SCHEMA.ROUTINES | Functions and stored procedures
INFORMATION_SCHEMA.PARAMETERS | Parameters associated with functions and stored procedures

The following example returns the list of stored procedures and functions along with information about any of their parameters. The PARAMETER_MODE indicates the direction of the parameter, such as IN or OUT, while the ORDINAL_POSITION field represents the order in which the parameters exist in the object:

SELECT     r.SPECIFIC_NAME, p.PARAMETER_NAME,
           p.PARAMETER_MODE, p.DATA_TYPE,
           p.CHARACTER_MAXIMUM_LENGTH
FROM       INFORMATION_SCHEMA.ROUTINES r
    INNER JOIN INFORMATION_SCHEMA.PARAMETERS p ON
               r.SPECIFIC_NAME = p.SPECIFIC_NAME
WHERE      r.SPECIFIC_NAME NOT LIKE 'dt%'
ORDER BY   r.SPECIFIC_NAME, p.ORDINAL_POSITION

Information schema views are very powerful tools that allow you to expose the information contained within the system tables without the complexity. The views also provide a layer of abstraction in case future releases of SQL Server change the schema of any of the system tables. If you've been developing with SQL Server for a long time and are comfortable with the system tables but have not yet explored the information schema views, you may be pleasantly surprised at how easy they are to use. Further, there are several additional information schema views available through SQL Server which expose information about domain constraints, referential constraints, and other valuable information.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Test Run: API Test Automation in .NET

Test Run
API Test Automation in .NET
James McCaffrey

Code download available at: TestRun0411.exe (124KB)
Contents
The System Under Test
The API Test Automation
Discussion

The most fundamental type of software test automation is automated API testing. API testing essentially entails testing the individual methods that make up a software system rather than testing the overall system itself. API testing is also called unit testing, module testing, component testing, and element testing. Technically the terms are quite different, but in casual usage you can think of them as having roughly the same meaning. The idea is that you must make sure the individual building blocks of your system work correctly because if they don't the system as a whole cannot be correct. API testing is absolutely essential for any software system.

In this month's column I will show you key techniques and principles of automated API testing in a .NET environment. The best way to illustrate where I'm headed is with two screen shots. Figure 1 shows a dummy Windows®-based application program that is the overall system which uses the library I'm testing. In testing we often call the program or API being tested the system under test (SUT) to distinguish it from the test harness system.

Figure 1 StatCalc App
Figure 1 StatCalc App

The StatCalc application calculates the mean of a set of integers. Behind the scenes, the StatCalc application references my MathLib.dll library which houses a class that contains the ArithmeticMean, GeometricMean, and HarmonicMean methods. You might recall from an introductory statistics class that the arithmetic mean is a normal average, the geometric mean is an average of ratio values, and the harmonic mean is an average of rate values. For this example, I want to test only these three methods from the library, not the whole StatCalc application which includes additional code for the user interface.

In order to manually API test a single test case in this scenario, I would be required to create a small tester program, copy and paste the source code of one of the methods I'm testing from the source code for MathLib.dll into my tester program, hardcode some input arguments to the method, run the program, visually examine the return output, somehow determine if the result was correct or not, and record this information somewhere, for example in an Excel spreadsheet. I would need to do this several thousand times to even begin to feel confident that I understand the behavior of the methods I'm testing. And worst of all, every time the code in the system under test or its underlying DLL changed, I'd have to start all over. A much better testing strategy is to use the powerful capabilities of the .NET environment to write automated API tests. Figure 2 shows a sample run of a program that does just that.

Figure 2 Test Program Output
Figure 2 Test Program Output

In this example, the test program is a console application that reads test case data (test case ID, method to test, input arguments, and expected result), calls the specified method, and compares the actual result with an expected result to determine whether the test case passed or failed. In the sections that follow I will briefly examine the system under test so you'll know what I'm testing and how I'm testing it. I'll present and explain in detail the test program that generated the output shown in Figure 2, and I'll describe how you can extend the techniques presented here to meet your own needs.

The System Under Test

One of the most fundamental principles of software testing is that you must understand the system under test in order to test it. And the best way to understand an SUT is to examine its source code. Figure 3 lists the core of the code for the dummy system under test that calls the methods to verify.

Figure 3 Code for the System Under Test (Close Figure 3)
Private Sub Button1_Click( _
    ByVal sender As System.Object, ByVal e As System.EventArgs) _
    Handles Button1.Click
  Dim sVals() As String
  sVals = TextBox1.Text.Trim.Split(" ")
  Dim iVals(sVals.Length - 1) As Integer
  For I As Integer = 0 To sVals.Length - 1
    iVals(I) = Integer.Parse(sVals(I))
  Next

  If RadioButton1.Checked Then
    Dim M As New MathLib.Methods
    TextBox2.Text = M.ArithmeticMean(iVals).ToString("F4")
  ElseIf RadioButton2.Checked Then
    Dim M As New MathLib.Methods
    TextBox2.Text = M.GeometricMean(iVals).ToString("F4")
  ElseIf RadioButton3.Checked Then
    MessageBox.Show("Not yet implemented")
  End If
End Sub

Let me emphasize that I am definitely not using good coding techniques here so that I can keep the example as short and clean as possible (for example, I've made no attempt to handle poor input). The application has a Project Reference to file MathLib.dll, which you'll see in a moment. Notice that the application is not ready for the HarmonicMean method yet. Software testing takes place in a highly dynamic environment and, consequently, you have to expect incomplete implementations will be prevalent throughout the development and testing process.

Next, let's take a look at the source code in the MathLib.dll file that implements the key methods used by the application. Figure 4 lists this code. Again, let me emphasize that this source code is presented for illustrative purposes only.

Figure 4 MathLib.dll (Close Figure 4)
Imports System

Public Class Methods

  Public Function ArithmeticMean( _
      ByVal ParamArray Vals() As Integer) As Double
    Dim Sum As Integer = 0
    For Each V As Integer In Vals
      Sum = Sum + V
    Next
    Return Sum / Vals.Length
  End Function ' ArithmeticMean()

  Private Function NthRoot( _
      ByVal X As Double, ByVal N As Integer) As Double
    Return Math.Exp(Math.Log(X) / CType(N, Double))
  End Function

  Public Function GeometricMean( _
      ByVal ParamArray Vals() As Integer) As Double
    Dim Product As Double = 1.0
    For Each V As Integer In Vals
      Product = Product * V
    Next
    Return NthRoot(Product, Vals.Length)
  End Function ' GeometricMean()

  Public Function HarmonicMean( _
      ByVal ParamArray Vals() As Integer) As Double
    ' not implemented yet
    Return 0.0
  End Function ' HarmonicMean()

End Class

In order to write test automation you must understand how to call each method being tested. And as you'll see in a moment, additional information is needed in order to create test case expected values. Here, each of the three methods accepts a variable number of int parameters and returns a single double value. Observe that both ArithmeticMean and GeometricMean are instance methods, so they are called from an object context. If I had implemented them as Shared methods, then I'd call them from a class context. The class contains a private helper method called NthRoot that's used by GeometricMean.

In general, API test automation does not test private helper methods because it is believed that any problems in them will be exposed by calling the methods that rely on the helpers. As with every general rule there are exceptions, and depending on the time and resources you have for your testing effort, you may want to test private methods, too (it can be very beneficial).

The API Test Automation

The basic API test harness is surprisingly short and is shown in Figure 5. The harness is a Visual Basic® .NET console application, often the type of program best suited for test automation. Console applications integrate easily into legacy test automation systems and can be manipulated easily in a Windows environment. As a general rule, you should write your test automation using the same language used by the system under test. If you use a different language you could run into cross-language issues (different interpretations of fundamental data types, for example). Before .NET this used to be a big problem but my recent experience at Microsoft has shown that test engineers can easily migrate among C#, Visual Basic .NET, and other .NET-compliant languages due to the unifying influence of the .NET Framework.

Figure 5 The Test Automation Harness (Close Figure 5)
Imports System
Imports System.IO

Module Module1
  Sub Main()
    Try
      Dim FS As New FileStream("..\TestCases.txt", FileMode.Open)
      Dim SR As New StreamReader(FS)
      Dim Line, CaseID, Method As String
      Dim Tokens(), TempInput() As String
      Dim Expected As String
      Dim Actual As Double

      Console.WriteLine()
      Console.WriteLine("CaseID Result Method         Details")
      Console.WriteLine("=============================================")
      Console.WriteLine()

      Line = SR.ReadLine()
      While Line <> Nothing
        Tokens = Line.Split(":")
        CaseID = Tokens(0)
        Method = Tokens(1).Trim()
        TempInput = Tokens(2).Split(" ")
        Expected = Tokens(3)

        Dim Input(TempInput.Length - 1) As Integer
        For I As Integer = 0 To Input.Length - 1
          Input(I) = Integer.Parse(TempInput(I))
        Next

        Dim M As New MathLib.Methods

        If Method = "ArithmeticMean" Then
          Actual = M.ArithmeticMean(Input)
          If Actual.ToString("F4") = Expected Then
            Console.WriteLine(CaseID & "   Pass   " & Method & _
              " actual = " & Actual.ToString("F4"))
          Else
            Console.WriteLine(CaseID & "  *FAIL*  " & Method & _
              " actual = " & Actual.ToString("F4") & _
              " expected = " & Expected)
          End If
        ElseIf Method = "GeometricMean" Then
          Actual = M.GeometricMean(Input)
          If Actual.ToString("F4") = Expected Then
            Console.WriteLine(CaseID & "   Pass   " & Method & _
              " actual = " & Actual.ToString("F4"))
          Else
            Console.WriteLine(CaseID & "  *FAIL*  " & Method & _
              " actual = " & Actual.ToString("F4") & _
              " expected = " & Expected)
          End If
        ElseIf Method = "HarmonicMean" Then
          Console.WriteLine(CaseID & "          " & _
          Method & " Not yet implemented")
        Else
          Console.WriteLine(" unknown method")
        End If

        Line = SR.ReadLine()
      End While ' test case loop

      Console.WriteLine()
      Console.WriteLine("=============== end test run ================")
      SR.Close()
      FS.Close()

      Console.ReadLine()

    Catch ex As Exception
      Console.WriteLine(ex.Message)
      Console.ReadLine()
    End Try
  End Sub
End Module

After creating a new Visual Basic .NET console application project using Visual Studio® .NET and adding a Project Reference to the MathLib.dll library, my first step was to create some test cases. I decided to use a simple text file. The following code displays the test case file that produced the output shown in Figure 2.

0001:ArithmeticMean:2 4 8:4.6667
0002:ArithmeticMean:1 5:3.0000
0003:ArithmeticMean:1 2 4 8 16 32:10.5000
0004:GeometricMean :1 2 4 8 16 32:6.6569
0005:GeometricMean :0:0.0000
0006:GeometricMean :2 4 8:4.0000
0007:HarmonicMean  :2 4 8:3.4286
0008:HarmonicMean  :2 3 6:3.0000

Each line represents a single test case. Each case has four fields separated by a ':' character delimiter (using a colon as a delimeter was an arbitrary choice). The first field is a test case ID, the second field signifies which method to test (I added spaces after GeometricMean and HarmonicMean purely for readability), the third field is a space-delimited list of inputs to the method, and the fourth field is the expected result. Placing test case data in a text file is simple and effective. Good alternative schemes for test case data storage are XML files, SQL databases, and Excel spreadsheets.

Where do the input values and expected results come from? As I mentioned in the previous section, examining the source code of the methods under test tells you how to call the methods, but it does not give you enough information to create a meaningful set of inputs and expected values for the test cases. A common practice—but a big mistake—is to use the system under test to generate expected results. In other words, you should not launch the application, enter some input integers, use the application to get a result, and then use the result as your test case expected result. This approach does not test the methods in the system under test; it just verifies that you get the same output given the same input!

So just how do you determine expected results? The theoretical guiding principle here is that you should use the product's specification documents. In theory, at least, the system under test will have documents that completely and precisely describe the product's behavior. Of course, the reality is that in a production environment the specification documents are often incomplete or out of date, and during the development cycle they may not have been written yet. The actual guiding principle for creating test cases is to do whatever is necessary: software testing is a practical endeavor. For the test cases in this example, to generate input and expected results I simply went to my old college statistics book and found some quizzes and solutions (you should always seek out some authoritative reference, like this statistics book, if at all possible). Additionally, you'll have to make use of software testing principles such as testing boundary conditions. Creating meaningful test cases is where experience and general knowledge of software testing comes into play.

The structure of the test automation in pseudocode is:

loop
  read a test-case line
  parse out test-case data
  build up input arguments
  call the method-to-test
  if actual value = expected value
    record a "pass" result
  else
    record a "fail" result
end loop

There are several alternatives for test case storage and test harness structure, but this simple one has proven to be remarkably robust and flexible on many projects. An important test automation guideline is to keep everything simple. I begin by declaring the key variables that the automation uses:

Dim FS As New FileStream("..\TestCases.txt", FileMode.Open)
Dim SR As New StreamReader(FS)

Dim Line, CaseID, Method As String
Dim Tokens(), TempInput() As String
Dim Expected As String
Dim Actual As Double

I open the test case's text file using classes from the System.IO namespace. The string variable line holds a single line representing one test case from the test case file. The variables caseID, method, and expected hold the test case ID, which method to test, and the expected result, respectively. Notice that the expected result is declared as a string type (because it is read in from a text file), but the actual result returned by the method under test is of type double so I'll have to do a type conversion at some point. This is an important rule of thumb in test automation: test case input arguments and expected values will usually initially be of type string (unless you use SQL for test case storage) so you'll have to convert the input to the appropriate data type. In addition, you'll also either convert the expected value from type string to match the actual value or convert the actual value to type string so that actual and expected values can be compared. Note that you should always explicitly convert data types and not rely on the implicit conversion available in some .NET-compliant languages like Visual Basic .NET. This means you, as the tester, are in control and have to think about the conversion process, adding another little push to get correct and accurate results while testing.

Figure 6 Processing a Test Case
Figure 6 Processing a Test Case

The string array variable tokens holds each part of a test case line. Variable tokens(0) holds the test case ID, tokens(1) holds which method to test, and so forth. String array tempInput holds each input argument. For example, if tokens(2) is "2 4 8", then tempInput(0) is "2", tempInput(1) is "4", and tempInput(2) is "8". Figure 6 shows the relationship between these variables for the first test case. Because these variables are logically related I could have placed them together into a struct or class but there are so few variables I wouldn't gain any clarity.

After displaying a simple output header line to the command shell, I'm ready to test. The test harness consists of a single test case data-controlled loop, as you can see in Figure 7.

Figure 7 Test Case Loop (Close Figure 7)
Line = SR.ReadLine()
While Line <> Nothing
  Tokens = Line.Split(":")
  CaseID = Tokens(0)
  Method = Tokens(1).Trim()
  TempInput = Tokens(2).Split(" ")
  Expected = Tokens(3)

  Dim Input(TempInput.Length - 1) As Integer
  For I As Integer = 0 To Input.Length - 1
    Input(I) = Integer.Parse(TempInput(I))
  Next

  Dim M As New MathLib.Methods

  If Method = "ArithmeticMean" Then
    ' test ArithmeticMean()
  ElseIf Method = "GeometricMean" Then
    ' test GeometricMean()
  ElseIf Method = "HarmonicMean" Then
    Console.WriteLine(CaseID & "          " & Method & _
      " Not yet implemented")
  Else
    Console.WriteLine("Unknown method")
  End If

  Line = SR.ReadLine()
End While ' test case loop

After I read a line of test case data, I use the String.Split method to parse the line into the string array tokens. Then I use Split again to break the input arguments and store them into the string array tempInput. I could have used regular expressions but I find traditional string parsing techniques easier to debug. I branch on the value of the method to test, test the method, and log a pass or fail value based on the method's results.

Testing each method is relatively simple. But before I can call each method, I must convert the input arguments stored as string types in tempInput into integers:

Dim Input(TempInput.Length - 1) As Integer
For I As Integer = 0 To Input.Length - 1
  Input(I) = Integer.Parse(TempInput(I))
Next

I create a new integer array of the same size as tempInput and then use the int.Parse method to convert each string representation of an argument into its integer counterpart. Now I can test the ArithmeticMean method, like so:

Dim M As New MathLib.Methods
Actual = M.ArithmeticMean(Input)
If Actual.ToString("F4") = Expected Then
  Console.WriteLine(CaseID & "   Pass   " & Method & " actual = " _
    & Actual.ToString("F4"))
Else
  Console.WriteLine(CaseID & "  *FAIL*  " & Method & " actual = " _
    & Actual.ToString("F4") & " expected = " & Expected)
End If

After I get the actual result (as type double), I compare the actual result with the expected result to determine if the test case passed or failed. Notice that I converted the actual result from type double to type string formatted with four fixed decimal places. If I had converted the expected result from type string to type double then I'd have to be careful about comparing two double types for exact equality and worry about rounding because double values are sometimes only approximations.

Writing results to the command shell is simple and effective. Because the test harness is a console application, you can easily send results to a text file by using system redirection when you run the harness, as shown here:

C:\>TestAutomation.exe > results.txt

Testing GeometricMean is similar to testing ArithmeticMean:

Actual = M.GeometricMean(Input)
If Actual.ToString("F4") = Expected Then
  Console.WriteLine(CaseID & "   Pass   " & Method & " actual = " _
    & Actual.ToString("F4"))
Else
  Console.WriteLine(CaseID & "  *FAIL*  " & Method & " actual = " _
    & Actual.ToString("F4") & " expected = " & Expected)
End If

Because GeometricMean is an instance method, I must instantiate an object of type MathLib.Methods and then call GeometricMean in the context of the object. After determining a pass or fail result, I write the results along with minimal details to the command shell. In an actual production environment you will want to add as much detail as possible to the output for failed test cases.

Discussion

You can extend and adapt the API test automation techniques I've presented here in several ways. Because this column is primarily instructional, I removed all error checking for clarity. In a production system you'll want to add exception handling try/catch-finally blocks to make sure your automation does not stop in mid-run. You'll also want to add counters that track the number of test cases that pass and fail and then log summary results. Another useful extension is to automate the automation—schedule the API test harness to run automatically (using the Windows Task Scheduler) and automatically send test run result summaries via e-mail (using the classes in System.Web.Mail namespace).

API test automation does not eliminate the need for manual testing, but automated testing has significant advantages over manual testing. First, automated testing can be much faster than manual testing—it is possible to run several hundred thousand test cases in the same time it would take to run only several hundred manual tests. However, you must also factor in the time required to design, write, debug, and maintain the automation suite. Often with automation, the initial investment is significant, and a substantial part of the benefit lies in the longer term payoff. Second, automated testing can be more accurate than manual testing—manual testing is subject to all kinds of human error. Third, automated testing is more precise in that automated tests are deterministic and run the same way every time given the same input. Fourth, automated testing is more efficient—automated tests can run anytime, freeing you to do other work. And fifth, automated testing actively promotes your skill growth while manual testing can be very repetitive.

The custom API test harness system I've described here nicely complements more structured third-party test frameworks like NUnit and even more sophisticated commercial test framework systems. In the days before .NET, writing custom test automation was so time consuming that it often wasn't done except on the largest projects. But the enormous productivity gains made possible by the .NET environment allow you to write custom automation quickly. The advantage of custom automation over frameworks written by someone else is that you can test interesting scenarios not anticipated by test frameworks. Also, when using a third-party test framework there is a risk that you'll test based on the capabilities of the framework rather than on the needs of your system under test. The disadvantage of writing custom test automation harnesses versus using preexisting test frameworks is the additional time you sometimes spend to create a custom harness and the possibility that you might introduce bugs into the harness.

There are four closely related ways you can use API test automation in a production environment. First, during the early stages of the product cycle, the development of API test automation itself often uncovers serious functional bugs in the methods used by the SUT. Second, during the middle stages of the product cycle, API test automation is useful as part of build verification testing—after you build a new version of the system under test, you can verify that the system meets some minimal threshold of functionality before moving forward with production. Third, during the latter stages of the product cycle you can use API test automation as part of developer regression testing—making sure that new code does not break existing functionality before you check the new code into your source control/build repository system. And fourth, before releasing the product you can use API test automation as part of a complete test pass.

With software systems growing in complexity and security becoming increasingly significant, thorough software testing is more important than ever before. API testing as I've described here is an absolutely essential part of your testing effort and the .NET environment makes it simple to implement automated API testing.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Advanced Basics: Digital Grandma

Advanced Basics
Digital Grandma
Duncan Mackenzie

Code download available at: AdvancedBasics0411.exe (183KB)
Contents
The Manifest Generator and Manifest Files
Creating the UI
Populating the Manifest
Serializing the Manifest to Disk
Application Settings and the Options Form
Checking for New Images
Conclusion

As a parent of a young child, I take a lot of pictures—many more than anyone would ever be interested in seeing. Well, anyone except my mother. This is her first grandchild and the one or two pictures I send to her each week only brush the surface of her grandmotherly needs. Unfortunately, it has been difficult for me to send her any more than a few photos at a time. When she had only a dial-up account and very little room in her Inbox, full-size photos would leave her phone line tied up for hours, so I decided that I should set her up with a better way to get her grandson fix. A digital photo frame, a system that downloads photos automatically from a Web site and displays them on a small LCD screen, would be a good solution, but I didn't like the idea of paying a monthly fee indefinitely. (My mother lives in Canada where there were no toll-free dial-up numbers for digital devices, and she has a perfectly good computer and monitor right on her desk.)

To address these problems, I considered a system that would automatically download any new photos that I have put on a Web server whenever she is connected to the Internet. By default this system would save photos into her "My Pictures" folder. She is running the "My Pictures Slideshow" screensaver, so new photos will automatically appear on her screensaver, just like magic! Even if e-mail storage and connection speed weren't issues, the idea of new photos simply arriving on her machine and appearing on her monitor was enough to make me want to build this system.

Fortunately, since I last visited, technology has improved in the Manitoba town where my mother lives and her dial-up has been replaced with broadband. Now, a standard retrieval from the Web will be even easier. My original plan included using some form of file transfer that would work with minimum bandwidth so I would not interfere with her Web surfing or regular e-mail retrieval, and downloads would have to be able to pause and resume automatically as the connection status changed.

If I did end up having to build that system, the Background Intelligent Transfer System (BITS), which works well for copying files over slow or intermittent connections and which supports restarting file transfers even after a machine restarts, would have been my answer. And I would have been able to take advantage of the COM interface I wrapped in a .NET library (available at Using Windows XP Background Intelligent Transfer Service (BITS) with Visual Studio .NET). Perhaps I'll add BITS as an option to this system at some point in the future, but for the moment I've decided to build an application that is really best suited for use over a broadband connection.

Figure 1 Photo/File System Architecture
Figure 1 Photo/File System Architecture

Although I was focused on a very specific goal, I tried to design a fairly generic system that could be used for more than just photos (see Figure 1). Here's how it works:

  1. The client machine (user) has a small application installed and running constantly.
  2. The client application is configured to retrieve a listing of available files (the manifest) from a specific URL and to download those files into a specific local directory.
  3. Whenever new files are ready to be sent to the client, the files are made available on an HTTP server, and the server's manifest file is then updated to reflect the currently available set of files.
  4. The client pulls down the manifest at regular intervals and then begins downloads for any files that it doesn't already have. Along with the path to each file, the manifest contains a hash of the file so that the client application can verify that it has downloaded the right file. The manifest also defines an action—either Notify, Execute, or None in this sample—that should be taken on the file after it is downloaded.
  5. As downloads complete, a local history is kept of the files that have been downloaded and the ones already in progress, so as to avoid duplication.

The Manifest Generator and Manifest Files

I created two key systems—the client application and a manifest file generating a utility for use on the server. I could have created an initial manifest file by hand and then built the client system using that as input, but I decided to build the real manifest generator first. Once I had the ability to create manifest files, I could build code into my client application to read that data back into memory. The manifest generator's task is simple: take a set of files from the local machine (or a network share) and generate a manifest file complete with checksums, Web paths (you have to enter those based on the server to which you'll upload these files), and the action that should be taken at the client (Notify, Execute, None).

I used XML serialization to produce and consume the XML for the manifest file, so the first step in each case was to define the classes that I wanted to use. These classes are very simple, containing almost no code; they are just places to store data (see Figure 2).

Figure 2 The Class View
Figure 2 The Class View

I created the collections using CodeSmith, a code-generation tool (see Ten Must-Have Tools Every Developer Should Download Now for more information), but I customized the resulting classes by adding some Shared methods onto the collections to make it easier to save and restore the entire collection to disk (see Figure 3).

Figure 3 Adding Shared Methods (Close Figure 3)
Public Shared Sub SaveManifest( ByVal fileToSaveTo As IO.StreamWriter, _
        ByVal m As Manifest)
    Dim xw As New XmlTextWriter(fileToSaveTo)
    Dim xs As New XmlSerializer(GetType(Manifest))
    xs.Serialize(xw, m)
    fileToSaveTo.Close()
End Sub

Public Shared Function OpenManifest( _
        ByVal fileToReadFrom As IO.StreamReader) As Manifest
    Dim xr As New XmlTextReader(fileToReadFrom)
    Dim xs As New XmlSerializer(GetType(Manifest))
    Dim result As Manifest = DirectCast(xs.Deserialize(xr), Manifest)
    fileToReadFrom.Close()
    Return result
End Function

Note that I choose to pass in a StreamWriter and StreamReader to these methods instead of a file path. To access a local file using its path requires certain security permissions, and is usually not possible at all in an application running with restricted permissions, but in that situation you can still obtain a Stream through the Open and Save dialog controls or by opening a file in Isolated Storage. By accepting a Stream into my functions, I left my options open and didn't force full-trust requirements onto my code. I tried to do this as much as possible, even though I expected it to be installed locally where it could be run with Full Trust.

Creating the UI

The UI for the manifest generator, which I built with Windows® Forms, consists of a DataGrid to hold the list of files, a couple of textboxes to provide the destination address of the manifest file and of the images, and the following four buttons:

  • newManifest (to clear the current manifest and start a new one)
  • openManifest (to open an existing manifest as a starting point)
  • populateManifest (to pick the folder where the images are stored locally)
  • generateManifest (to generate the file out to disk)

If you choose to open an existing file, it is deserialized back into an instance of the Manifest class (see Figure 4).

Figure 4 Deserializing to the Manifest Class (Close Figure 4)
Private Sub openManifest_Click( ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles openManifest.Click
    If Me.pickManifestFileToOpen.ShowDialog = DialogResult.OK Then
        Me.currentManifest = Manifest.OpenManifest( _
                    New IO.StreamReader( _
                    Me.pickManifestFileToOpen.OpenFile))
        setupDataBinding()
    End If
End Sub

Private Sub saveManifest_Click( ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles saveManifest.Click
    If Not Me.currentManifest Is Nothing Then
        If Me.pickSaveLocation.ShowDialog = DialogResult.OK Then
            Manifest.SaveManifest( New IO.StreamWriter( _
                Me.pickSaveLocation.OpenFile), Me.currentManifest)
        End If
    End If
End Sub

I also added a Save Dialog, Open Dialog, and Folder Browser dialog to the form to support the rest of the code.

Populating the Manifest

Clicking the Populate button allows you to pick a directory from which to load your images, and then it loads all the files (matching a wildcard-based search pattern) from that folder into the current manifest file. For each file, a GUID is assigned (this GUID will be used by the client to track which files it has already downloaded), and then a hash is created using the MD5 classes in the Microsoft® .NET Framework (see Figure 5). The CreateChecksumString method converts the hash value from a byte array to a string of hexadecimal.

Figure 5 Populating the Manifest (Close Figure 5)
Imports System.Security.Cryptography
...
Private Sub AddFileToManifest(ByVal filePath As String)
    Dim hasher As New MD5CryptoServiceProvider
    Dim fi As New IO.FileInfo(filePath)
    Dim imageStream As IO.FileStream = fi.OpenRead

    Dim myFile As New File
    With myFile
        .action = "None"
        .fileSize = fi.Length
        .fileURL = fi.Name
        .checkSum = File.CreateChecksumString( hasher.ComputeHash(imageStream))
    End With
    Me.currentManifest.includedFiles.Add(myFile)
End Sub

Serializing the Manifest to Disk

The generateManifest button asks you to pick a file name and then takes the currently loaded manifest file and serializes it out to that location. With the file created, your next step is to copy the manifest and all of the images that it references up to your Web server. Make sure that wherever you put the files, the path for each of the images matches the value you entered when creating the manifest. Now, with the manifest and the files all uploaded to the Web server, I'll explain the client side of this system.

The client application is run in the user's context, as a regular application (not a Windows service), and polls a single Web location for updates to the manifest file. Whenever that file is modified, it is pulled down and any new images are downloaded to a local directory.

The whole point of this system is to automate the delivery of these photos to the grandmother, so the client system should be unobtrusive. Although I am loath to add yet another system tray icon to anyone's machine, it seems to be the best way to make the application available for configuration but not too visible to the user. I decided to create a Windows Forms application that, by default, does not display any UI other than in the system tray. If needed, the user can open up an options dialog (see Figure 6) from that icon, but otherwise it will just quietly go about its business.

Figure 6 The Options Dialog
Figure 6 The Options Dialog

To produce a Windows Forms application that doesn't (initially) display any forms, I created a Sub Main for my application and within that routine I set up the system tray icon, a timer (to check for updates), and a context menu for the icon. The context menu has only two choices: exit the application or open the Options dialog. I've made the Options choice bold to indicate that it's the default action that would occur if you double-clicked the tray icon. Once everything was set up, I started my application's main message loop (without any Form instances at all) and then I just sat around and waited for the timer to fire an event or for the user to interact with the system tray icon (see Figure 7).

Figure 7 The Windows Forms App (Close Figure 7)
Sub Main()
    'load the history and settings
    'information from serialized files
    Reload()

    'the Settings files stores the interval in minutes
    'Timers work in milliseconds, so a little conversion
    'is necessary to make them work together.
    checkTimer.Interval = appSettings.CheckInterval * 60000
    checkTimer.Enabled = True

    'add a handler for the Tick event of the timer
    AddHandler checkTimer.Tick, AddressOf Tick

    'Create system tray icon, use a module level variable for the control
    ni = New NotifyIcon
    ni.Icon = New Icon(GetType(Main), "camera.ico")
    ni.Visible = True
    ni.Text = "Retriever: Waiting..."

    'add an event handler for when the user double-clicks the icon
    AddHandler ni.DoubleClick, AddressOf NotificationIconDoubleClicked

    'set up the context menu, including event handlers
    Dim ctxMenu As New ContextMenu
    ctxMenu.MenuItems.Add("View Options", _
       AddressOf ViewOptionsForm).DefaultItem = True
    ctxMenu.MenuItems.Add("Exit", _
       AddressOf EndApplication)
    'and assign it to the NotifyIcon so that it will pop up when
    'the icon is right-clicked.
    ni.ContextMenu = ctxMenu

    'run the application, using Application.Run
    'to create a new message loop
    Application.Run()

    'when the app is exiting hide the notify icon
    ni.Visible = False

    'save the server info and settings to XML files
    Persist()
End Sub

Application Settings and the Options Form

I used a settings class to store the user's preferences for delivery location, URL of the manifest file, frequency of polling, and whether actions should be taken (Execute or Notify) if they are specified in the manifest file. I also stored in the settings file the last modified date of the manifest file, which is a value the user wouldn't change but that I need in order to manage the download process.

The settings information is stored in isolated storage using XML serialization, with static methods similar to the ones I created on the manifest class earlier. By default, the delivery location is the user's My Pictures folder, the manifest URL is taken from the application's configuration file (installed with the application), and the polling frequency is once per hour. When the user decides to view the options dialog, I populate it with the current settings upon opening and then, if the user clicks OK to close, I store those into the settings file and serialize it back into isolated storage. All the dialog-related code is included in the complete code sample that you can download from the MSDN®Magazine Web site. I used another file, history.dat, also stored in isolated storage, which is serialized to and from a hashtable in my app. I'll cover the history hashtable and file when I go through the image retrieval code.

Checking for New Images

When checking for updates, I disabled the timer to avoid code that reenters whenever the timer fires, and then I created an HttpWebRequest to check for updates to the manifest file. By constructing my request to include the last-modified date, I get a "Not Modified" (304) response from the server if the file hasn't changed, and I receive the file back if it has been modified (see Figure 8).

Figure 8 Checking for Updates (Close Figure 8)
Sub CheckForUpdates()
    Dim urlToRetrieve As String = appSettings.ManifestURL
    Dim wr As HttpWebRequest _
        = DirectCast(WebRequest.Create(urlToRetrieve), _
          HttpWebRequest)
    wr.IfModifiedSince = appSettings.LastDownload
    wr.AllowAutoRedirect = True
    wr.Credentials = CredentialCache.DefaultCredentials
    Try
        Dim resp As HttpWebResponse _
            = DirectCast(wr.GetResponse(), HttpWebResponse)

        If resp.StatusCode = HttpStatusCode.OK Then
            Dim xmlDoc As New Xml.XmlDocument
            Dim respStream As IO.Stream
            respStream = resp.GetResponseStream
            xmlDoc.Load(respStream)

            Dim rootURL As String _
                = xmlDoc("Manifest")("rootURL").InnerText
            If Not rootURL.EndsWith("/") Then
               rootURL &= "/"
            End If

            Dim navigator As _
                Xml.XPath.XPathNavigator = xmlDoc.CreateNavigator
            Dim manifestItems As Xml.XmlNodeList
            manifestItems = xmlDoc.GetElementsByTagName("File")
            For Each n As Xml.XmlNode In manifestItems
                If Not InHistory(n("GUID").InnerText) Then
                    GetFile(n, rootURL)
                End If
            Next
            appSettings.LastDownload = resp.LastModified
            Persist()
        End If
    Catch ex As WebException
        Debug.WriteLine(ex.Message)
        Dim webResp As HttpWebResponse _
            = DirectCast(ex.Response, HttpWebResponse)
        If webResp.StatusCode = HttpStatusCode.NotModified Then
            appSettings.LastDownload = webResp.LastModified
            Persist()
        End If
    End Try

End Sub

Assuming it has been changed, I download that XML file into memory and (using an XPathNavigator) walk through all of the file entries, comparing their GUIDs against entries in my download history. If a file hasn't been downloaded before, I synchronously download it using another instance of HttpWebRequest. I chose not to use the async method (BeginGetResponse) because I wanted this application to have little or no impact on the performance of the client machine (and running several requests at once would certainly increase its impact).

Once downloaded, the file is saved into a buffer. If its hash matches the value from the manifest file, I move it into the final target location and update the history file. Checking the hash is accomplished with almost the exact same code as was used to compute the hash in the manifest generator. If it fails to match, it is deleted but will be downloaded again if it is included in the manifest file the next time it is downloaded (see Figure 9).

Figure 9 Downloading and Validating a File (Close Figure 9)
Private Sub GetFile(ByVal n As Xml.XmlNode, ByVal rootPath As String)
    Dim basePath As New Uri(rootPath)
    Dim url As New Uri(basePath, n("fileURL").InnerText)
    Dim localPath As String = IO.Path.Combine( _
        appSettings.DropDirectory, n("fileURL").InnerText)

    Dim wc As New WebClient
    wc.Credentials = CredentialCache.DefaultCredentials
    Dim buffer() As Byte = wc.DownloadData(url.ToString)
    Dim hash As String = CreateChecksumString( _
        New MD5CryptoServiceProvider().ComputeHash(buffer))

    If hash = n("checkSum").InnerText Then
        Dim fs As IO.FileStream
        Dim bw As IO.BinaryWriter
        Try
            fs = New IO.FileStream(localPath, IO.FileMode.Create)
            bw = New IO.BinaryWriter(fs)
            bw.Write(buffer)
            Dim he As New HistoryEntry
            he.DateDownloaded = Now
            he.GUID = n("GUID").InnerText
            he.localPath = localPath
            history.Add(he.GUID, he)
            DoAction(n, localPath)
        Catch ex As Exception
            MsgBox(ex.Message)
        Finally
            bw.Close()
            fs.Close()
        End Try
    Else
        Debug.WriteLine("Hash Not Matched: " & hash)
    End If
End Sub

If an action (Execute or Notify) has been specified for the file you've just downloaded (and the user has not disabled actions through her settings), then the file should be run or a message should be displayed to the user (Notify). I could see adding an additional action, Print, and choosing to send documents instead of images. It would be an asynchronous, high-quality alternative to using a fax machine (see Figure 10).

Figure 10 Execute or Notify (Close Figure 10)
Private Sub DoAction(ByVal n As Xml.XmlNode, ByVal localPath As String)
    If appSettings.Actions Then
        Select Case n("Action").InnerText
            Case "Execute"
                System.Diagnostics.Process.Start(localPath)
            Case "Notify"
                MsgBox(String.Format("{0} Downloaded", localPath), _
                    MsgBoxStyle.Information, "New File Downloaded")
            Case Else
                'do nothing
        End Select
    End If
End Sub

Conclusion

Although it works for its limited purpose, this application is intended to serve as a sample. Starting from this concept, you can create a variety of applications or just extend this one to include more functionality. You can turn it into a BITS-based version, but you can also apply the concept to work-related data, setting up security on the drops and associating certain manifests with specific clients. If your Web host runs ASP.NET, you might want to change the manifest file generation to occur dynamically through an ASPX page or through a custom HTTP Handler, instead of through a static file.

If you are interested in photo blogs, you could modify the app to use RSS instead of a custom format for the manifest, although you'll need to add your own namespace so that you can add a couple of additional elements such as the hash value.

For me, though, I'll probably stick with the solution just as I designed it. Grandma Mackenzie is happy with it, so I can stop coding and get back to my real job—taking pictures of my son.

Send your questions and comments for Duncan to basics@microsoft.com.

Duncan Mackenzie is the MSDN Content Strategist for Visual Basic and C# and the author of the Coding 4 Fun column on MSDN online. He can be reached through his blog at http://weblogs.asp.net/duncanma or personal site at http://www.duncanmackenzie.net.


Cutting Edge: The ASP.NET 2.0 Wizard Control

Cutting Edge
The ASP.NET 2.0 Wizard Control
Dino Esposito

Code download available at: CuttingEdge0411.exe (129KB)
Contents
What's Your Definition of a Wizard?
The Wizard Control in ASP.NET 2.0
Using the Wizard Control
Adding Steps to a Wizard
The "Add New Employee" Wizard
Wizard Events
The Wizard Sidebar

ASP.NET has a lot to offer to both the low-level programmer willing to control every little step of the code and the busiest of developers who needs to point-and-click his way through Web app development using just a few existing components.

For the fan of pre-built controls, ASP.NET 2.0 provides a number of rich controls that form the foundation for feature-rich Web applications. Components like the TreeView and Wizard are actually comprehensive services that the ASP.NET runtime makes available to Web pages. In order to give you a feel for how they operate, I'll dive into the Wizard control and its underpinnings in this month's column. As you read, however, remember that ASP.NET 2.0 is still in Beta and is therefore subject to change.

What's Your Definition of a Wizard?

Frequently when you develop for the Web and for Windows® you need to collect user input using forms (dialog boxes in the jargon of Windows). When the input that needs to be collected can be divided into specific categories, multiple forms are typical, one for each category. Often, the whole procedure is broken into various steps, each of which validates a particular subset of the expected data. This multistep procedure is usually called a wizard.

A wizard lets you proceed linearly (from Step 1 to Step N) but also lets the user skip unnecessary steps or return to previous ones to change values (nonlinear design can also be driven programmatically, as will be discussed shortly). Implementing a wizard over the Web is a task that many developers need to tackle, but in the past managing navigation and data collection in each step has not always been a simple process. Thankfully, ASP.NET 2.0 provides a new Wizard control to make the process easier.

To link a series of forms into a wizard-like user interface, you have to manage the navigation between forms, handle data persistence, and manage the state in each step. Cross-page posts and show/hide panels are the two main techniques that can be employed in ASP and ASP.NET to build wizards. The cross-page post technique consists of posting the input from the first page to a different page and storing previous content in hidden fields. With the show/hide panel technique, the various views are incorporated in a single page and are shown or hidden based on the current step.

The latter technique is ideal for ASP.NET 1.x applications. As you'll see later, show/hide is also the key idea behind the ASP.NET 2.0 implementation of the Wizard control.

The Wizard Control in ASP.NET 2.0

The ASP.NET Wizard control simplifies many of the tasks associated with building a series of forms to collect user data. The control provides a mechanism that allows you to easily build the desired wizard as a collection of steps, add a new step, or reorder the steps. You don't have to write any infrastructure whatsoever for navigation or to persist user data between steps.

To use the Wizard control—the <asp:Wizard> element—you follow these simple steps: drop a Wizard control onto your Web Forms, add constituent controls, images, and text to each wizard step, and access the wizard's data between steps.

The Wizard control is a composite control that inherits from the CompositeControl base class and gets integrated designer support from Visual Studio® 2005. It posts back to itself to maintain view state information. This fact alone allows for nonlinear navigation and guarantees that form elements are automatically repopulated across multiple views of the wizard procedure.

Figure 1 Wizard Control Architecture
Figure 1 Wizard Control Architecture

Each step in a wizard is a sort of panel—a container of text, markup, images, and custom user controls. Figure 1 shows the architecture of the Wizard control in ASP.NET 2.0.

Using the Wizard Control

A Wizard control is made up of four main parts: header, view of the current step, navigation bar, and sidebar. All of these constituent parts, and much more, can be styled using a number of properties, such as those listed in Figure 2. The contents of the header, sidebar, and navigation bar can be further customized through the templates listed in Figure 3.

Figure 3 Wizard Control Templates (Close Figure 3)

Template | Description
FinishNavigationTemplate | Specifies the navigation bar shown before the last page of the wizard; by default, navigation bar contains the Previous and Finish buttons
HeaderTemplate | Specifies the title bar of the wizard
SideBarTemplate | Used to display content in the left side of the wizard control
StartNavigationTemplate | Specifies the navigation bar for the first view in the wizard; by default, contains only the Next button
StepNavigationTemplate | Specifies the navigation bar for steps other than first, finish, or complete; by default, contains Previous and Next buttons
Figure 2 Style Properties (Close Figure 2)

CancelButtonStyle | Wizard Cancel button
FinishStepButtonStyle | Wizard Finish button
FinishStepPreviousButtonStyle | Wizard Previous button at the finish step
HeaderStyle | Wizard header
NavigationButtonStyle | Navigation buttons
NavigationStyle | Navigation area
NextStepButtonStyle | Wizard Next button
PreviousStepButtonStyle | Wizard Previous button
SideBarButtonStyle | Buttons on the sidebar
StartStepNextButtonStyle | Wizard Next button at the start step
StepStyle | Area where steps are displayed

The code in Figure 4 outlines the typical code of a Wizard control in an ASP.NET application. Top-level attributes let you configure the appearance of the control by changing its fonts, border, colors and title. Figure 5 lists the main properties of the Wizard class.

Figure 5 Main Properties of the Wizard Class (Close Figure 5)

Property | Description
ActiveStep | Returns the current wizard step object; the object is an instance of the WizardStep class
ActiveStepIndex | Gets and sets the zero-based index of current wizard step
DisplaySideBar | Toggles the visibility of the sidebar; the default value is True
FinishStepButtonText | Gets and sets the text for the Finish button
HeaderText | Gets and sets the title of the wizard
NextStepButtonText | Gets and sets the text for the Next button
PreviousStepButtonText | Gets and sets the text for the Previous button
Figure 4 Typical Wizard Control Skeleton (Close Figure 4)
<asp:wizard runat="server" id="MyWizard"
   HeaderText="Add a New Employee">
   <wizardsteps>
      <asp:wizardstep runat="server" steptype="auto"
         title="Enter Employee Name">
      ...
      </asp:wizardstep>
      ...
      <asp:wizardstep runat="server" steptype="auto"
         title="Finalizing...">
      ...
      </asp:wizardstep>
      <asp:wizardstep runat="server" steptype="complete">
      ...
      </asp:wizardstep>
   </wizardsteps>
</asp:wizard>

Adding Steps to a Wizard

The <WizardSteps> block gathers all the defined steps in which the wizard procedure is articulated. The WizardSteps collection can be edited visually through the property dialog shown in Figure 6. You can add two types of steps to the collection—a simple wizard step or a templated wizard step. In the former case, you add a WizardStep control; in the latter case, an instance of the TemplatedWizardStep control is added to the page. The two-step controls work in a similar manner and accept a collection of server controls defined explicitly. A WizardStep control looks like this:

<asp:wizardstep runat="server" steptype="auto" title="One step">
    <div>
      <!-- any controls go here -->
    </div>
</asp:wizardstep>

Figure 6 WizardStep Collection Editor
Figure 6 WizardStep Collection Editor

The contents of a WizardStep can be dynamically adjusted and steps can be programmatically added or removed as you please. However, to allow for further customization, TemplatedWizardStep can be used instead of WizardStep, allowing users to change the content and navigation templates within it. By default, you can only have three different navigation templates: start, step, and finish. Using TemplatedWizardStep allows you to have different navigation buttons on each of the steps.

Here's how you define a templated step:

<asp:TemplatedWizardStep runat="server" steptype="auto">
  <ContentTemplate>
      <!-- any UI goes here -->
  </ContentTemplate>
</asp:TemplatedWizardStep>
The <ContentTemplate> element corresponds to an instance of an object that implements the ITemplate interface. This object gets or sets the template for displaying the content of a page in the Wizard. It goes without saying that you can define or change the template programmatically. In addition, a templated wizard step control lets you modify the navigation bar through a template:
<asp:TemplatedWizardStep runat="server" steptype="auto">
  <ContentTemplate>
      <!-- any UI goes here -->
  </ContentTemplate>
  <CustomNavigationTemplate>
      <asp:Button Runat=Server Text="My Button" />
      ...
  </CustomNavigationTemplate>
</asp:TemplatedWizardStep>

By adding a <CustomNavigationTemplate> block you can take full control of the navigation usually displayed at the bottom of the current view. You normally fill the <CustomNavigationTemplate> block with a series of buttons, but other controls are acceptable as well, with the restriction that those controls must support command bubbling, as does Button.

A wizard step is just one of the forms in which you break up the original user interface to build the wizard. So a wizard step will typically contain input controls such as text and checkboxes, dropdown lists, calendars, validators, and whatever else you can use to collect data from users.

All step classes inherit from the WizardStepBase class. The base class provides each child class with a property named StepType. The property gets or sets the type of navigation buttons to display for a page in a Wizard control. Possible values are listed in Figure 7. By default, the type of a step is dynamically determined by the control. The first step is a Start step, and the last is the Finish step; unless specified by the user, the control doesn't automatically assign steps to be Complete steps. All other intermediate steps are ordinary steps. The main difference between the step types is in the list of buttons displayed in the navigation area. An ordinary step allows a user to move back and forth, whereas a Start step doesn't display a Back button, and a Complete step doesn't provide any navigation whatsoever. As mentioned, all these built-in rules can be altered if you use a templated solution.

Figure 7 Step Types (Close Figure 7)

Setting | Description
Auto | Default setting; forces the wizard to determine how each contained step should be treated.
Complete | The last page that the wizard displays, usually after the wizard has been completed. The navigation bar and the sidebar aren't displayed.
Finish | The last page used for collecting user data. It lacks the Next button, and it shows the Previous and Finish buttons.
Start | The first screen displayed, with no Previous button.
Step | All other intermediate pages, in which the Previous and Next buttons are displayed.

Figure 8 shows the full source code of a sample wizard that collects any data that is relevant to add a new employee to a database. Needless to say, the employee of the sample can be replaced with any entity such as an invoice, a registration form, or a user of some sort of service.

Figure 8 Data Collection Wizard (Close Figure 8)
<%@ page language="C#" %>

<html>
<head runat="server">
    <title>Employee Wizard</title>
</head>
<body>

<form runat="server">
    <asp:wizard runat="server" id="MyWizard"
        Font-names="verdana"
        BackColor="lightcyan"
        ForeColor="navy"
        Style="border:outset 1px black"
        HeaderText="Add a New Employee"
        ActiveStepIndex="0">
        <stepstyle backcolor="gainsboro" borderwidth="1"
            borderstyle="Outset" />

        <sidebartemplate>
            <div style="height:300px">
            <img src="/source/images/wizard.jpg" width=100% />
            <asp:datalist runat="Server" id="SideBarList">
                <ItemTemplate>
                    <asp:linkbutton runat="server" id="SideBarButton" />
                </ItemTemplate>
            </asp:datalist>
            </div>
        </sidebartemplate>

        <wizardsteps>
        <asp:wizardstep runat="server" steptype="auto"
                title="Enter Employee Name">
            <div style="height:200px">
                <table><tr><td>First Name</td><td>
                    <asp:textbox runat=server id="FirstName" />
                    <asp:requiredfieldvalidator runat="server"
                        id="FirstNameValidator"
                        text="*"
                        errormessage="Must indicate a first name"
                        setfocusonerror="true"
                        controltovalidate="FirstName" />
                    </td></tr>
                <tr><td>Last Name</td><td>
                    <asp:textbox runat=server id="LastName" />
                    <asp:requiredfieldvalidator runat="server"
                        id="LastNameValidator"
                        text="*"
                        errormessage="Must indicate a last name"
                        setfocusonerror="true"
                        controltovalidate="LastName" />
                </td></tr>
                <tr><td height="100"></td></tr></table>
                <asp:validationsummary runat="server"
                    displaymode="List" id="Summary" />
            </div>
        </asp:wizardstep>
        <asp:wizardstep runat="server" steptype="auto"
                title="Personal Information">
            <div style="height:200px">
                <table>
                    <tr><td>Hire date</td><td><asp:textbox runat="server"
                        id="HireDate" /></td></tr>
                    <tr><td>Title</td><td><asp:textbox runat="server"
                        id="TheTitle" /></td></tr>
                    <tr><td height="100px"></td></tr>
                </table>
            </div>
        </asp:wizardstep>
        <asp:wizardstep runat="server" steptype="auto"
                title="Optional Information">
            <div style="height:200px">
                <table>
                    <tr><td valign="top">Notes</td><td>
                        <asp:textbox runat="server" id="Notes"
                            rows="10" columns="25" textmode="MultiLine"/>
                    </td></tr>
                </table>
            </div>
        </asp:wizardstep>
        <asp:wizardstep runat="server" steptype="auto"
                title="Finalizing...">
            <div style="height:200px">
                <asp:label runat="server" id="ReadyMsg" />
            </div>
        </asp:wizardstep>
        <asp:wizardstep runat="server" steptype="complete">
            <div style="height:200px">
                <asp:label runat="server" id="FinalMsg" />
            </div>
        </asp:wizardstep>
        </wizardsteps>

        <navigationbuttonstyle borderwidth="1" width="80"
            borderstyle="Solid" backcolor="lightgray" />
        <headerstyle horizontalalign="Right" font-bold="true"
            font-size="120%" />
        <sidebarstyle backcolor="snow" borderwidth="1"
            font-names="Arial" />
    </asp:wizard>

</form>
</body>
</html>

Speaking of users, I'd like to mention another new control in ASP.NET 2.0—the CreateUserWizard control—which provides a built-in wizard to add a new user to the current membership data provider. This built-in control differs from other controls you might write in that it automatically targets the configured membership provider and data store for membership information you employ at login time in a Forms Authentication scenario.

The "Add New Employee" Wizard

As you can see, Figure 8 doesn't include any server-side code. You can save that code to an ASPX file, point the browser to it, and see the page shown in Figure 9.

Figure 9 Employee Data Wizard
Figure 9 Employee Data Wizard

The wizard consists of three steps. First you collect the name of the new employee, personal information, and notes. Next, you finalize the wizard by creating a summary page where all the collected information is shown to the user for confirmation. Then the wizard completes the operation, persists any data, and displays the final message through a completion step. As Figure 9 shows, you can easily (and declaratively) implement validation to make sure users move to the next step only if proper data has been entered. You should use ASP.NET validators in the wizard's view for quick client-side checks; you should use transition events (which I'll discuss a little later on) if you need to access any server-side resources to validate input.

Although a wizard's forms are related logically, they still need to look related. A few simple tricks shown in Figure 8 will help you get there. Most importantly, each step should be the same size, with special attention paid to each form's height. This ensures that the button bar is not moved up and down as you move through pages (an annoying habit of wizards that frustrates many users). If you're still a little bit skeptical, just try it: write a frequently clicked button bar (like a DataGrid pager), change the page size so that the button bar is displayed at different heights each time, then try clicking your way through it. Annoying, isn't it?

ASP.NET 2.0 provides customizable themes to make your wizard especially attractive. A theme is a set of graphical settings (stylesheets and control attributes) that apply to all server controls in the page or application. They provide a useful way to apply a standard, consistent look to Web pages and controls. Themes are deployed as text files and can be global to the application as well as local to the various individual pages. It's worth noting, however, that no global themes will be provided in the ASP.NET 2.0 final product. If you're looking for an attractive, easily applied set of built-in styles to use with your server controls you should use the Visual Studio 2005 AutoFormat feature instead.

In the top-left corner of your wizard you can add an image. It provides a little visual interest and can convey some additional information, especially if you're using a sidebar (see Figure 9).

Some of the style properties listed in Figure 2 refer to buttons in the navigation area. There are many ways for you to change the look of these controls. The easiest way to give all buttons a consistent appearance is to define a NavigationButtonStyle object. This setting ensures that all buttons share a common set of styles while allowing you to customize each button individually through a specific style such as CancelButtonStyle.

You can control the text on each button with text properties. You can also use images and alternate text. Here's an example:

MyWizard.NextStepButtonText = ">>";
MyWizard.NextStepButtonImageUrl = "...";

If both are set, the text takes precedence. Note that the NextStepButtonXxx properties used in the preceding example take effect only on ordinary steps. The Next button displayed in the Start page is customized using a different set of properties—further proof of the extreme flexibility of the Wizard control user interface.

Wizard Events

The Wizard control does a great job of shielding you from many of the details connected with a multistep input, especially in a Web scenario. However, there's nothing it can do about the core task for which the wizard exists; that must be manually coded. For this to happen, an event model is needed that signals when the wizard is about to accomplish a given task like moving to the Next or Previous page, or finishing or starting the procedure.

Besides the classic events of any server control like Init, Load, and PreRender, the Wizard control also features a number of more specific events, as listed in Figure 10. As you can see, five out of six events relate to click events on the navigation buttons and sidebar elements. The sixth event is ActiveStepChanged, which occurs when the active view changes as the result of user action. When the user clicks, the page posts back and after the default page reinitialization the first server event that fires is the click event for the selected button. Next, if the previously invoked event handler caused a view switch, the ActiveStepChanged event is triggered.

Figure 10 Wizard Control Events (Close Figure 10)

Event | Description
ActiveStepChanged | User switches to a new page in the control; requires an EventHandler delegate
CancelButtonClick | Cancel button is clicked; requires an EventHandler delegate
FinishButtonClick | Finish button is clicked; requires a WizardNavigationEventHandler delegate
NextButtonClick | Next button is clicked; requires a WizardNavigationEventHandler delegate
PreviousButtonClick | Previous button is clicked; requires a WizardNavigationEventHandler delegate
SideBarButtonClick | Button in the sidebar area is clicked; requires a WizardNavigationEventHandler delegate

The delegate required by the ActiveStepChanged event is the simplest of all—EventHandler—which doesn't pass on any additional information and just works as a plain notification that something happened. CancelButtonClick denotes the abort of the wizard and likewise doesn't pass the page more information than the notification itself (as a note of warning, if you commit data as the user moves through steps rather than committing everything at the end, you are responsible for writing code to "undo" any changes that might have occurred prior to the point at which a user cancels). All other events require a richer delegate—WizardNavigationEventHandler—which uses the WizardNavigationEventArgs class to carry some extra information: public class WizardNavigationEventArgs

{
   bool Cancel {get; set;}
   int CurrentStepIndex {get;}
   int NextStepIndex {get;}
}

It is worth noting that all click events occur before the view switch happens; the click event is your last chance to perform any necessary validation and draw any definitive conclusions about the step. The two integer properties indicate the step index that's set when the user clicked (CurrentStepIndex) and the new step index to be set after the event processing is completed (NextStepIndex). This model is nearly identical to the one that ASP.NET 1.x uses for DataGrid paging. In that context, the PageIndexChanged event holds the CurrentPageIndex value and the NewPageIndex property which you can peruse. This also lets you programmatically accept the proposed step change.

Any step change can be rejected by setting the Cancel property to true. For example, you define a next button click handler and when the user clicks to see the next page, you check the last name entered against your own blacklist of names and, in that case, refuse to switch. Here's an example:

void OnNext(object sender, WizardNavigationEventArgs e)
{
   if (LastName.Text == "Esposito")
   {
      e.Cancel = true;
      return;
   }
}

To get and set values on view controls, you use the usual ASP.NET ID-based syntax. Technically speaking, this is supported because all the wizard views are part of the same page, although only one view is active and visible at a time. (For the same reason, you can't have two controls, even in different views, which share the same ID.)

You can set any required event handlers either using the declarative, attribute-based syntax of ASP.NET, or the delegate syntax supported by .NET languages. For example, the following snippet shows how to define an event handler through an attribute:

<asp:wizard runat="server" id="MyWizard" HeaderText="Add a New Employee"
  ...
  OnNextButtonClick="OnNext" />

FinishButtonClick is the place to add any code that finalizes the wizard. However, in most cases you might also want to display a summary of the input before proceeding with the finishing step. This is normally the function of that step. However, the text displayed in the Finish step view can only be determined dynamically by looking at what users entered in previous steps. To prepare the Finish step for display, you can write a Next button and Previous button click handler that share the same preliminary code to check the type of the step. You can do as follows:

WizardStepType t = MyWizard.WizardSteps[e.NextStepIndex].StepType;
if (t == WizardStepType.Finish)
{
   // Prepare the summary and fill the UI of the Finish step
   ...
}

Basically, you check the type of the step and if it is a Finish step, you collect the input data and prepare the summary for the user. As mentioned previously, a Finish step displays only Previous and Finish buttons in the navigation bar in order for the user to move back or finalize the wizard.

Data persistence and integration of the input with the host application are tasks accomplished in the final Complete step. You need a FinishButtonClick handler for this.

The MoveTo method can be used to move to a particular step programmatically. The method's prototype is as follows:

public void MoveTo(WizardStep step)

The method is a simple wrapper around the setter of the ActiveStepIndex property and requires you to pass a WizardStep object, a requirement which can sometimes be problematic. If you want to jump to a particular step, however, setting the ActiveStepIndex property directly is just as effective. This can be useful in scenarios where a user's choice in a previous step eliminates the need to show a future step. For example, if a user selects "no gift wrapping" when making an online purchase, there's no point in showing them the step that presents gift wrapping options. When doing nonlinear navigation such as this, it's often beneficial to disable the sidebar display. Otherwise, a user would be able to randomly go to steps that might not have been part of their path.

To get the WizardStep object of a particular step, you can access the history of the wizard through the GetHistory method:

public ICollection GetHistory()

GetHistory returns a collection of WizardStep objects. The order of the items is determined by the order in which the wizard's pages were accessed by the user. The first WizardStep object returned—the one with an index of 0—is the currently selected step if the user post back to the same page. Otherwise, the first WizardStep is the previous step. The second object represents the view before the current one, and so on.

The Wizard Sidebar

The sidebar is an optional wizard element that, if enabled, sits on the left side of the control. It provides the user with a quick-launch user interface for the steps in the wizard. The sidebar is enabled by default and can be hidden with the DisplaySideBar Boolean property set to false. As mentioned, it can be useful to hide the sidebar when using nonlinear navigation. It can also be beneficial when using a commit-as-you-go model.

The layout of the sidebar is only partially customizable. You can use the SideBarTemplate property to define what the contents should be. You can do that either declaratively through the <SideBarTemplate> tag or programmatically by loading an ITemplate object into the SideBarTemplate property. Whichever approach you choose, bear in mind that the template must contain a DataList with a well-known ID and relatively standard internal structure. The DataList must have the ID "SideBarList," and its ItemTemplate block must contain a button object with the ID "SideBarButton." Any other template set on the DataList might cause runtime errors. The IDs are set in stone in the Wizard class through a couple of protected static members—DataListID and SideBarButtonID.

At loading time, the Wizard control populates the sidebar's button bar, which dynamically generates buttons that are each mapped to a defined wizard step. The Wizard seeks the internal DataList through the ID and fills its user interface. In addition to the DataList, any other combination of markup and controls, including images, can be added to the sidebar.

The ASP.NET Wizard control and all of the functionality it brings provides developers with advanced building blocks for feature-rich Web applications.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the newest Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


Service Station: Improving Web Service Interoperability

Service Station
Improving Web Service Interoperability
Aaron Skonnard

Contents
Interoperability Requires Teamwork
Web Services Profiles
Messaging
Service Description
Service Publication, Discovery, and Security
Profile Conformance
Conformance Testing Tools
Sample Applications
Where Are We?

If interoperability is the main promise of Web services, why is it that so many developers and organizations have a difficult time achieving it in practice? With all due respect to our hard-working standards bodies, the primary culprits are the imperfect specifications guiding today's implementations. Ambiguities and too many choices often lead to differing interpretations, resulting in incompatible implementations.

Hence, responsibility lies with the developer for early identification of problem areas that are likely to produce interoperability issues. Once identified, developers should avoid them religiously in their designs. Most developers know that achieving their scalability, reliability, or performance goals depends largely on their work during the design phase. The same holds true for achieving interoperability goals. This month I'll discuss some of these problem areas and how to pursue a design focused on interoperability.

Interoperability Requires Teamwork

Identifying problem areas is a team effort that takes time. It requires the input of numerous developers representing the different vendors involved. It requires thorough interoperability testing. This is where all of the vendors come together in a single room and run a suite of interoperability tests across their implementations. The group of developers document the encountered problems and discuss resolutions. Once they've worked through the interoperability issues at hand, they publish their guidance in what's known as a "profile." Because this type of work requires a significant amount of time, coordination, and political cooperation, a special organization, called the Web Services Interoperability Organization (WS-I), was formed in February 2002. All organizations interested in promoting interoperability are encouraged to join the effort.

The WS-I complements other standards bodies such as the W3C and the IETF. Instead of producing new specifications, the WS-I provides guidance around existing specifications. The WS-I organizes Working Groups (WG) that focus on specific Web services specifications and deliverables. Each WG is chartered to produce a specific deliverable that will assist developers in their quest for interoperability. The types of deliverables produced by the WS-I include profiles, usage scenarios and use cases, sample applications, and testing tools (see Working Group Deliverables for more information on this topic).

Profiles provide the design guidance needed to avoid the most obvious interoperability problems, while the usage scenarios demonstrate how to apply the guidance to your business scenarios. Samples demonstrate how to implement a fully WS-I-compliant application and the testing tools allow you to verify conformance in your own applications. A suite of samples and testing tools will be provided for each WS-I profile once they solidify. When used properly, these collective resources offer helpful insight into your Web services designs.

Web Services Profiles

In many ways, the WS-I is tasked with constraining agreement around different groups of Web services specifications. The WS-I completed their first profile, called Basic Profile (BP) version 1.0, in April 2004. The BP states that conforming Web services use SOAP, Web Services Description Language (WSDL), and Universal Description, Discovery, and Integration (UDDI), and it provides valuable guidance on using them properly.

In conjunction with the Basic Profile guidance, the WS-I has also released the WS-I Usage Scenarios and a complete sample application based on a supply-chain management scenario (complete with use cases, design, and implementation files). In addition, the WS-I has provided a suite of testing tools for verifying BP conformance in your own applications. The work on the Basic Profile has already helped to improve interoperability across the industry's first-generation implementations.

The WS-I is also actively working on additional profiles that focus on higher layers of functionality and different sets of specifications. The WS-I has published a Working Group Draft of the Basic Security Profile 1.0, which focuses on guidance around Web services security. The Basic Security Profile includes coverage of WS-Security, WS-Security UsernameToken Profile, WS-Security X.509 Certificate Token Profile, XML Signature, XML Encryption, and other security-related topics. The WS-I also recently released an Approval Draft for the Attachments Profile 1.0, which provides guidance around using SOAP Messages with Attachments, XML 1.0 + Namespaces, and MIME for attaching non-XML resources to SOAP messages.

These additional profiles (and future ones) will likely take some time to complete; however, it's never too early to begin learning about the interoperability issues addressed by each profile. And once they solidify, the WS-I will provide samples and tools like they did for the BP. Serious Web services architects should frequent the WS-I Web site to internalize the profile guidance as it becomes available.

The guidance provided by the BP is organized into four categories: messaging, service description, service publication and discovery, and security. However, the bulk of the guidance is focused on how to properly design WSDL definitions for service description. Since most toolkits map between WSDL definitions and methods, incompatibilities at this level can (and usually do) cause significant problems. Let's briefly take a look at some of the design guidance provided in each of these areas.

Messaging

The guidance on SOAP messaging clarifies some of the ambiguities that exist in the SOAP and HTTP-related specifications. For example, the guidance states that receivers must be able to process SOAP messages that contain the Unicode byte order mark (BOM), and that all messages must use either the UTF-8 or UTF-16 character encodings. It states that messages should not contain processing instructions or DTDs. And it clarifies how to properly structure the SOAP fault element and use the soap:mustUnderstand and soap:encodingStyle attributes in SOAP messages.

The messaging guidance also clarifies how to use HTTP with SOAP. For example, the guidance states that senders should prefer to use HTTP 1.1, but must use at least HTTP 1.0. It also states that messages must use HTTP POST in all cases and that the SOAPAction header value must always be quoted. And the guidance clearly defines which HTTP status codes to use in different situations (such as always use 500 Internal Server Error for a SOAP fault, and so on).

Most of this guidance is intended for implementing Web services frameworks and isn't something that you typically need to be concerned with in your designs and implementations. This section only provides a few pieces of behavioral guidance that are directly applicable to all Web services developers.

The first one has to do with properly processing mandatory header blocks. According to the profile "this requirement guarantees that no undesirable side effects will occur as a result of noticing a mandatory header block after processing other parts of the message. The Profile requires that receivers generate a Fault when they encounter header blocks that they do not understand targeted at them. When a Fault is generated, no further processing should be performed."

You actually have to write some code in ASMX 1.x in order to adhere to this behavioral requirement because the ASMX framework doesn't check to see if unknown headers were understood until after the method finishes processing. The ASP.NET team wanted to make it possible for your method to walk through the collection of unknown headers and make a decision about "understanding it" at run time.

However, the unfortunate reality of this situation means side effects can occur even when the infrastructure decides to throw a SOAP fault when it determines a header wasn't understood by your code. In order to make your ASMX code compliant with this Basic Profile requirement, you must add code to each of your WebMethods to check for unexpected mandatory header blocks, as illustrated in Figure 1.

Figure 1 CheckUnknownHeaders (Close Figure 1)
public class MyService
{
   public SoapUnknownHeader[] unknownHeaders;

   [WebMethod]
   [SoapHeader("unknownHeaders", Required=false)]
   public string Method1()
   {
      CheckUnknownHeaders();
      ... // normal Web service processing here
   }

   [WebMethod]
   [SoapHeader("unknownHeaders", Required=false)]
   public string Method2()
   {
      CheckUnknownHeaders();
      ... // normal Web service processing here
   }

   public static void CheckUnknownHeaders(
      SoapUnknownHeader[]unknownHeaders)
   {
      if(unknownHeaders != null)
      {
         foreach(SoapUnknownHeader hdr in unknownHeaders)
         {
            if(hdr.MustUnderstand == true)
            {
               hdr.DidUnderstand = false;
               throw new SoapHeaderException(
                 "Header was not understood",
                 SoapException.MustUnderstandFaultCode);
            }
         }
      }
   }
}

Using cookies to implement stateful Web services interactions is another area that wasn't explicitly defined in the original SOAP messaging specifications. The HTTP State Management Mechanism ("Cookies") allows the creation of stateful sessions between Web clients and servers. The BP doesn't disallow cookies, but rather limits their use. The guidance basically states that cookies should not be required for proper operation. Instead, when used, they should serve as a hint that can be used to optimize processing without directly affecting the execution of the Web service. And consumers should not have to understand cookie values—they should only need to pass them along in future requests.

Service Description

The BP states that WSDL must be used to describe Web services. The WSDL specification was originally designed to accommodate a wide variety of scenarios and development styles. WSDL was intentionally designed to be open and flexible with numerous options and choices available to developers. As a result of this general philosophy, WSDL has emerged as the sore point of Web services interoperability. The WS-I BP has addressed this situation by removing many of the choices and reducing the number of things we need to agree on in the specification.

Figure 2 describes the various clarifications made by the BP to the WSDL specification. For example, it clarifies many of the ambiguities revolving around the proper use of WSDL and XML Schema "import" elements and "targetNamespace" attributes. It clarifies how to deal with such things as parameter order, one-way operations, response message wrapper elements, fault descriptions, and message validation.

Figure 2 Basic Profile Service Description Clarifications (Close Figure 2)

Item | Guidance
WSDL and XSD import elements | The WSDL import must be used to import another WSDL description while the XML Schema import statement must be used to import another XML Schema definition. Import statements must provide a location value and must precede other WSDL elements except wsd:documentation.
WSDL and XSD "targetNamespace" attributes | The targetNamespace of an imported description must have the same value as the namespace of the importing description. However, the target namespace for WSDL definitions and the target namespace for schema definitions may be the same.
WSDL type and element attributes | WSDL 1.1 does not clearly state that both type and element attributes cannot be specified to define a wsdl:part in a wsdl:message. A wsdl:message in a description must not specify both type and element attributes on the same wsdl:part.
Parameter order | Permitting the use of parameterOrder helps code generators in mapping between method signatures and messages on the wire. The order of the elements in the soap:body of a message must be the same as that of the wsdl:parts in the wsdl:message that describes it. A description may use the parameterOrder attribute of a wsdl:operation element to indicate the return value and method signatures as a hint to code generators.
One-way operations | There are differing interpretations of how HTTP is to be used when performing one-way operations. For one-way operations, an instance must not return an HTTP response that contains a SOAP envelope. Specifically, the HTTP response entity-body must be empty. A consumer must ignore a SOAP response carried in a response from a one-way operation.
Response message wrapper elements | A message described with an rpc-literal binding that is a response message must have a wrapper element whose name is the corresponding wsdl:operation name suffixed with the string "Response".
Fault descriptions | A Web service description should include all faults known at the time the service is defined. There is also a need to permit generation of new faults that had not been identified when the Web service was defined.
Message validation | When an instance receives a message that does not conform to the WSDL description, a fault should be generated unless the instance takes it upon itself to process the message.

Figure 3 describes all of the BP restrictions imposed on the WSDL specification. For starters, the BP disallows the use of type systems other than XML Schema. It also disallows solicit-response and notification operations, operation name overloading, MIME and HTTP GET/POST bindings, the user of transports other than HTTP, encodings (including the SOAP encoding), mixing styles within a single binding, and requiring custom WSDL extensions. All of these BP restrictions have a recent history of negatively impacting interoperability.

Figure 3 Basic Profile Service Description Restrictions (Close Figure 3)

Item | Guidance
Type systems other than XML Schema | The Profile mandates the use of XML Schema as the type system for WSDL descriptions of Web services.
Solicit-response and notification operations | Not well defined by WSDL 1.1; furthermore, WSDL 1.1 does not define bindings for them. Hence, a description must not use solicit-response and notification type operations in a wsdl:portType definition.
Operation name overloading | Disallowed by the Profile in a wsdl:portType. A wsdl:portType in a description must have operations with distinct values for their name attributes.
MIME and HTTP | Not permitted by the Profile. A wsdl:binding element in a description must use WSDL SOAP.
GET/POST bindings | Binding as defined in WSDL 1.1 Section 3.
Use of transports other than HTTP and HTTPS | The Profile limits the underlying transport protocol to HTTP. A wsdl:binding element in a description must specify the HTTP transport protocol ("http://schemas.xmlsoap.org/soap/http"). Note that this requirement does not prohibit the use of HTTPS.
Encodings (must use document/literal or rpc/literal | The Profile prohibits the use of encodings, including the SOAP encoding. A wsdl:binding in a description must use the value of literal for the use attribute. When not specified, the default for the use attribute must be assumed to be literal.
Mixing styles in a single binding | The style (document or rpc) of an interaction is specified at the wsdl:operation level, permitting wsdl:bindings whose wsdl:operations have different styles. This has led to interoperability problems. A wsdl:binding in a description must use either an rpc-literal binding or a document-literal binding.
WSDL extensions | Requiring support for WSDL extensions that are not explicitly specified by this or another WS-I Profile can lead to interoperability problems with development tools that have not been instrumented to understand those extensions.

As you can see in Figure 3, this guidance can have a significant impact on your WSDL design. However, most Web services frameworks (including ASMX) don't encourage developers to start with WSDL, but rather with classes and methods. Such frameworks typically generate an appropriate WSDL definition from the method signatures and type definitions at run time. Although this approach shields you from WSDL complexities, it also leaves you completely dependent on the framework to get your WSDL design right with respect to BP guidance.

By default, ASMX does a good job of conforming to the BP rules related to WSDL. However, you can easily configure an ASMX service to violate the BP (say, by applying the [SoapRpcService] attribute). The Microsoft® Patterns & Practices group has published "Building Interoperable Web Services: WS-I Basic Profile 1.0", which provides clear guidance on how to apply the various BP rules when implementing and consuming Web services with ASMX.

In general, a WSDL-first development model can increase interoperability since it gives you complete control over WSDL design. Starting with WSDL allows you to validate that you've followed all of the BP WSDL guidance before you generate implementation code. See "Place XML Message Design Ahead of Schema Planning to Improve Web Service Interoperability" in the December 2002 issue of MSDN®Magazine for more details on this approach. Otherwise, you need to intimately understand your tool's support for WSDL and the BP.

Service Publication, Discovery, and Security

When you need to publish and discover Web services, the BP outlines how to use UDDI for this purpose. Support for UDDI is not required to conform with the BP since many services simply don't need the functionality. Figure 4 summarizes the BP's guidance on using UDDI.

Figure 4 Profile Publication and Discovery Clarification (Close Figure 4)

Item | Guidance
tModel descriptions | Registry elements of type uddi:tModel representing a conformant Web service type must use WSDL as the description language and must follow V1.08 of the UDDI Best Practice for Using WSDL in a UDDI Registry. The wsdl:binding referenced by the uddi:tModel must itself conform to the Profile.
tModel categorization | Registry elements of type uddi:tModel representing a conformant Web service type must be categorized using the uddi:types taxonomy and a categorization of "wsdlSpec."
Address descriptions | UDDI V2 provides two alternatives for specifying the network address of instances it represents. The uddi:accessPoint mirrors the WSDL mechanism by directly specifying the address. The uddi:hostingRedirector provides a Web service-based indirection mechanism for resolving the address and is inconsistent with the WSDL mechanism. A uddi:bindingTemplate representing a conformant instance must contain the uddi:accessPoint element.

The BP also provides guidelines for using HTTPS to secure Web services today. It basically states that Web services may require HTTPS at a particular endpoint. With HTTPS, Web service endpoints can guarantee server authentication, integrity, and privacy for all message exchanges. The profile also states that Web services can require client (mutual) authentication. And since HTTPS happens to be a mature and widely available standard, using it to provide basic Web services security features can be a big win for promoting interoperability.

Profile Conformance

When a Web service conforms to the BP, your chances for interoperability go up. So how do you know if a given Web service is conformant? Conformance is really something that has to be determined and asserted by the Web services developer. Testing tools can assist in this determination by validating WSDL designs and runtime SOAP messages. However, the tools cannot validate every BP rule since some of them are related to external specifications or runtime behaviors. This is why the WS-I has not attempted to produce official certification criteria. The testing tools should be viewed as a good indicator of conformance and not an official certification tool.

Now that the Basic Profile is finished, vendors are likely to provide more assistance in the area of conformance. All developers should follow the implementation guidance provided by their vendor surrounding their tools (such as the Microsoft Patterns & Practices document mentioned earlier). ASP.NET 2.0 is going to make BP conformance even easier with its internal understanding of the BP rules. The ASP.NET 2.0 framework provides a ConformanceClaims property on the WebServiceBindingAttribute that influences an endpoint to behave in a way that's conformant with the BP. The following example illustrates how it's used:

[WebServiceBinding(ConformanceClaims = WsiClaims.BP10)]
public class MyService
{
  •••
}

Applying this property also causes the WSDL definition to contain an assertion that the service conforms to the BP. You can still do things in your code to break conformance with the BP, but you'll get runtime exceptions once you interact with the service thanks to its built-in conformance validation. Also, when generating Web services proxies with wsdl.exe, it will check for conformance assertions in the WSDL and complain if it finds anything that violates the Basic Profile.

Conformance Testing Tools

The WS-I Testing Tools provide a great resource to developers designing and implementing Web services. You can download the testing tools from the WS-I site and choose either a C# or Java-language version. Using these tools during design and development can assist in conformance. In order to install the C# version, all you need to do is unzip the files into a directory.

The architecture of the two main tools you'll be using, the monitor and the analyzer, is illustrated in Figure 5. The monitor uses a simple port-forwarding technique to intercept and log SOAP messages traveling between a sender and a receiver. The analyzer is then used to analyze the log file along with the corresponding WSDL definition and UDDI entries. The analyzer produces a report describing BP conformance.

Figure 5 WS-I Testing Tool Architecture
Figure 5 WS-I Testing Tool Architecture

Both the monitor and the analyzer rely on configuration files to specify settings, such as ports and locations. The tools come with sample configuration files but you'll need to modify them to fit your needs. Once you have the configuration files in place, type monitor.exe from a command prompt and it will begin listening for requests on the port specified in monitorConfig.xml (9090 by default). You'll also need to specify where to forward the messages (such as http://localhost:80).

Then you need to update your Web services calling code to send the requests to the specified port (like 9090). Here's an example of doing this in C#:

MathService svc = new MathService();
svc.Url = "http://localhost:9090/myservice/service1.asmx";
double sum = svc.Add(3, 4);
Console.WriteLine("3 + 4 = {0}", sum);

When you run this code, you'll notice the monitor displays information about the SOAP messages it has processed. After you've finished invoking all of the operations you want to monitor and analyze, shut down monitor.exe. At this point, the monitor writes the log file to disk, and then you can send the log file through the analyzer. Before running the analyzer, however, you'll need to modify some of the settings in analyzerConfig.xml to specify the locations of the log file, the WSDL definition, and the UDDI entries (if there are any). When you run analyzer.exe, it will produce a human readable report file.

Figure 6 Analyzer Report Showing General Failure
Figure 6 Analyzer Report Showing General Failure

The report will give you a summary of BP conformance and indicate whether it passed or failed (see Figure 6). It will also give you a thorough breakdown of each BP rule and whether it passed or failed (see Figure 7). In this example, I analyzed a WebMethod that had been configured with the SoapRpcMethodAttribute—this causes it to use an rpc/encoded binding in violation of the BP.

Figure 7 Analyzer Report Showing Specific Failure
Figure 7 Analyzer Report Showing Specific Failure

For a complete walk-through of using these tools in .NET, see "Using the WS-I Test Tools". You may also want to check out the Testing Tools v1.0 User Guide for more details on installation and configuration.

Sample Applications

A few of the deliverables that accompany the BP are part of a complete sample application that illustrates a conforming design and implementation. The WS-I has published a document called "Supply Chain Management Use Cases", which presents a high-level definition of a Supply Chain Management (SCM) application. It has also published a document called "Sample Application Supply Chain Management Architecture", which details the technical design and implementation of the sample application. The architecture includes standardized XML Schema and WSDL definitions that illustrate proper contract design. The sample application was modeled after the SCM use cases and is intended to demonstrate how conformant Web services might be designed, implemented, and deployed.

Numerous vendors (including Microsoft, IBM, BEA, Sun Microsystems, Novell, Oracle, SAP, and others) have provided implementations of this well-defined sample application. They are available for download from the WS-I site. The Microsoft implementation was built using ASP.NET 2.0. Most of the ASMX code was generated from the predefined WSDL definitions provided with the architecture document. The sample illustrates the importance of proper WSDL design and the benefits of using a WSDL-first development approach.

Figure 8 SCM Sample Application Configuration
Figure 8 SCM Sample Application Configuration

Once you download the sample, you'll need to follow the setup instructions in the accompanying document files to configure the necessary virtual directories and SQL Server™ databases. When you first run the sample, you'll need to configure it and specify which SCM endpoints you'd like to use (see Figure 8). You can use the Microsoft endpoints that come with the sample or endpoints provided by other vendors, or even your own. This allows you to experiment with interoperability in a real-world business scenario. Once configured, you can place orders that will be processed through a supply chain consisting of several warehouses and you can track the order as illustrated in Figure 9.

Figure 9 SCM Sample Application Tracking Orders
Figure 9 SCM Sample Application Tracking Orders

If you're designing Web services for your organization, you should definitely take some time to download the test tools along with the WS-I sample application. Check out the various WSDL definitions and walk through the code. Doing so will show you by example how to apply the BP guidance discussed in this column.

Where Are We?

Interoperability is not a given when using Web services. It requires research, thought, and a good understanding of the issues surrounding the various Web services specifications. Luckily, the WS-I does the heavy lifting and provides summarized guidance in the form of profiles, samples, and testing tools. Taking advantage of these resources during design and implementation will greatly improve your interoperability success, which ultimately saves you time and money during integration.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, an education and content creation company, where he focuses on XML and Web services technologies. He's a contributing editor to MSDN Magazine and the author of several books, including Essential XML Quick Reference (Addison-Wesley, 2001). Reach him at http://www.pluralsight.com/aaron.


.NET Matters: ThreadPoolPriority, and MethodImplAttribute

.NET Matters
ThreadPoolPriority, and MethodImplAttribute
Stephen Toub

Code download available at: NETMatters0411.exe (148KB)

Q I'm writing an app that queues thousands of operations to the ThreadPool during a run. Some operations are more important than others, and I want them to have a higher priority. In other words, if there are any high-priority operations waiting to be run, I want the ThreadPool to pick up and run those first, regardless of when they were queued. How can I accomplish this?

A The heart of your problem lies in the fact that, while amazingly useful, the ThreadPool is a fairly inextensible class. There is no way to change or influence the internal data structures used to store the queued WaitCallbacks and associated state (the ThreadPool internally uses a queue implemented as a linked list, a fact you can verify by examining the source for ThreadPool in the Shared Source Common Language Infrastructure (SSCLI)). Thus, one way to accomplish your request is to maintain a data structure yourself and force the ThreadPool to ask your data structure for the next WaitCallback to execute.

In the October 2004 edition of the .NET Matters column, I demonstrated how to create a ThreadPoolWait class that provided a mechanism for synchronizing multiple threads from the ThreadPool. To do so, ThreadPoolWait exposed a QueueUserWorkItem method that relegated to ThreadPool.QueueUserWorkItem. But instead of passing along the user-supplied WaitCallback as the callback parameter to ThreadPool.QueueUserWorkItem, it created a new WaitCallback that wrapped its own private handler and allowed ThreadPoolWait to run custom code as soon as the user-supplied callback had completed. You can use a similar kind of mechanism to influence the order in which operations are handled by the ThreadPool.

Figure 1 ThreadPoolPriority Interacting with ThreadPool
Figure 1 ThreadPoolPriority Interacting with ThreadPool

A diagram for a solution is shown in Figure 1 with the corresponding code shown in Figure 2. Just as with ThreadPoolWait, ThreadPoolPriority has a static QueueUserWorkItem method. However, in addition to taking as parameters the WaitCallback to be executed and the associated object state, it also takes a Boolean value that signals whether this callback should be treated as a priority. Rather than immediately passing the WaitCallback along to ThreadPool.QueueUserWorkItem, the method captures the callback and the associated state object into a temporary state class and then appends that new object to an internal queue. As you'll notice though, there are multiple queues. If the priority Boolean is set to true, the object is enqueued onto the priority queue, whereas if it's false it's enqueued onto the standard queue. Then, instead of calling ThreadPool.QueueUserWorkItem with the original WaitCallback, a new WaitCallback is created that wraps the HandleWorkItem static method, and that WaitCallback is passed. Note, too, that no state is explicitly associated with this callback.

Figure 2 ThreadPoolPriority (Close Figure 2)
public class ThreadPoolPriority {
    private ThreadPoolPriority(){}

    private static Queue _priorityQueue = new Queue();
    private static Queue _standardQueue = new Queue();

    public static void QueueUserWorkItem(
            WaitCallback callback, object state, bool isPriority) {
        if (callback == null) throw new ArgumentNullException("callback");
        new PermissionSet(PermissionState.Unrestricted).Demand();

        QueuedCallback qc = new QueuedCallback();
        qc.Callback = callback;
        qc.State = state;

        lock(_priorityQueue) {
            (isPriority ? _priorityQueue : _standardQueue).Enqueue(qc);
        }

        ThreadPool.UnsafeQueueUserWorkItem(
            new WaitCallback(HandleWorkItem), null);
    }

    private static void HandleWorkItem(object ignored) {
        QueuedCallback qc;
        lock(_priorityQueue) {
            qc = (QueuedCallback)(_priorityQueue.Count > 0 ?
                _priorityQueue : _standardQueue).Dequeue();
        }
        qc.Callback(qc.State);
    }

    private class QueuedCallback {
        public WaitCallback Callback;
        public object State;
    }
}

When invoked, HandleWorkItem checks to see whether any callbacks are queued on the priority queue. If so, the first is removed and that one is invoked. Otherwise, an object from the standard queue is dequeued and invoked. Unless something goes drastically wrong, there should always be an object in one of the queues. All access to both queues is protected by a lock in order to prevent race conditions and other multithreading perils.

Of course, my use of two Queues as the data structure is just an example. You can use any data structure that suits your situation. For example, if you want last in first out (LIFO) semantics, you can substitute a single Stack. Or if you wanted more levels of priority, you can implement an actual priority queue data structure (often based on a binary heap). Also, keep in mind that just because a work item begins executing first, there's no guarantee it'll finish first. As an enhancement, you could modify HandleWorkItem to change the current thread's scheduling priority (available through Thread.CurrentThread.Priority) according to the priority of the queued work item.

One more thing you might question is the use of the security demand in the QueueUserWorkItem method:

new PermissionSet(PermissionState.Unrestricted).Demand();
This has the unfortunate consequence of limiting the usability of the solution to those callers with full trust, but I'll explain why it's necessary. When a method calls ThreadPool.QueueUserWorkItem, information about the current call stack (in the form of a CompressedStack) is stored in the ThreadPool's queue along with the WaitCallback and its associated state. When the WaitCallback is executed on one of the ThreadPool's threads, that CompressedStack is pushed onto the executing Thread. Since Code Access Security (CAS) bases its permission checks on the permissions of all the callers on the stack, without this process in place to transfer information about the stack, you could accidentally (or maliciously) open up a security hole. The ThreadPool exposes another public member, UnsafeQueueUserWorkItem, that does not perform this call stack transfer.

So why is my code using UnsafeQueueUserWorkItem? By not invoking the queued WaitCallbacks in the order in which they were queued, I'm inadvertently disassociating the user-supplied WaitCallback from its CompressedStack. Thus, the permissions for the callers associated with the executing thread may have nothing to do with the callback being invoked. The best solution in my class would be to manually store the CompressedStack with the user-supplied WaitCallback and then set this CompressedStack on the ThreadPool's worker thread used to execute the callback.

While the Thread class exposes two methods, GetCompressedStack and SetCompressedStack, that could be used for this very purpose, their use is restricted to libraries signed with the Microsoft strong name, and thus are unavailable. As one of the few valid solutions left to me, my QueueUserWorkItem method demands that any caller attempting to queue a work item must already have full trust. That way, no direct caller could maliciously take advantage of this shortcoming. Given that full trust is required and that the CompressedStack generated for the executed method could be erroneous, I opted to avoid any correlated issues by then using ThreadPool.UnsafeQueueUserWorkItem (which should also be slightly faster given that it doesn't have to capture and store the stack).

The situation changes a bit with the Microsoft® .NET Framework 2.0, currently at Beta 1. In the .NET Framework 1.x, the current call stack is transferred as a CompressedStack to threads used from the ThreadPool and to new threads when they are spun up. However, other important context such as the Thread.CurrentPrincipal is not transferred. In the .NET Framework 2.0, not only is the CAS-context flowed across asynchronous points, but additional execution context is flowed as well, including impersonation information, call context, and HTTP context. As such, with the solution outlined in Figure 2 running on the .NET Framework 2.0, the disconnect between the queued delegate and the context is associated with the work item is worsened.

Fortunately, the .NET Framework team has provided an elegant solution for these types of situations. The new System.Threading.ExecutionContext class contains tons of contextual information about a particular thread and environment, including but not limited to the call stack, the security context, the call context, and the synchronization context. It provides a static Capture method that can be used as a factory to create an ExecutionContext for the current thread. The ExecutionContext.Run static method can then be used with a captured ExecutionContext to execute a ContextCallback delegate under that execution context. Using this, I can rewrite the solution in Figure 2 to better target the .NET Framework 2.0. This new solution is shown in Figure 3. (For more information on security and asynchronous invocation, see Chris Brumme's excellent blog post on the subject at Security & Asynchrony.)

Figure 3 ThreadPoolPriority for the .NET Framework 2.0 (Close Figure 3)
public static class ThreadPoolPriority {
    private static Queue<QueuedCallback>
        _priorityQueue = new Queue<QueuedCallback>(),
        _standardQueue = new Queue<QueuedCallback>();

    public static void QueueUserWorkItem(
            WaitCallback callback, object state, bool isPriority) {
        if (callback == null)
            throw new ArgumentNullException("callback");

        QueuedCallback qc = new QueuedCallback();
        qc.Callback = new ContextCallback(callback);
        qc.State = state;
        qc.ExecutionContext = ExecutionContext.Capture();

        lock(_priorityQueue) {
            (isPriority ? _priorityQueue : _standardQueue).Enqueue(qc);
        }

        ThreadPool.UnsafeQueueUserWorkItem(HandleWorkItem, null);
    }

    private static void HandleWorkItem(object ignored) {
        QueuedCallback qc;
        lock(_priorityQueue)
        {
            qc = (_priorityQueue.Count > 0 ?
                _priorityQueue : _standardQueue).Dequeue();
        }
        ExecutionContext.Run(qc.ExecutionContext, qc.Callback, qc.State);
    }

    private struct QueuedCallback {
        public ContextCallback Callback;
        public object State;
        public ExecutionContext ExecutionContext;
    }
}

There are a few important points to discuss about this new version. First, in the previous version of ThreadPoolPriority, QueuedCallback's Callback field stored a WaitCallback. In the new version, Callback is a ContextCallback since that is the type ExecutionContext.Run expects as its first argument. However, the public QueueUserWorkItem signature has not changed and still accepts a WaitCallback. Coercing a WaitCallback into a ContextCallback requires a bit of indirection:

qc.Callback = new ContextCallback(callback);
ContextCallback has the same delegate signature as WaitCallback, and thus I can create a ContextCallback that targets the WaitCallback's Invoke method. A call to qc.Callback.Invoke will then result in the execution of callback.Invoke, which is exactly what I want. Of course, in C# a delegate's Invoke method is implied when using this syntax. In other languages, I might have to explicitly specify which method is the target for the ContextCallback delegate. For example, in Visual Basic® the same code would look like:
qc.Callback = New ContextCallback(AddressOf callback.Invoke)
While I'm a big fan of C#, I actually prefer the explicitness of this Visual Basic statement to the previously shown C#.

In addition to the change from WaitCallback to ContextCallback, you'll notice that in a few places I've taken advantage of some of the new C# 2.0 features. Rather than adding a private constructor to the ThreadPoolPriority class in order to prevent external code from instantiating it, I've modified its signature to include the static keyword. This results in the class being compiled to Microsoft intermediate language (MSIL) as:

.class public abstract auto ansi sealed beforefieldinit ThreadPoolPriority
     extends [mscorlib]System.Object
Since the class is abstract, it can't be instantiated, and as it's sealed, it can't be derived from. Thus, it's impossible to ever have an instance of this class created and can only be used for its static members.

Also worth noting is that I'm taking advantage of generics. Rather than using System.Collections.Queue as I did in Figure 2, I'm using System.Collections.Generics.Queue<QueuedCallback>. This strongly types the Queue so that it can only be used with QueuedCallback structures, improving performance and eliminating the need for me to cast when using Dequeue. I'm also taking advantage of the C# compiler's ability to infer delegate types, so rather than having to write

ThreadPool.UnsafeQueueUserWorkItem(
    new WaitCallback(HandleWorkItem), null);
I can simply write the following and the compiler generates the equivalent MSIL:
ThreadPool.UnsafeQueueUserWorkItem(HandleWorkItem, null);
Had I wanted to be even fancier, I could have implemented HandleWorkItem inline with QueueUserWorkItem as an anonymous delegate. For more information on the new features in C# 2.0, see Juval Lowy's article in the May 2004 issue of MSDN®Magazine, available at Create Elegant Code with Anonymous Methods, Iterators, and Partial Classes.

Q I've been trying to determine how Guid.NewGuid generates the new GUID. Looking at the code for Guid in the SSCLI and then later using Lutz Roeder's .NET Reflector, I came across a method called by Guid's constructor that looks like the following:

[MethodImpl(MethodImplOptions.InternalCall)]
private extern bool CompleteGuid();
The method body is missing, and this odd attribute is applied to the method signature. What does that mean, and where can I find the actual implementation of CompleteGuid?

A There are actually a few questions here, namely what is the MethodImplAttribute, what does it mean for something to be an InternalCall, as a result where can I find the implementation of CompleteGuid, and how does CompleteGuid work? To answer the first question, MethodImplAttribute is what's known as a pseudo-attribute. Unlike custom attributes, pseudo-attributes are not stored as attributes in the metadata; rather, they're used to set bits or fields within the metadata tables. In other words, rather than extending the metadata for a type and being saved along with that type, they actually modify the metadata for a type and are then discarded. For example, the SerializableAttribute is actually a pseudo-attribute that, when applied to a class, sets the CorTypeAttr.tdSerializable metadata bit, while the ComImportAttribute sets the CorTypeAttr.tdImport metadata bit. Similarly, rather than retrieving these attributes from a class using Type.GetCustomAttributes, they're retrieved using Type.Attributes.Serializable and Type.Attributes.Import, respectively. MethodImplAttribute sets the CorMethodImpl metadata bits according to the options specified in its constructor and can be retrieved using MethodBase.GetMethodImplementationFlags.

MethodImplAttribute uses the MethodImplOptions to configure details about the corresponding method implementation. For example, if there is a method that for some reason you want to ensure is never inlined by the just-in-time (JIT) compiler, you can attribute that method with [MethodImpl(MethodImplOptions.NoInlining)]. If you want to specify that the entire method should be single threaded, meaning that only one thread will be able to run that method at a time, you can attribute it with [MethodImpl(MethodImplOptions.Synchronized)]. In fact, with the current implementation of the JIT, on an instance method this is equivalent to surrounding the contents of the method with a lock(this) block. This implies that if multiple instance methods in the same class are attributed with this implementation option, all of those methods will be synchronized around the same sync block (a sync block can be associated with any object and is lazily created for that object the first time a lock is taken on it). As a result, you should shy away from using MethodImplOptions.Synchronized for the same reasons you wouldn't use lock(this). For more information, see Rico Mariani's blog post at [MethodImpl(MethodImplOptions.Synchronized)] promised followup.

The extern modifier in C# informs the C# compiler that a method is implemented externally to the class, though it doesn't tell it where it's implemented. You'll often see this keyword used in conjunction with the DllImport attribute, which informs the runtime that the implementation is actually part of an unmanaged DLL. In contrast, MethodImplOptions.InternalCall is used in conjunction with extern to tell the runtime that the method is implemented internally within the system itself. This is done for many of the core .NET Framework methods that are better served by being implemented in unmanaged code. For example, many of the methods on the String, GC, and Math classes are marked as InternalCall. As you've noticed, Guid.CompleteGuid is also an InternalCall.

So how do we find the implementation? The ecall.cpp C++ file (which is located at \clr\vm\ecall.cpp in the SSCLI) contains a table that maps managed internal call methods to unmanaged C++ implementations. Search ecall.cpp for "CompleteGuid" and you'll find the following code:

static ECFunc gGuidFuncs[] = {
    {FCFuncElement("CompleteGuid", NULL,
        (LPVOID)GuidNative::CompleteGuid)}, {NULL, NULL, NULL}
};

This tells the runtime that the implementation for the managed Guid.CompleteGuid method is actually the native C++ GuidNative::CompleteGuid function. And where is GuidNative::CompleteGuid implemented? It can be found in the comutilnative.cpp file, also located in the \clr\vm\ folder. A cursory examination of this method shows that it simply uses CoCreateGuid to create an unmanaged GUID and then uses memcpy to blit the unmanaged GUID into the managed Guid.

So in the end, Guid.NewGuid relies on CoCreateGuid (and, in turn, on UuidCreate) to generate the globally unique 128-bit identifiers we all know and love. Simplicity at its best.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


C++ Q&amp;A: Calling Virtual Functions, Persisting View State, POD Type

C++ Q&A
Calling Virtual Functions, Persisting View State, POD Type
Paul DiLascia

Code download available at: CQA0411.exe (189KB)

Q In C++, you can't successfully call a derived virtual function from a constructor because the vtable isn't fully set up. But it appears you can do this in C#. Is this correct? Why is C# different?

Clifton F. Vaughn

A Indeed, C# behaves differently from C++ in this respect. In C++, if you call a virtual function from a constructor or destructor, the compiler calls the instance of the virtual function defined for the class being constructed (for example, Base::SomeVirtFn if called from Base::Base), not the most derived instance. As you say, this is because the vtable is not fully initialized until the most derived constructor executes. Another way to think of it is that the derived class is not created yet.

Figure 2 Virtual Function Test
Figure 2 Virtual Function Test

Similarly, when you call a virtual function from a destructor, C++ calls the base class function because the derived class has already been destroyed (its destructor has been called). While this behavior can lead to unexpected results (which is why it's considered bad programming practice to call a virtual function from a constructor or destructor), it is at least familiar to most C++ programmers and makes some sense.

As you point out, things are different in C#. Managed objects—whether in C#, managed C++, or any other .NET-compliant language—are created as their final type, which means that if you call a virtual function from a constructor or destructor, the system calls the most derived function. Figure 1 shows a program that illustrates this. If you compile and run it, you'll see the output shown in Figure 2.

Figure 1 VirtFnTest (Close Figure 1)
////////////////////////////////////////////////////////////////
// MSDN Magazine — November 2004

// To compile:
//   cl /clr VirtFnTest.cpp
//
#include <stdio.h>
#include <vcclr.h>
#using <mscorlib.dll>

using namespace System;

/////////////////
// C++ native base class. Ctor calls virtual fn: always calls Base fn.
class Base {
public:
   Base()
   {
      printf(" Base::ctor\n");
      Hello();
   }
   ~Base()
   {
      printf(" Base::dtor\n");
      Goodbye();
   }
   virtual void Hello()   { printf(" Base::Hello\n"); }
   virtual void Goodbye() { printf(" Base::Goodbye\n"); }
};

// Derived native C++ class overrides fn, but not called from ctor.
class Derived : public Base {
public:
   Derived()
   {
      printf(" Derived::ctor\n");
   }
   ~Derived()
   {
      printf(" Derived::dtor\n");
   }
   virtual void Hello()   { printf(" Derived::Hello\n"); }
   virtual void Goodbye() { printf(" Derived::Goodbye\n"); }
};

// Managed base class. Ctor calls virtual fn: always calls most derived
// override.
public __gc class MBase {
public:
   MBase() {
      printf(" Base::ctor\n");
      Hello();
   }
   ~MBase() {
      printf(" Base::dtor\n");
      Goodbye();
   }
   virtual void Hello()   { printf(" Base::Hello\n"); }
   virtual void Goodbye() { printf(" Base::Goodbye\n"); }
};

// Managed derived class overrides virtual fn.
public __gc class MDerived : public MBase {
public:
   MDerived()
   {
      printf(" Derived::ctor\n");
   }
   ~MDerived()
   {
      printf(" Derived::dtor\n");
   }
   virtual void Hello()   { printf(" Derived::Hello\n"); }
   virtual void Goodbye() { printf(" Derived::Goodbye\n"); }
};

int main()
{
   printf("Create native object:\n");
   Derived *pd = new Derived();
   printf("Destroy native object:\n");
   delete pd;

   printf("\nCreate managed object:\n");
   MDerived *pmd = new MDerived();
   printf("Destroy managed object:\n");
   delete pmd; // force dtor call
   return 0;
}

This behavior may seem bizarre to C++ programmers. It means you can call a virtual function for a derived type before the derived type has been initialized—that is, before its constructor has run. Likewise, if you call a virtual function from the base destructor, the function runs after the derived class has been destroyed—that is, after the destructor is called. So once again—but for different reasons—it's considered bad practice to call a virtual function from a constructor or destructor.

Why did the Redmontonians choose to design C# this way? Because it simplifies memory management. The garbage collector needs to know how big an object is in order to free it. If C# constructed objects the way C++ does, you could end up with a situation in which you have two objects, Obj1 and Obj2, where these two statements are both true

typeof(Obj1)==typeof(Obj2)
sizeof(Obj1)!= sizeof(Obj2)
because one of the objects is partially constructed. (Don't forget the garbage collector runs asynchronously.) By constructing objects as their final type, the garbage collector can determine an object's size from its type. If C# did partial construction à la C++, the garbage collector would need more code to determine the true size of a partially constructed object. This would add complexity and slow things down, all to handle a situation that's discouraged in the first place. So in the interest of a faster garbage collector, the Redmondtonians decided to implement C# as described. For a discussion of this, see Raymond Chen's blog, "The Old New Thing".

Q In your March 2004 column, you showed how to change the last view setting in the file open dialog, but didn't cover the part about saving the last view used by the user. The problem I encountered is how to read the setting the user has made in the file open dialog. The only way I found was reading the list control directly, but that doesn't give me the correct information when the user selected thumbnail view. Do you have any solution for this?

Maarten van Dillen

Q I'm trying to do something with the common CFileDialog class that should be trivial, but it does not seem to be. I want to force the view mode in the file open dialog box to be thumbnail view. Can you suggest a way to do this Visual C++®?

Elliot Leonard

A Several readers have asked questions about thumbnail view in the file open dialog. In my March column, I showed how to send WM_COMMAND messages to the special SHELLDLL_DefView window in the file open dialog to set various view modes—but how can you tell which mode the view is currently in? To discover the view mode, you have to get the list control and call CListCtrl::GetView:

// in dialog class
HWND hlc = ::FindWindowEx(m_hWnd,
  NULL, _T("SysListView32"), NULL);
CListCtrl* plc =  (CListCtrl*)CWnd::FromHandle(hlc);
DWORD dwView = plc->GetView();
CListCtrl::GetView returns one of the LV_XXX codes, but as Maarten discovered, Windows® returns LV_VIEW_ICON for both icon and thumbnail views.

So how can you distinguish between them? I scratched my head pondering this one and spent a while poking through the header files until I discovered a message called LVM_GETITEMSPACING that fetches the—what else?—icon spacing. As the name suggests, the icon spacing is the number of pixels between icons in icon view. LVM_GETITEMSPACING is so obscure that MFC doesn't have a wrapper for it (there's no CListCtrl::GetIconSpacing). So in MFC, you have to send the message yourself:

CSize sz = CSize(plc->SendMessage(LVM_GETITEMSPACING));

Windows returns the size the usual way, as a long with cx/cy encoded in the high- and low-order words, which CSize politely decodes for you. Once you have the icon spacing, you can compare it to the system spacing returned from GetSystemMetrics(SM_CXICONSPACING). You mean that wasn't obvious? If the list view's icon spacing is the same as the system spacing, the list view is in icon mode. If it's bigger, the view's in thumbnail mode:

if (sz.cx > GetSystemMetrics(SM_CXICONSPACING)) {
  // thumbnail view
} else {
  // icon view
}

So much for thumbnail view. Next, how do you persist the view state across user sessions? For that, you need the Profile functions to save the last-used mode in the user's profile when your program terminates and restore it again the next time it starts up. I wrote a little program, DlgTest, that shows how (see the code download at the MSDN®Magazine Web site). It has a class CPersistOpenDlg that implements the persistent behavior. CPersistOpenDlg uses another class, CListViewShellWnd, to encapsulate the SHELLDLL_DefView window (see my March 2004 column). CListViewShellWnd has functions to set and get the view mode; these functions differentiate icon from thumbnail view:

CListViewShellWnd m_wndLVSW;
•••
m_wndLVSW.SetViewMode(ODM_VIEW_THUMBS);

CListViewShellWnd's OnDestroy handler saves the view mode in a data member, m_lastViewMode. CPersistOpenDlg's destructor calls WriteProfileInt to write this value to the user's profile as the dialog is destroyed. When the dialog starts up, CPersistOpenDlg posts an initialization message to itself; the handler for this message calls GetProfileInt to read the stored value from disk and set the view mode. PostMessage is required because the normal initialization messages, WM_INITDIALOG and CDN_INITDONE, come before the file open dialog is fully initialized—for an explanation of this, please read my March 2004 column.

By the way, you should use GetProfileXxx and WriteProfileXxx any time you want to persist your application settings across program invocations. MFC has CWinApp wrappers for these API functions. If you call CMyApp::SetRegistryKey("KeyName") when your application starts up (typically in your InitInstance function), MFC uses the registry for profile settings instead of an INI file. Here's the INI file for DlgTest:

[settings]
ViewMode=28717

Q I've seen the term "POD type" occasionally in literature and documentation about C++ and the Microsoft® .NET Framework. What is a POD type?

Shelby Nagwitz

A You might think a POD type is a data type that arrives from outer space wrapped in a green protective covering, but POD stands for Plain Old Data and that's just what a POD type is. The exact definition is rather gnarly (see the C++ ISO standard), but the basic idea is that POD types contain primitive data compatible with C. For example, structs and ints are POD types, but a class with a user-defined constructor or virtual function is not. POD types have no virtual functions, base classes, user-defined constructors, copy constructors, assignment operator, or destructor.

To conceptualize POD types you can copy them by copying their bits. Also, POD types can be uninitialized. For example:

struct RECT r; // value undefined
POINT *ppoints = new POINT[100]; // ditto
CString s; // calls ctor ==> not POD

Non-POD types usually require initialization, whether by calling a default (compiler-supplied) constructor or your own.

In the old days, POD was important mostly for people writing compilers or C++ programs compatible with C. Nowadays, POD comes up in the context of .NET. In managed C++, managed types (both __value and __gc) can contain embedded native POD types. Figure 3 shows code that illustrates this. The managed Circle class can contain a POINT but not a CPoint. If you try to compile pod.cpp, you'll get error C3633: "Cannot define 'm_center' as a member of managed 'Circle' because of the presence of default constructor 'CPoint::CPoint' on class 'CPoint'."

Figure 3 Pod.cpp (Close Figure 3)
/////////////////////////////////////////////////////////////////////
// You can embed native POD types in a managed (__gc or __value) type.
// Compile with cl /clr.

#using <mscorlib.dll>

// simple struct is a POD type
struct POINT {
   int x;
   int y;
};

// native class w/ctor: not POD!
class CPoint : public POINT {
   CPoint() { x=y=0; }
};

// Managed __gc type
__gc class Circle {
public:
   POINT center;        // ok: embedded POD type
   CPoint m_center;     // Error: not POD!
   int radius;
};

The reason .NET restricts embedded native objects to POD types is so that it can safely copy them without worrying about calling constructors, initializing vtables, or any of the other machinations that non-POD types require.

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


{ End Bracket }: A Tidal Wave of Change

{ End Bracket }
A Tidal Wave of Change
Larry Osterman

I surf the Net somewhat obsessively when I have spare time, and I often read various discussion forums such as Slashdot and Neowin. Now, the journalism at those sites isn't always the highest quality, and I have noticed a marked lack of perspective on the level of changes that have occurred in this industry. Working at Microsoft over the past 20 years has given me a unique point of view on the magnitude of these changes. Many people are unaware of what the computing universe was like in the dark ages.

When I started working in the industry, computing was dominated by big iron machines. Mini-computers had made some impact, but mostly only at universities. The personal computer was a toy; the floppy drive on the original IBM PC was only an option—you could buy them with just a cassette interface.

In 1984, when I started at Microsoft, PCs had a 4.77MHz processor. Most of them came with 256KB of RAM and a 10MB standard hard disk. It was a huge improvement when the IBM PC/AT came out with a 20MB hard disk. I remember clearly a developer in the Xenix division running down the hall with a rectangular object the size of a shoebox exclaiming that it was a 70MB hard disk! Nobody had ever seen anything with that kind of capacity before, especially nothing that big for a PC. The highest quality PC display that you could buy in 1984 (the IBM EGA) offered 16 color images at 640¥350 resolution. While Microsoft and a couple of other companies made mice, having a mouse on your computer meant adding an adapter board to support it.

Corporate networking was in its infancy, and the modern Internet had literally just been created. They deployed DNS at Carnegie-Mellon shortly after I left; before that, the Internet used static host tables. Microsoft didn't have a corporate network; there was an ad hoc network between the offices of the people who were working on the networking support in MS-DOS® 3.1, but that was about it. Since PCs didn't have the horsepower to do anything, we built our product on Sun mini-computers (running Xenix, for example) and downloaded the images onto our machines to test.

When Windows® debuted in 1985, it was the start of the first of the tidal waves that rocked the PC industry (if you include all PC-class machines, the tidal wave started when the first Macintosh shipped in January of 1984). With Windows, the computing paradigm shifted from text-based to graphics-based. Mice were required equipment rather than options, and computer manufacturers started including them in their systems by default.

Even so, systems were still primitive. A fast computer in 1990 was a 66MHz 486 machine. In 1995, the machine I used for development was a 120MHz Pentium, with 4MB of RAM, and that was considered a beefy machine at the time. If I remember, it came with a 200MB hard disk, which was huge. Later in 1995, the second of the tidal waves hit—the Internet. In six months the number of active Web sites quadrupled.

The Internet didn't just change the computing industry, it changed the world. Now every machine was connected to every other machine. Suddenly, everyone and anyone could publish data. The world's largest reference library became available to all. Now you can find out anything with very little effort, but of course you still have to use your good sense when evaluating this vast information. For example, http://www.buydehydratedwater.com is a wonderful site that tells you of the evils of tap and well water and will quite happily sell you cartons of dehydrated water.

The Internet in its current form has barely existed for 10 years. People forget that it's only been in the past few years that computers have had enough horsepower to be able to render 3D graphics. They forget that it's only been in the past three years that digital photography has been a possibility for nonprofessionals. They forget that just eight years ago, CD-ROM drives were optional features on new computers. They forget that the DVD has only been around for 10 years. They forget that MP3s and ripped music have only been around for about eight years. And they forget that although CDs have only existed for 20 years, in that time discs have managed to practically eliminate all the storage forms used in the 80-plus years of recorded music. They forget that it's only been in the past five or so years that hard disks have been cheap enough to even consider keeping large libraries of audio files.

There are times that I sit back and marvel at all the changes that have taken place in the industry since I started at Microsoft, and there are times when I wonder how we got by before. And then I try to imagine what will happen over the next 20 years.

Larry Osterman is a Software Engineer at Microsoft. During his 20-year career at Microsoft he has worked on a wide variety of products, including MS-DOS and Microsoft Exchange. He is currently working in the Windows Multimedia division.