 MSDN Magazine November 2006

MSDN Magazine November 2006

Secure Habits: 8 Simple Rules For Developing More Secure Code

Never trust data, model threats against your code, and other good advice from a security expert.

Michael Howard Threat Modeling: Uncover Security Design Flaws Using The STRIDE Approach

Whenever you build a new system you should consider how an in¬truder might go about attacking it and then build in appropriate defenses at design time.

Shawn Hernan, Scott Lambert, Tomasz Ostwald, Adam Shostack Single Sign-On: A Developer's Introduction To Active Directory Federation Services

Use Active Directory Federation Services to allow other organizations to use your Web applications without the need for you to grant access explicitly.

Keith Brown Smart Storage: Protect Your Data Via Managed Code And The Windows Vista Smart Card APIs

Smart cards are a compelling alternative to the reliance on passwords, which are the weakest link in authentication systems. Get the Windows smart card programming basics here.

Dan Griffin Extending SDL: Documenting And Evaluating The Security Guarantees Of Your Apps

In this article, the author presents an extension to the Security Development Lifecycle Which could promote a better flow of information between users and designers of software security features.

Mark Novak SQL Security: New SQL Truncation Attacks And How To Avoid Them

Exploits using SQL injection have drawn a lot of attention for their ability to get through firewalls and intrusion detection systems to compromise your data layers. Whether it's a first-order or second-order injection, if you look at the basic code pattern, it is similar to any other injection issue where you use untrusted data in the construction of a statement.

Bala Neerumalla Editor's Note: Healthy Printing

Why we use the paper we do.

Joshua Trupin Toolbox: Synchronize Files, Rich Textboxes, and More

Compare files and folders, create demo and support videos, add rich textboxes in your web apps, and more.

Scott Mitchell Basic Instincts: Server-Side Generation of Word 2007 Docs

This month, Office Open XML, which allows ASP.NET and SharePoint developers to read, write, and generate Word, Excel, and PowerPoint documents on the server without running an Office desktop application there.

Ted Pattison Test Run: Using Excel For Test Data

This month see how to use Excel for test automation storage, whether you’re just starting out with NET, or you’re an advanced programmer.

Dr. James McCaffrey Data Points: Revisiting System.Transactions

The System.Transactions namespace of the Microsoft .NET Framework makes handling transactions much simpler than previous techniques. Read all about it this month.

John Papa CLR Inside Out: Investigating Memory Issues

Memory issues can manifest in a wide variety of ways. This column shows you how to collect the data you need to determine what types of mem¬ory issues you are experiencing.

Claudio Caldato and Maoni Stephens Cutting Edge: A Tour of Windows Workflow Activities

Windows Workflow Foundation supports virtually any scenario where human opera¬tors are involved. Learn how to use it to tame your workflows.

Dino Esposito Bugslayer: Minidumps for Specific Exceptions

This installment of Bugslayer covers the use of ADPlus to create a minidump of your Microsoft .NET Framework 2.0 pro¬cesses on specific exceptions.

John Robbins Security Briefs: Limited User Problems and Split Knowledge

Keith Brown Concurrent Affairs: The ReaderWriterGate Lock

Jeffrey Richter .NET Matters: Event Accessors

Creating events on classes by adding the event keyword to a delegate member variable declaration.

Stephen Toub Netting C++: Introducing Regular Expressions

This month Stanley Lippman introduces the support for regular expressions in the .NET Framework.

Stanley B. Lippman { End Bracket }: Peripheral and Foveal Vision.

Considering human visual fields in software design.

Bill Hill


Secure Habits: 8 Simple Rules For Developing More Secure Code

Secure Habits

8 Simple Rules For Developing More Secure Code

Michael Howard
This article discusses:
  • Using analysis tools and experts to review your code
  • Reducing risk using fuzzing and threat modeling
  • Keeping bad input out of your applications
  • Learning all you can about security concepts
Contents
Habit #1: Take Responsibility
Habit #2: Never Trust Data
Habit #3: Model Threats against Your Code
Habit #4: Stay One Step Ahead
Habit #5: Fuzz!
Habit #6: Don't Write Insecure Code
Habit #7: Recognize the Strategic Asymmetry
Habit #8: Use the Best Tools You Can

I
have been lucky enough to work with thousands of good developers over the years who wanted to learn how to write more secure software. During this time, I've also learned a great deal from people who are very good at building secure systems, and this got me thinking. I wondered if there were common skills or habits the "secure developers" share. It turns out the answer is a resounding yes! This article presents that list of habits shared by developers of secure code.

Now one thing I know for sure is that anyone reviewing this list will immediately see missing habits. That's fine. I know there are other great ideas out there. This is my list! So, with that out of the way, here are the exemplary habits I've noticed over the years.

Habit #1: Take Responsibility

This is a variation of the classic "There is no silver bullet" comment made over 25 years ago by Fred Brookes in The Mythical Man Month. Getting security right in your product is totally up to you. No one else—and certainly no magic tool or programming language—solves all security ills. Don't get me wrong, I like source code analysis tools, but they will not magically fix all your security vulnerabilities. Only you can do that.

Secure products are built by developers who create secure designs and write secure code. Ultimately, writing code is an individual endeavor. You are an individual, and you cannot be replaced by a tool. Therefore, the security of your product is your responsibility! The Blaster and CodeRed worms exploited code that was written by individuals (see Figure 1).

Figure 1 Vulnerable Code is Written by Individuals
Figure 1 Vulnerable Code is Written by Individuals

Remember that all code will be scrutinized and possibly attacked. That's totally okay. Being attacked is fine. The big question is, will your code be compromised? Only you can determine that outcome. So take pride in your code. You must be happy with the quality of your code and be able to sleep at night knowing that if it's attacked, you've done everything possible to prevent the code from being whacked.

If at all possible, have your code peer-reviewed by a security expert. Don't have it examined by someone who knows nothing about security and expect that person to find security bugs and vulnerabilities. Go out of your way to have someone who really knows what he is doing look at the code.

And don't be so bigheaded that you can't ask for help when you need it. I mentioned you shouldn't solely rely on tools, but you should definitely take advantage of any that are readily available to you. Do run all available source code analysis tools on your code, and run them often. Take advantage of every possible defensive language construct and library trick available to you. For example in C#, wrap network-facing code that performs array access, where the array index is derived from a network request, in checked operators to detect possible integer arithmetic errors.

Habit #2: Never Trust Data

I've said this a billion times and I'll say it again: all input is evil until proven otherwise. If you look at the most heinous security vulnerabilities, they all share the common trait that the developer trusted the incoming data. The problem is if your code assumes the data is well formed, what happens if your assumption is incorrect? On a good day, your application will probably crash. On a bad day the attacker could inject malicious code into your process and wreak havoc.

The somewhat whimsical definition of a secure system is one that performs the tasks it is supposed to, and no more. But when there are input trust issues, you can usually get the system to perform other tasks. A cursory analysis of Common Vulnerabilities and Exposures (CVE) data (cve.mitre.org) shows that from 2001-2004, 47 percent of all security vulnerabilities tracked by CVE were input trust issues. The most well-known issues are buffer overruns, integer arithmetic bugs, cross-site scripting, and SQL injection bugs. We are starting to see new variations on this theme, such as XPath injection and Lightweight Directory Access Protocol (LDAP) injection vulnerabilities.

You can remedy input trust issues by following a few simple rules. First, don't look only for things you know are bad; this assumes you know all the bad things and you can predict all the bad things into the future. Looking for bad things is OK so long as it's not your only defense. A better strategy is to constrain the input to what you know is good. For high-level languages such as C# and Perl, I like to use regular expressions to achieve this.

Next, reject what you know to be bad. For example, if someone remotely requests a file through your code and the file name includes a dodgy character (such as : or \), reject the request. And don't tell the attacker why; just say "file not found."

Finally, and this does not work for all scenarios, sanitize the data. For example, in the case of a Web server, you should HTML-encode output that came from potentially untrusted input.

Habit #3: Model Threats against Your Code

You do have threat models, right? Threat models allow you to understand the potential risks to your software and to make sure you have the appropriate mitigations in place. But the benefits of threat modeling extend beyond secure design. Threat models can help with your code quality, too. Threat models tell you where the data came from. Is the data remote or local? Is the data from anonymous users or is it from more trusted (authenticated) users, perhaps administrators?

With this information at hand, you can determine whether your defenses are appropriate. For example, code accessible to anonymous and remote users had better be very secure code. I'm not saying that code accessible only to local admins should not be secure, but I am saying that remotely accessible code, especially code running by default, has to be bulletproof and that means more defenses, more review, and more attention to detail. Moreover, the threat model can tell you the nature of the data being protected. High value business data and personally identifiable information, for example, must be protected very well. Are your defenses appropriate?

Make sure your threat models are accurate and up-to-date, then identify all the entry points into your code and rank them by accessibility—remote versus local and high-privilege versus low-privilege (or no-privilege) users. The most accessible code should be reviewed the deepest and the earliest. Finally, review all code along anonymous data paths; in other words, start at each anonymously accessibly entry point and trace the data along that path, checking for code correctness.

Habit #4: Stay One Step Ahead

The security landscape evolves constantly. It seems that every week there are new variations of security issues. This means you must evolve and learn about new threats and defenses or you'll suffer the consequences.

Some simple strategies to stay ahead of the curve include reading a few good books on the subject of software security every now and then. Also, learn from your past mistakes and, better still, the mistakes of others. You can do this by reading bugtraq—go to securityfocus.com and sign up to receive bugtraq posts in your inbox. But trust me on this next bit of advice: create an inbox rule to move the posts to a special folder so you can handle the volume. It's significant.

Habit #5: Fuzz!

Fuzzing is a testing technique that was invented to find reliability bugs. It turns out that a percentage of reliability bugs are security vulnerabilities waiting for the right exploit! Sure, a buffer overrun might crash an application, but given a well-crafted malicious payload, the crash might not happen, and the attacker could run code to do his bidding instead. Our motto around here is "today's denial of service is tomorrow's code execution."

Just about every file-parsing bug/vulnerability was found by dumb luck or fuzzing. Microsoft has found security vulnerabilities parsing a number of file formats including the XLS, PPT, DOC, and BMP files. Most vendors have had similar vulnerabilities because parsing complex data structures is a complex task, complex code will have bugs, and some of those bugs will reveal security vulnerabilities.

You must fuzz all code that parses files and network traffic. The Security Development Lifecycle (SDL) at Microsoft is very specific about what this means for file formats. You must fuzz all parsers with 100,000 iterations of malformed files using a file fuzzer. There are some reasonable fuzzers available, and we included a file fuzzer, as well as C++ source code, in the book The Security Development Lifecycle, which I coauthored with Steve Lipner (microsoft.com/MSPress/books/8753.asp).

One last note about fuzzing. If you get a crash, don't think it is only a crash. It is likely that a good percentage of these so-called crashes are begging for someone to write an exploit. So don't simply punt a crash as "just a crash."

Habit #6: Don't Write Insecure Code

At Microsoft, we use the concept of quality gates to help reduce the chance a developer will check vulnerable code into the product. The gates run a battery of source code analysis tools on the code prior to check-in to flag any issues. And any identified issues must be fixed before the check-in can be completed. You can also enforce strict code rules such as excluding the use of banned functionality, like no calls to strcpy or strncat and no lousy crypto. (Microsoft has banned over 100 C runtime functions for new code!) For example, with regard to cryptography, we don't allow DES (the key length is too small), MD4, or MD5 (they are both broken now) in new code, unless an industry standard dictates their use.

Don't reinvent functionality. If you have code that parses a specific file format, you don't need two or three sets of parsing code; stick with the one set, make it robust, and wrap it up in a form that can be used across multiple projects.

Finally, remember that tools are no replacement for knowing how to write secure code. That's why security and privacy education are so important. You need a solid understanding of the concepts to make the judgment calls and insights your tools aren't capable of making.

Habit #7: Recognize the Strategic Asymmetry

This is one of my favorites. Remember that as a software developer, the security odds are stacked against you. I like to call this the "Attacker's Advantage, and the Defender's Dilemma." You need to get the code and the designs 100 percent correct 100 percent of the time, and that is impossible. To make matters worse, you must reach this insurmountable goal on a fixed budget, on time, while having to consider requirements of supportability, compatibility, accessibility and other "-ilities." An attacker can spend as long as he wants to find one bug, and then announce to the world that your application is insecure.

In Habit #6, I mentioned that you should stop writing new insecure code. For Habit #7, you should focus on all code because attackers attack all code regardless of age. Spend time reviewing old code for security vulnerabilities, and seriously consider deprecating old, insecure functionality. If you use agile development methods, you should think about dedicating one or more sprints to fixing old code to bring it up to the quality of newer code.

Habit #8: Use the Best Tools You Can

Finally, use the best tools you possibly can. I love source code analysis tools and I love any technology that helps me write more secure code. As I mentioned, tools are no panacea but they help. A lot! Tools also help scale the problem of source code analysis. Tools can scan huge quantities of code rapidly, much faster than a human could. And this helps give you a feeling for how "bad" some code might be.

A favorite trick of mine is to compile code using the highest possible warning levels, for example /W4 when using Visual C++® or –Wall when using gcc. If you see a large number of warnings in the code, perhaps the code has other bugs that were not found by the compiler or other tools. Such code should be subject to a greater degree of scrutiny before it ships (see Habit #3).

These are eight good habits I have seen employed by developers I deeply respect, both inside and outside of Microsoft. By themselves, the habits won't make you a stellar, secure developer, but they surely will help!

Michael Howard is a senior Security Program Manager at Microsoft focusing on secure process and improvement and best practice. He is the coauthor of five security books including The Security Development Lifecycle, Writing Secure Code, and 19 Deadly Sins of Software Security.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Threat Modeling: Uncover Security Design Flaws Using The STRIDE Approach

Threat Modeling

Uncover Security Design Flaws Using The STRIDE Approach

Shawn Hernan and Scott Lambert and Tomasz Ostwald and Adam Shostack
This article discusses:
  • The importance of threat modeling
  • How to model a system using a data flow diagram
  • How to mitigate threats
This article uses the following technologies:
STRIDE
Contents
Designing Secure Software
Threat Modeling and STRIDE
Data Flow Diagrams
A Sample System
Applying STRIDE to the Fabrikam Analyzer Database
Analyzing Data Flows and Data Stores
Analyzing Processes
Mitigating the Threats
Finding Manifestations of Threats
Attack Patterns
Conclusion

W
hether you're building a new system or updating an existing one, you'll want to consider how an intruder might go about attacking it and then build in appropriate defenses at the design and implementation stages of the system. At Microsoft, we approach the design of secure systems through a technique called threat modeling—the methodical review of a system design or architecture to discover and correct design-level security problems. Threat modeling is an integral part of the Security Development Lifecycle.

There are multiple approaches to threat modeling, and anyone who tells you his method is the only right one is mistaken. There aren't any well-established ways to measure the quality of a threat model, and even the term "threat" is open to interpretation. Of course that's the nature of the beast; even in the more mature field of cryptography, many popular algorithms have not been proven to be secure. But, while we can't often prove that a given design is secure, we can learn from our mistakes and avoid repeating them. That is the essence of threat modeling.

In this article we'll present a systematic approach to threat modeling developed in the Security Engineering and Communications group at Microsoft. Like the rest of the Security Development Lifecycle, threat modeling continues to evolve and to be applied in new contexts. As you create your own processes for developing secure code, this approach might serve you well as a baseline.

Designing Secure Software

It's difficult enough to design good software, and security makes it even tougher. Flaws that are embedded in a system may or may not be encountered during ordinary use. Indeed, under ordinary use some flaws don't really matter. But in a security context, flaws do matter because attackers can induce failures by setting up the highly specific conditions necessary to trigger a flaw. Something that may have only a one in a billion chance of happening randomly might be dismissed as irrelevant. But if that flaw has security implications, you can bet that an attacker will take advantage of it.

One of the problems in designing secure software is that different groups think of security in different terms. Software developers think of security primarily in terms of code quality while network administrators think of firewalls, incident response, and system management. Academics may think of security mostly in terms of the classic Saltzer and Schroeder design principles, security models, or other abstractions. Of course, all of these things are important in building secure systems. See Figure 1 for a summary of Saltzer and Schroeder's design principles. But maybe the single biggest problem is a lack of security success criteria. If we want to avoid security failures, it means we have to have some idea of what security success looks like.

Figure 1 Security Design Principles (Close Figure 1)

Principle | Explanation
Open design | Assume the attackers have the sources and the specs.
Fail-safe defaults | Fail closed; no single point of failure.
Least privilege | No more privileges than what is needed.
Economy of mechanism | Keep it simple, stupid.
Separation of privileges | Don’t permit an operation based on a single condition.
Total mediation | Check everything, every time.
Least common mechanism | Beware of shared resources.
Psychological acceptability | Will they use it?

Fortunately, we do have some idea of what security means. It means that the systems have the properties of confidentiality, integrity, and availability, that users are authenticated and authorized correctly, and that transactions are non-repudiable. Figure 2 explains each property. You want your systems to have all of these properties, but there is no integrity or availability API. What do you do?

Figure 2 Security Properties (Close Figure 2)

Property | Description
Confidentiality | Data is only available to the people intended to access it.
Integrity | Data and system resources are only changed in appropriate ways by appropriate people.
Availability | Systems are ready when needed and perform acceptably.
Authentication | The identity of users is established (or you’re willing to accept anonymous users).
Authorization | Users are explicitly allowed or denied access to resources.
Nonrepudiation | Users can’t perform an action and later deny performing it.

Threat Modeling and STRIDE

One way to ensure your applications have these properties is to employ threat modeling using STRIDE, an acronym for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege. Figure 3 maps threats to the properties that guard against them.

Figure 3 Threats and Security Properties (Close Figure 3)

Threat | Security Property
Spoofing | Authentication
Tampering | Integrity
Repudiation | Non-repudiation
Information disclosure | Confidentiality
Denial of service | Availability
Elevation of privilege | Authorization

To follow STRIDE, you decompose your system into relevant components, analyze each component for susceptibility to the threats, and mitigate the threats. Then you repeat the process until you are comfortable with any remaining threats. If you do this—break your system down into components and mitigate all the threats to each component—you can argue that the system is secure.

Now, the truth of the matter is that we can't prove it. We can't yet show that the interactions between components that are individually immune to a spoofing threat aren't susceptible to a spoofing threat when they're composed into a system. In fact, frequently threats materialize only when systems are joined to create larger systems. In most of those cases, the very act of combining subsystems into larger systems involves violating the original assumptions the subsystem made. If a system was never designed to be used over the Internet, for example, when you expose it to the Internet, new security concerns will emerge.

In any case, what is true is that if any component of the system is susceptible to a spoofing threat, you can't say that all your users are properly authenticated.

Data Flow Diagrams

Data flow diagrams (DFDs) are typically used to graphically represent a system, but you can use a different representation (such as a UML diagram) as long as you apply the same basic method: decompose the system into parts and show that each part is not susceptible to relevant threats.

DFDs use a standard set of symbols consisting of four elements: data flows, data stores, processes, and interactors, and for threat modeling we add one more—trust boundaries. Figure 4 shows the symbols. Data flows represent data in motion over network connections, named pipes, mail slots, RPC channels, and so on. Data stores represent files, databases, registry keys, and the like. Processes are computations or programs run by the computer.

Figure 4 DFD Symbols (Close Figure 4)

Item | Symbol
Data flow |
Data store |
Process |
Multi-process |
Interactors |
Trust boundary |

Interactors are the end points of your system: the people, Web services, and servers. In general, they are the data providers and consumers that are outside the scope of your system, but clearly related to it.

Trust boundaries are perhaps the most subjective of all: these represent the border between trusted and untrusted elements. Trust is complex. You might trust your mechanic with your car, your dentist with your teeth, and your banker with your money, but you probably don't trust your dentist to change your spark plugs.

Getting the DFD right is key to getting the threat model right. Spend enough time on yours, making sure all the pieces of your system are represented. Have you noted all the files and registry keys your app touches? Are you reading data from the environment?

Each of the elements (processes, data stores, data flows, and interactors) has a set of threats it is susceptible to, as you see in Figure 5. This chart, along with your DFD, gives you a framework for investigating how your system might fail. It's probably a good idea to parcel out the investigative work to subject matter experts and build checklists to ensure you don't make the same mistake twice. For example, you could have your networking team investigate how information disclosure threats apply to your network data flow. They will understand the relevant technologies and be well suited to do the research on security as it pertains to their portion of the application.

Figure 5 Threats Affecting Elements (Close Figure 5)

Element | Spoofing | Tampering | Repudiation | Information Disclosure | Denial of Service | Elevation of Privilege
Data Flows |  |  |  |  |  |
Data Stores |  |  |  |  |  |
Processes |  |  |  |  |  |
Interactors |  |  |  |  |  |

A Sample System

Let's say you need a system to collect the accounting files from your sales force, compute sales data on your database server, and produce weekly reports. We'll call the system the Fabrikam analyzer database. The goal is fairly simple: getting files from a set of systems and performing some analysis of the files on a centralized server. It's the business goal as the customer stated it, but you need to turn the problem statement into specifications and plans.

There are many obvious potential threats to this system, and many of them come from the implicit security requirements of the problem statement. The collection process alone raises a number of questions. Collecting information means moving it from one place to another. How are you going to secure it in transit?

You'll be manipulating accounting files, which by their very nature are sensitive and often subject to legal requirements. And you'll need to identify a specific group of people—the sales force. How will you know them? The problem statement implies a number of security requirements:

  • The data has to be protected from inadvertent disclosure in transit and in storage.
  • The sales force needs to be authenticated and authorized.
  • The application must respond gracefully to attacks that rely on malicious input, like SQL injection and buffer overflows.
  • The server needs to be able to perform the calculations at least weekly.
Customers may never state these explicitly, so designers must find the security requirements inherent in the problem statement.

Of course, there are also lots of less obvious security requirements that need to be addressed. What happens if the files can be overwritten on the server in such a way that a salesperson could conceal a suspicious sale? What happens if one salesperson can claim credit for the sales of another?

And just who is "we"? What if Contoso Corporation is able to pose as the Fabrikam server to the Fabrikam sales force? Remember, the attacker doesn't have to respect your protocol or use your tools to access your servers; he will spend more time than you imagine looking for flaws and he may well have a large number of machines available for performing complex calculations.

At this point, if the component is a low-risk system or you have a great deal of experience in similar systems, you may decide that it's reasonable to go off and start planning mitigations for the threats. And that's OK. But what if it's a high-risk component? How do you know what you don't know?

If you stop at this point, your threat model will be limited not only by what you know, but what you happen to remember at the time you're working on it. You not only have to think like an attacker, you have to think like all attackers. Simultaneously.

Applying STRIDE to the Fabrikam Analyzer Database

Now, let's try to create a data flow diagram from the problem statement. An initial attempt might look like Figure 6. On the server side, there are two processes, two data stores, and three data flows. There is one trust boundary between the server and client sides of the system, with one data flow that crosses the trust boundary for each client. For each client, there is a salesperson, an accounting application, the accounting data, and the sending process.

Figure 6 An Initial DFD for the Analyzer Database
Figure 6 An Initial DFD for the Analyzer Database

But is this the right DFD? Is this the DFD that will yield the most complete picture of threats? Some simple rules of thumb suggest that it's probably not the right one. To begin with, there is a data sink. Data goes into the Analysis Database and never gets read. The customer never mentioned anything about reading the data explicitly because it wasn't related to the problem at hand. A designer might say something like "that part of the problem is out of scope." But from a security point of view, it's not out of scope at all. So you need to represent the reader of the data somehow.

Here are some general rules for understanding if your DFD is sensible. First, be careful of magic data sources or sinks: data isn't created out of thin air. Make sure you have a user represented as a reader or writer for each data store. Second, beware of psychokinesis as a data transport. In other words, make sure there is always a process that reads and writes data. It doesn't go directly from a user's head to the disk, or vice versa. Third, collapse similar elements within a single trust boundary into a single element for modeling purposes. If they are implemented in the same technology and are contained within the same trust boundary, you may be able to collapse them. Fourth, be careful when modeling details on either side of a trust boundary. The temptation is to model things on both sides of a trust boundary simultaneously. It's good practice to have a context DFD and breakout diagrams that show more detail. Our system here does represent the client and server systems simultaneously in a single model. But remember that the attacker is under no obligation to use your tools or respect your protocols.

Revising the data flow diagram to take these rules into account, the DFD in Figure 7 is probably a better representation. The changes here are notable. We've collapsed the collection and analysis processes into one. This doesn't necessarily mean that they will be implemented together—the point is to not lose the forest for the trees. Think "major function," not implementation.

Figure 7 A Better DFD
Figure 7 A Better DFD

We've also represented the client side as nothing more than external interactors, and that is a powerful change. This reflects the truth that an attacker is free to do whatever he or she chooses. Consequently, we've limited this threat model to the server side only—a complete threat model would include a similar representation of the client side systems, showing the server as nothing more than a set of external interactors. That's the very definition of a trust boundary—you don't trust what's on the other side.

Now we can represent all this in lists and tables and start breaking the large problem into a series of smaller problems.

Analyzing Data Flows and Data Stores

Let's first look at the data flows. There are three of them as you saw in Figure 6. Each of these data flows is subject to threats listed in Figure 3. Let's see how each process is vulnerable.

Data Flow 1: Sales to Collection Where sales data is transferred to the collection process, tampering is possible. In other words, data might be modified as it travels over the Internet, especially if the data comes from laptops in a variety of security settings. Information disclosure is another risk. Data in transit may be read by those who should not have access.

The collection process might also fall victim to denial of service attacks; an attacker might prevent the collection server from being accessible to the sales people. (If the attacker does this on the last day of a quarter, it could have a material impact on the company.)

Data Flow 2: Sales System List to Collection Similar threats exist for sales system list collection. Someone could tamper with the data by inserting a new system into the list of salespeople that could allow the input of false data. Removal of a system could prevent a salesperson from being able to register sales. (This could be modeled as a denial of service. We call it tampering. Don't get too hung up over the terminology.)

As you consider threats to this data flow, it may occur to you that the authentication system for the sales force has not been identified. An inadequate authentication system would present a spoofing threat to the sales force. Make sure the threat gets recorded and addressed. The same threat will occur to you when you examine spoofing threats against the sales force. A little redundancy is great.

The list of salespeople is probably interesting to Fabrikam's competitors. It could also be interesting to insiders if layoffs are happening, so the threat of information is significant and needs to be addressed. Don't be lulled into complacency because you can't imagine why someone would want your data. You simply have to assume someone will.

Depending on how large Fabrikam is, and how often the sales force changes, denial of service may not be very important. It's OK (and even sensible) to perform some level of risk analysis when looking at threats. But remember that the further away you are from your customer, the harder it is to know what the customer's tolerance is for different risks. Don't assume too much about your customer's situation or tolerance for risk.

Data Flow 3: Analysis Process to Analysis Store Here we encounter an interesting situation regarding tampering. We have a data flow contained entirely within a trust boundary. In this situation, a hardcore security theorist might say there's absolutely no need to worry about processes entirely within a trust boundary—after all, you trust them.

On the other hand, however, a hardcore security practitioner might reply that anything can fail, and this isn't any different. We have sympathy for both views and would resolve the issue in terms of overall priority. Data flow 1 is clearly more exposed than data flow 3, and the effort you spend examining data flow 1 ought to reflect that greater level of exposure. But data flow 3 isn't entirely without risk.

For example, what if the analysis database is stored on another machine? The threat model as written seems to suggest that the collection process is housed on the same machine as the database, but perhaps it's not. Perhaps the modeler made an incorrect assumption, or the decision hasn't yet been made.

If it turns out that the analysis database is stored on a remote machine, it's very likely that data flow 4 has a similar threat profile to data flow 1. Finally, even if you can rely on the trust boundary, don't forget that situations change. Defense in depth may be a worthwhile investment.

Information disclosure and denial of service are also threats here. In fact, the situation is the same as with the tampering threat. In other words, is the trust boundary reliably and correctly set? If not, these threats need to be considered.

Now let's look at the data stores.

Data Store 1: Analysis Database This data store is vulnerable to tampering, especially because what's in the database hasn't been fully specified. Is this a sales database that includes order fulfillment and salesperson bonuses? Is it a forecasting database that produces sales predictions? Different data types attract different attackers. Note that attackers may be insiders, not outsiders. They may have legitimate access to the database in order to do their jobs.

Information disclosure is a growing problem. If Fabrikam takes sales orders from consumers on credit, there may be Social Security Numbers or credit card numbers in the database. There is probably proprietary information in the database that competitors would like to see. If Fabrikam is providing health or medical supplies, it may be subject to HIPAA privacy constraints.

Denial of service may be a threat as well. Who uses this database, and for what? Filling up the database is a simple attack. When the database is full, common responses are to stop servicing new requests or to overwrite old data. Denial of service attacks matter most when they prevent business goals from being met.

Data Store 2: List of Sales Systems Tampering is a threat here. Attackers could add salespeople who can make mischief or delete salespeople and prevent them from getting their jobs done. Information disclosure and denial of service are also viable threats.

Data Store 3: Laptops Tampering with laptops may allow an attacker to steal data or access control tidbits such as passwords, keys, or certificates. An attacker could install spyware to give himself ongoing access to the system. This probably crosses threat models into other projects, but if you're designing a system that can't reasonably be managed, that may make your customers or users less secure. All information on a laptop is fair game if the attacker controls it. Again, threats here probably involve more than just this part of the threat model.

Analyzing Processes

A process that pretends to be the collection process (spoofing) can collect all of the data that sales is trying to submit. This leads to information disclosure and denial of service. There are many threats that, when realized, can lead to other threats. If you can pretend to be the administrator, for example (spoofing), you can shut down the system (a denial of service threat). Also, if an attacker tampers with the data here and corrupts or reads memory, he may be able to cause a denial or service or elevation of privilege. Don't get too caught up in trying to track down all the follow-up consequences, though.

Repudiation, to which the collection process is vulnerable, is about lying. A repudiation threat is when you can take an action and plausibly deny having taken it. In this case, repudiation threats might involve, for example, submitting revised sales data that overwrites existing data and being able to claim that you did not do it.

Information disclosure threats against a process can be tricky—you normally think of these threats as affecting data flows and data stores. But a process holds lots of valuable data in memory. If an attacker can read from the memory of a process, he might not have to break into the database.

Similarly, if an attacker discovers something about the internal structure of a program, it can help him conduct other types of attacks. An information disclosure attack that allows an intruder to discover the memory address of certain variables (for example) can be a very valuable stepping stone in the attack process.

Both denial of service and elevation of privilege are also quite possible here. Insiders and competitors might be motivated to prevent the ordinary collection of the data, and this could be performed in a variety of ways. For example, an attacker who can cause the collection process to run code they have written can spoof, tamper, or deny service. The collection process is probably exposed to the Internet, and proper secure code will be very important.

You can continue this sort of analysis for each of the threats and each of the elements in the DFD until you can say that you've addressed tampering, information disclosure, and denial of service against all the data flows and data stores; each of the six STRIDE threats against all of the processes; spoofing and repudiation threats for all interactors; and unique threats that affect your trust boundaries. None of this will guarantee that the system is secure, but it will certainly help you sleep better at night.

Mitigating the Threats

The ideal situation is to mitigate a threat with a strong, well-understood solution. For example, using strong cryptography appropriately is believed to be a strong countermeasure to many types of information disclosure threats. You may never be able to prove that a defense is perfect. However, one of the nice things about the STRIDE model is that it gives you insight into the nature of the mitigations you need. Simply recasting Figure 3 in terms of available technologies gives you an idea of what kinds of mitigations are necessary. Choosing a technology can be challenging, however. In general, I've found that asking two simple questions can be helpful: can the technology be used to mitigate the threat, and would it actually be used in the scenario you're concerned with?

Finding Manifestations of Threats

Now that you have broken down the big problem into smaller problems, how do you solve those smaller problems? You've got some research to do. You want to find previous failure modes for the technologies you'll be using. If a data flow, for example, will be using Remote Procedure Call (RPC) over TCP, you need to find not only general classes of failures in data flow but also specific known failures for TCP and RPC.

One excellent approach to learning how threats manifest themselves is using Chapter 22 of The Security Development Lifecycle, by Michael Howard and Steve Lipner (Microsoft Press®, 2006) in which threat trees are developed for STRIDE threats against each of the four standard DFD elements. For example, one threat tree explores how tampering might manifest itself against a data flow in a general sense. The trees are presented such that the leaf nodes of the trees suggest attacks that will realize the threat. The idea is to use the leaf nodes of the threat trees as leading questions—sort of augmenting the brainstorming you did earlier.

So, for example, let's say there is a tampering threat against the Sales to Collection data flow. Looking at the list of questions from The Security Development Lifecycle, the first one related to tampering with a data flow is the following: is the data flow defended (hashed, MAC'd, or signed) using anti-replay defenses such as time stamps or counters?

This suggests a simple attack—replaying valid messages—that has thus far been overlooked in the entire discussion! It's not that the attack is novel or previously unknown. It's just that it wasn't obvious in the problem statement, and it would be an easy attack for even seasoned security folks to overlook.

The impact of the attack in this scenario is this: what if a salesperson could submit a set of sales data twice? Imagine that Vishal and Eric are competing for a trip to Hawaii. Eric might be tempted to submit a set of data twice to make his sales numbers look really good and win the trip.

You might think the accounting system would catch that, because after all, that's part of the purpose of an accounting system. But how do you know? This is just a reporting system after all, and maybe the data generated from this system isn't ever double-checked against the official accounting data.

Maybe the attack is mitigated somehow and maybe it isn't. But understanding whether an attack is possible and uncovering attacks that haven't been thought of before are part of the main goal of threat modeling.

Given this newly uncovered threat, you are now presented with a new task: understanding whether the threat has been mitigated. In the end, a likely result of an investigation like this would be that the files are uniquely identified with what is effectively a random number. Now you have a business decision to make: are you comfortable with the risk, or do you need a stronger solution?

The use of prebuilt threat trees has proven effective in a number of situations at Microsoft to ensure that known attacks aren't overlooked. And as new attacks are discovered, the threat trees themselves can be augmented with these new threats. In this way, the threat trees can form a sort of institutional knowledge. Remember, though, that they are very general in nature. There are whole classes of attacks that apply to one technology and not another technology.

In the case of a data flow, for example, how would you know that the security properties of TCP are different from the security properties of User Data Protocol (UDP), and that factors like the initial sequence number and window size play a role in the security of your network connection? If you're familiar with network security, you might recognize these problems; if not, you may overlook these concerns when designing a solution. That is where attack patterns come in.

Attack Patterns

Attack patterns are the manifestation of one or more threats in the context of some specific technology. For example, the strcpy operation in C might permit an attacker inputting long data strings to corrupt the memory of a target program, allowing him to execute code of his choice. This is how a buffer overflow can pose an elevation of privilege threat. Of course, an attacker might use the buffer overflow to do other things as well—tamper with paycheck data for instance. Sometimes people talk about attack patterns in more formal terms, expressing them as a set of preconditions and postconditions. But for our purposes here, they are the specific things that can allow an attacker to realize a threat.

You may or may not be familiar with these kinds of attacks, but in fact there are many more types of attacks that affect specific technologies in specific contexts, and there's little that can be done about it. You have to be familiar with the attacks that affect your technology and build in appropriate defenses.

Conclusion

Designing secure software can be difficult, but as with any challenge, a good strategy is to break down the problem into smaller parts that are more easily solved. For high-risk activities, we believe it is helpful to have an organized framework for doing so, and using the STRIDE model with threat modeling is one such approach. We have used this model in a number of teams within Microsoft and the results have been promising—in nearly every case, we turned up a design issue that might have gone unnoticed until much later. Once you get used to applying the STRIDE model, it often boils down to getting the DFD right, brainstorming attacks, and reviewing the known checklists. This can go pretty quickly.

As you strive to develop secure software, we recommend threat modeling as a key part of your process, and specifically the STRIDE model presented in this article. But the key point is to find a method that works for you, apply it early in your design, keep in mind that any component can fail, and do the necessary research to ensure you've accounted for known attack patterns.

Finally, design is just one part of building secure software. Executive support, implementation, testing, building and delivering, and servicing and maintenance all play crucial roles in the ultimate security of your systems. You've got to get it all right.

Where to Go for Technical Details

For an excellent overview of many common types of attacks, we suggest Writing Secure Code, Second Edition, by Michael Howard and David LeBlanc (Microsoft Press, 2002). For more technically specific texts, see:

  • Securing Web Services with WS-Security, by Jothy Rosenberg and David Remy (SAMS, 2004).
  • SQL Server Security, by Chip Andrews (Osborne McGraw-Hill, 2003).
  • Secure Coding in C and C++, by Robert Seacord (Addison Wesley, 2005).
  • Building Secure Microsoft ASP.NET Applications (Microsoft Press, 2006).
  • Security for Microsoft Visual Basic .NET, by Ed Robinson and Michael James Bond (Microsoft Press, 2003).
  • The Software Vulnerability Guide, by Herbert H. Thompson and Scott G. Chase (Charles River Media, 2005)

Finally, for a list of common security-related defects, one emerging resource is Mitre’s Common Flaw Enumeration project.

If you work frequently in any technologies, it’s probably a good idea to incorporate common flaws from those technologies into your own threat trees or checklists. And of course, you should review current and past security advisories from your own organization as well as other organizations who have implemented similar technologies. Microsoft makes security bulletins available at microsoft.com/security. Notable third parties include:

  • The CERT Coordination Center
  • Security Focus
  • Secunia

If you’re adding a new feature that your competitors may already have, research their security mistakes as well as their successes. Many vendors and third parties make security bulletins freely available.

Shawn Hernan is a Security Program Manager for Microsoft, currently working on training to help meet the vision of delivering software that is secure by design, secure by default, and secure in deployment. Prior to joining Microsoft, he was the vulnerability team leader at the CERT Coordination Center at Carnegie Mellon University.

Scott Lambert is a Security Program Manager on the Secure Windows Initiative (SWI) team at Microsoft. He owns enhancing the internal security tools, including various fuzzing tools. Leveraging his industry experience, Lambert works to ensure that SWI tools identify the vast majority of vulnerability classes.

Tomasz Ostwald currently works as a Security Program Manager in the Security Windows Initiative team, where he conducts Final Security Reviews of products released by Microsoft.

Adam Shostack is a Program Manager on the Security Development Lifecycle (SDL) team at Microsoft. He owns the threat modeling component of the SDL.


Single Sign-On: A Developer's Introduction To Active Directory Federation Services

Single Sign-On

A Developer's Introduction To Active Directory Federation Services

Keith Brown
This article discusses:
  • What federation means
  • Implementing federation in ASP.NET applications using ADFS
  • Trust relationships and security considerations
This article uses the following technologies:
ADFS, ASP.NET
Contents
A Solution Using ADFS
ADFS Architecture
Federated Logon Walkthrough
Securing the Federated Logon
Where Do You Come From, User?
Claims and Transformation
Where Do Claims Come From?
Anonymity
The Federation Service Proxy
ADFS-Enabling Your Web Application
Getting Started with ADFS in the Lab

O
ne of the most important components of Windows Server® 2003 R2 is Active Directory Federation Services (ADFS). ADFS solves a number of problems—one of the most obvious and compelling being business-to-business automation. In this article I'm going to take a look at ADFS from the perspective of a developer who is building a Web application and wants to allow other organizations to use it (for an administrator's viewpoint, see Matt Steele's article in TechNet Magazine.

What kind of business-to-business problems am I referring to? Imagine that a bicycle manufacturer called Fabrikam wants to expose a Web application that will allow authorized dealers to purchase bikes and parts at wholesale prices. There are over two hundred dealers, each with several people who need to use the application. Fabrikam is going to need a secure logon mechanism.

An obvious solution would be to create a database containing user names and passwords, but this could become very costly to manage. If someone makes a call to Fabrikam claiming to be an employee of a dealer, how is Fabrikam going to verify this claim? They'll probably want to contact someone they trust at the dealership to verify the employee's status before provisioning a new account. Just consider the maintenance cost of such a user account: people forget user names, passwords, and have other problems. And what happens when the employee is terminated from the dealership? Is anyone going to remember to notify Fabrikam that a user account should be removed (or deprovisioned, in identity lingo)? If not, that user could go home and place false orders on the dealer's behalf.

Passwords themselves pose another problem. As computing power has increased, passwords have become easier and easier to attack, and many organizations now prefer to use stronger authentication techniques like smart cards. But because Fabrikam must work with so many different dealerships, it's going to have a difficult time supporting anything stronger than passwords.

Notice that trust is a factor here as well. Fabrikam trusts each dealership to supply an accurate list of employees who should be allowed to make purchases using Fabrikam's Web application.

A Solution Using ADFS

ADFS helps you establish trust relationships and reduces the need for provisioning user accounts. Why should you bother building a user account database for your application when your dealerships already have software that authenticates their users?

ADFS is the Microsoft implementation of the WS-Federation passive requestor profile protocol (passive indicates that all the client needs is a cookie- and JavaScript-capable Web browser—a passive agent that does not run any special code to help implement the protocol). Here's how this protocol works: when a user at a dealership points her browser at Fabrikam's purchasing application, her browser will eventually be redirected back to a Web site at the dealership where she works, so that the dealership can authenticate her. Then her browser will be redirected back to Fabrikam, and this new request will carry a signed statement by the dealership indicating that this is an actual employee who is authorized by the dealership to use the application. (I'm simplifying this quite a bit.) The point is that Fabrikam trusts the dealership to authenticate its own users, which is key to federation technology.

If a dealership has software that supports the WS-Federation passive profile, Fabrikam doesn't need to provision any user accounts for employees at that dealership. It doesn't need to worry about password reset requests. When a user at the dealership switches roles and no longer works in the purchasing department, as long as her identity information is changed to reflect this, when she tries to use Fabrikam's purchasing application the statement from the dealership will indicate that she's no longer a purchasing agent and she'll be denied access. Or if she quits her job at the dealership and her account is deprovisioned, she'll also no longer be able to use Fabrikam's purchasing application. By federating with the dealership, Fabrikam is virtually guaranteed to receive more relevant, up-to-date information about the user's identity.

ADFS is built on standards like WS-Federation, which was coauthored by Microsoft, IBM, Verisign, BEA, and RSA Security. Different organizations often run very different software. If Fabrikam uses Windows Server 2003 R2 with ADFS, but has dealerships running IBM WebSphere or BEA WebLogic, this really shouldn't be a problem because WebSphere and WebLogic both implement WS-Federation.

End users also get a better deal with federated identity. Instead of having to remember yet another password, a purchasing agent at a dealership can simply point her browser at Fabrikam's application and immediately start working. If the dealership's authentication system supports an integrated logon through a browser, as Windows does with Internet Explorer®, the user won't even be prompted for her credentials; she'll be authenticated silently and the federation service will translate the local knowledge of her identity into the signed statement for Fabrikam that I mentioned earlier. This is the brass ring of Web security: single sign-on across the Internet and across organizational and technological boundaries.

ADFS Architecture

Now I'm going to show you the various parts of ADFS and explain some terminology. In any given federation relationship, one side supplies the users (accounts) and the other side supplies the applications (resources). When you install ADFS, you'll configure its trust policy using the ADFS administration snap-in, which shows up in your Administrative Tools folder, to indicate the list of partners with which you want to federate. Users from the account partner will be accessing ADFS-enabled applications in the resource partner. Figure 1 shows the tree view of ADFS trust policies from either side of a relationship, as seen from the ADFS administration snap-in.

Figure 1 Policy for Two Partners
Figure 1 Policy for Two Partners(Click the image for a smaller view)
Figure 1 Policy for Two Partners
Figure 1 Policy for Two Partners (Click the image for a larger view)

On either side sits what is known as a federation service. Each federation service exposes a Web application, and it's expected that the user's browser will be redirected to these applications in order to establish a logon. The federation service resolves the impedance mismatch between the account and resource partners, so much so that the partners don't have to be using the same identity or operating system technology. Your app doesn't need to worry much about the federation service; the ADFS team provides a Web agent that runs in the ASP.NET pipeline as an HttpModule that will do the heavy lifting for you. All you need to do is configure your application to use the Web agent and supply some configuration settings so it knows where to find your company's federation service. Figure 2 shows the basic structure of ADFS and how the user's browser interacts with the various components. This shows what things would look like if ADFS were on both sides of the wire, but it's easy enough to imagine the account partner, for example, being implemented in WebSphere with an IBM directory service behind it, exposing a Java-language federation service.

Figure 2 ADFS Architecture
Figure 2 ADFS Architecture

Federated Logon Walkthrough

When Alice, an authorized user from one of Fabrikam's dealerships, points her browser at Fabrikam's purchasing application for the first time, the Web agent notices that she doesn't have an ADFS cookie. She's not logged in. So the Web agent redirects her browser to Fabrikam's federation service before the purchasing application even sees the request. Fabrikam's federation service then redirects the browser to the dealership's federation service, adding to the request a unique identifier for Fabrikam's federation service so the dealership will know which partner is requesting a logon. When Alice's browser ends up back at her own federation service, she's asked to authenticate. Because her dealership is using integrated Windows authentication, Alice's browser automatically responds, establishing a logon with her federation service. Once that logon is successful, her federation service issues a Security Assertion Markup Language (SAML) token describing Alice, and redirects Alice's browser back to Fabrikam's federation service, sending the SAML token along with the request as part of the POST body (the page that is returned has JavaScript that auto-submits the form containing hidden input elements). Fabrikam reads the contents of the token, and issues a different SAML token that contains the set of claims that the application will ultimately see (I'll explain why two separate SAML tokens are used in the section on claims transformation). This second token is sent using the POST method and written out to a cookie from fabrikam.com, allowing Alice to use the purchasing application until her cookie expires, which by default is in 10 hours.

Remember the Web agent in the purchasing application that was looking for a logon cookie? That's what started this whole sequence of events. Now that the cookie exists, the Web agent peels it apart and reads the claims in the SAML token that was issued by Fabrikam's federation service. The Web agent allows the Web page in the purchasing application to execute. If the application needs to know the logged-on user's name, it needs only to look in the usual place: HttpContext.User.Identity.Name.

The implementation of IIdentity used to convey this information is of type SingleSignOnIdentity, which also exposes a number of other useful properties, including the authentication method used by the account partner to authenticate the user in her home realm. From this class the application can also discover the entire set of claims sent by its federation service.

You should note that the partner federation services never directly talk to one another. It's only through browser redirections and associated query strings and POST bodies that the services are able to communicate.

Securing the Federated Logon

Now let's dig into some of the details to show you how the logon is secured. One attack vector would be to try to read the SAML tokens off the wire and then replay them at your convenience so you could impersonate a legitimate user. To prevent this, all of the communication occurs over HTTPS. This is critical; if you try to install ADFS without already having an SSL certificate installed for the default Web site in IIS, the ADFS installer will warn you and quit before it even gets started.

What about the logon cookies that contain claims? If the user wanted to elevate her privileges to an application she could simply add claims to the SAML token in her cookie! But this type of tampering can be detected because each SAML token is signed with a private key known only by the issuing federation service. This has implications for deploying ADFS. If you're acting as a resource partner, each of your account partners must supply certificates for their account federation services. Your resource federation service will use the account partner's certificate to verify signatures on the SAML tokens that it issues. The Web agent must do something similar, verifying that each SAML token it receives via a logon cookie was signed by the resource federation service.

Cookies issued by ADFS are always marked with the Secure bit, which indicates that the browser should only send the cookie over HTTPS, not HTTP. So if any part of your application runs over HTTP, the logon cookie won't be present and you won't have access to the user's identity information. It'll feel as though the user is anonymous. I'd recommend running the entire application over SSL to simplify your life and reduce the chances of leaking sensitive data to an eavesdropper.

Cross-site scripting (XSS) bugs in an ADFS-enabled Web application are devastating, as they would be in any system that uses cookies to represent logon sessions. Cookies can easily be stolen from an application that has this type of vulnerability, and if you steal the cookie, you've stolen the logon. As a defense-in-depth measure, ADFS logon cookies time out after a work day by default, and you can adjust these timeout values in the ADFS trust policy. If you're not entirely comfortable with what XSS is and how to prevent it, take a half an hour to work through the interactive XSS lab I wrote.

Where Do You Come From, User?

Fabrikam has many dealerships that act as federated account partners. When one uses the purchasing application, how does Fabrikam's federation server know to which partner to redirect the user's browser? It doesn't. With multiple partners, Fabrikam's federation server must pause the logon process to display a list of partners to the user so she can select one to authenticate her. This is known as home realm discovery. If the client lies about her home realm, she's only going to inconvenience herself, as she won't have credentials to authenticate with any of the other dealerships.

One of the goals of building a great ADFS application is to avoid bothering the user with these sorts of details. The Northwind bike shop can eliminate this step by having its users access Fabrikam's Web site through a link with a magical query string parameter, whr. (I like to think of this as "which home realm?") The Web agent looks for this query string argument, and if it finds it, strips it out of the request during preprocessing. The value of this parameter is the URI of the partner federation server (each federation server names itself with a URI). So employees at Northwind shop could be presented with a link that looked something like this:

https://fabrikam.com/purchasing.aspx/?whr=urn:federation:
Northwindbikeshop
This gives ADFS enough of a hint to avoid having to display an interactive home realm discovery page.

Claims and Transformation

I've been talking a lot about claims so far in this article. A new, increasingly important security concept on the Windows platform, claims are a general way of making statements about an entity such as a user. Consider a Kerberos ticket, which contains user and domain group identifiers for the logged-on user. This is really just a set of claims signed by the domain controller that issued the ticket. The user identifier is an identity claim. But while this may be useful for auditing or personalization, it's not typically used for access control. The groups in the ticket are more likely to be used for performing access checks. For example, if you're a member of the Managers group, you may be allowed to approve high-value purchases.

ADFS supports three types of claims: identity claims, group claims, and custom claims. An identity claim takes the form of a user principal name (UPN), an e-mail address, or an arbitrary string called a common name. A group claim is Boolean (either a member or not a member), and it doesn't necessarily have to represent a Windows group: it might simply represent a logical role that your application understands. A custom claim contains a string value. Age, gender, or perhaps even ManagerName would be examples of custom claims that you can configure in ADFS. These types of custom claims should make you start thinking about privacy issues.

Claim transformation is how ADFS addresses both privacy issues and other practical concerns such as the fact that not every company uses the same terminology, and that not every resource partner needs to know a user's e-mail address or age. This is why ADFS trust policy allows you to configure each partner differently; you should only send the claims that partner needs.

Each company defines a set of organizational claims; this is like determining the vocabulary that it understands. For example, Fabrikam might define a group claim called "Owner" that it uses to represent the owner of a bike dealership. But the Northwind bike shop may represent this role with a claim called "Manager." That's OK as long as the meaning is the same, because the owner of Northwind can use his resource federation trust policy to map his outgoing "Manager" claim into an "Owner" claim that Fabrikam will understand. Administrators can set up these sorts of mappings on either side of a federation relationship. Figure 3 shows an example of a claim mapping in a resource partner's trust policy.

Figure 3 Claim Mapping
Figure 3 Claim Mapping(Click the image for a smaller view)
Figure 3 Claim Mapping
Figure 3 Claim Mapping (Click the image for a larger view)

Some claim mappings are too dynamic to be represented by a static mapping in trust policy. Suppose a resource partner needs a claim called IsOfLegalVotingAge to certify that someone is 18 years or older, but all you have is the user's date of birth as a custom claim from Active Directory®. What you need is a claims transformation module, which is an assembly with a class that implements a managed class called IClaimTransform. You can wire this up to your trust policy via the trust policy property sheet shown in Figure 4. This module is called twice: once before the normal trust policy mapping occurs and once afterward, so you can perform pre- or post-processing. Every ADFS trust policy allows a module like this to be installed, which means you can do dynamic claims transformation on both the account and resource side of a federation.

Figure 4 Claims Transformation Module
Figure 4 Claims Transformation Module

Where Do Claims Come From?

Unless you're writing a claims transformation module that generates claims dynamically, all of your claims will originate from an account store, which can be either Active Directory or Active Directory Application Mode (ADAM), a lightweight directory server based on the Active Directory code base. For Active Directory account stores, you can map group claims onto one or more groups in Active Directory; with an ADAM store, you must supply the name of an attribute on the user object and a value to look for to indicate whether the group claim should be sent for a given user. Each custom/identity claim is mapped directly onto a single attribute of the user object in the directory.

The beauty is that an administrator can take advantage of existing identity data in the company directory to integrate with partners and, in many cases, there's no need to write any code.

Anonymity

One thing that's really interesting about claims-based systems is that there's often no need to send the user's name at all. Perhaps all the partner cares about is whether or not the user is authorized to perform the request she's making. But it's often convenient for the partner to at least have some identifying handle from which he can hang personalization data or other state for the user.

To support this anonymous mode of operation, there's a special built-in transformation for identity claims called enhanced identity privacy. If you're an account partner and would like to ensure that your users remain anonymous with a particular resource partner, just enable this feature for that partner in your trust policy, and instead of seeing

alice@northwindbikes.com
the partner will instead see something like this:
tQZPfFuodGysa4t40oj+kM2vBIU=@northwindbikes.com

To generate this handle, the federation service hashes a combination of the actual identity claim, the resource partner's URI, and a salt value that only your federation service knows (this is called the "privacy key" in ADFS terminology). Including the partner's URI ensures that handles differ for each resource partner, preventing them from colluding to build a dossier of data on a user. The privacy key is included to make dictionary attacks difficult.

Despite protecting the identity from the resource partner, this feature still maintains traceability. In the event that an identity needs to be determined, the hashed identity can be found in the resource logs and provided to the account partner administrator, who can use the account event logs to determine which user received that identity.

The Federation Service Proxy

When you install the ADFS federation service, you'll see a new Web application in IIS manager called "adfs." Underneath this, you'll find subdirectories called "fs" and "ls," which stand for Federation Service and Logon Service. The Logon Service is really just the browser-based front end of ADFS; if you poke around under the ls directory you'll see a set of ASPX pages (by the way, you can customize these pages if you like, so it's worthwhile to explore them). This is where the browser is redirected during a federated logon. I've not found the term "Logon Service" anywhere in the published documentation for ADFS, but it's apparent by looking at the code that this is what it's called internally. Under the fs directory you'll find a file called FederationServerService.asmx, the Web service which is the back end of ADFS, technically called a Security Token Service (STS).

The reason these are split up is so that the front end (the Logon Service) can be hosted in a perimeter network (often referred to as a DMZ) and can communicate with the back-end of the federation service via the ASMX Web service, which lives on the internal network. In this configuration the front-end is known as the federation service proxy.

Whether or not your administrator splits up the federation service by using the proxy on a separate machine has little impact on how you write your application, but it's good to know that ADFS was designed to be DMZ-friendly.

ADFS-Enabling Your Web Application

If you've been tasked with building a Web application that supports ADFS logons, you'll need to do a few things. You'll need to know how to set up your web.config file to load and configure the ADFS Web agent. Figure 5 shows the web.config file I used for the sample application for this article.

Figure 5 Sample Web.config (Close Figure 5)
<configuration>

  <configSections>
    <sectionGroup name="system.web">
      <section name="websso" type=

        "System.Web.Security.SingleSignOn.WebSsoConfigurationHandler,
         System.Web.Security.SingleSignOn, Version=1.0.0.0,
         Culture=neutral, PublicKeyToken=31bf3856ad364e35, Custom=null"/>
    </sectionGroup>
  </configSections>

  <system.web>

    <!-- we’re not using any of the standard ASP.NET auth techniques,
         we’re using ADFS! -->
    <authentication mode="None" />
    <customErrors mode="Off"/>
    <sessionState mode="Off" />

    <!-- pull in ADFS assemblies -->
    <compilation debug=‘true’ defaultLanguage=‘c#’>
      <assemblies>
        <add assembly="System.Web.Security.SingleSignOn, Version=1.0.0.0,
                       Culture=neutral, PublicKeyToken=31bf3856ad364e35,
                       Custom=null"/>
        <add assembly="System.Web.Security.SingleSignOn.ClaimTransforms,
                       Version=1.0.0.0, Culture=neutral,
                       PublicKeyToken=31bf3856ad364e35, Custom=null"/>
      </assemblies>
    </compilation>

    <!-- pull in the module containing the ADFS web agent -->
    <httpModules>
      <add name="ADFS Web Agent" type=
          "System.Web.Security.SingleSignOn.WebSsoAuthenticationModule,
           System.Web.Security.SingleSignOn, Version=1.0.0.0,
           Culture=neutral, PublicKeyToken=31bf3856ad364e35,
           Custom=null" />
    </httpModules>

    <!-- the web agent looks here for its configuration -->
    <websso>
      <authenticationrequired />
      <eventloglevel>55</eventloglevel>
      <auditsuccess>2</auditsuccess>
      <urls>
        <returnurl>https://resource.local/web/</returnurl>
      </urls>
      <cookies writecookies="true">
        <path>/web</path>
        <lifetime>240</lifetime>
      </cookies>
      <fs>https://resource.local/adfs/fs/federationserverservice.asmx</fs>
    </websso>
  </system.web>

  <system.diagnostics>
    <switches>
      <!-- enables full debug logging -->
      <add name="WebSsoDebugLevel" value="255" />
    </switches>
    <trace autoflush="true" indentsize="3">
      <listeners>
        <!-- either create a c:\logs directory and grant Network Service
             permission to write to it, or remove this listener -->
        <add name="MyListener" type=
            "System.Web.Security.SingleSignOn.
                 BoundedSizeLogFileTraceListener,
             System.Web.Security.SingleSignOn, Version=1.0.0.0,
             Culture=neutral, PublicKeyToken=31bf3856ad364e35,
             Custom=null"
             initializeData="c:\logs\webagent.log" />
      </listeners>
    </trace>
  </system.diagnostics>
</configuration>

Once your web.config file is set up, you should be able to accept logins from ADFS account partners. The next step is to start making security decisions based on claims.

There are actually two different ADFS Web agents. One is called the Web Agent for Windows NT® Token-based Applications. This Web agent will generate a real Windows logon session for the remote user. It generates a login from the user principal name via the S4U Kerberos extensions I described in my April 2003 Security Briefs column (a custom authentication package is used if not running in Windows Server 2003 native mode). The benefit is that you can impersonate this logon and pass off authorization duties to existing secure resource managers behind you, such as the file system or SQL Server™. (If these resources are remote, you'll also need to enable protocol transition in Active Directory.) One of the biggest drawbacks is that you're going to have to provision a user account in your domain for each incoming user, which destroys a lot of the benefit of using ADFS in the first place! (However, users will not need to know their resource account password, and most likely they won't even know there is another account provisioned for them.)

If you want to buy into federation, you'll need to build your app to be claims aware, and you won't be able to rely on impersonation because you won't have a Windows domain account representing the users from your account partners. You'll use the Web Agent for Claims-Aware Applications, which doesn't attempt any mapping onto Windows accounts, and hands you the incoming claims via HttpContext.User. This is the approach I took in my sample.

Writing code to check claims is not at all difficult. In fact, if you primarily rely on group claims, you need not think much about claims at all. Just continue to use HttpContext.User.IsInRole to check for the presence or absence of group claims, treating them as application roles. This means you can also use controls like the ASP.NET LoginView, which bases its output on roles discovered via the HttpContext.User property.

If you want to read the value of custom claims, you'll need to write a bit of new code. This sample looks for a claim called Title:

string GetTitle()
{
    SingleSignOnIdentity id = (SingleSignOnIdentity)User.Identity;
    SecurityPropertyCollection c =
        id.SecurityPropertyCollection.GetCustomProperties("Title");
    return (1 == c.Count) ? c[0].Value : string.Empty;
}

One interesting piece of data that's passed along is the original method used to authenticate the client in her home realm. ADFS actually supports four authentication techniques: Windows integrated, SSL client certificate, Basic authentication, and ASP.NET Forms authentication. You can discover which was used via the AuthenticationMethod property on the SingleSignOnIdentity object. Don't confuse this with the original AuthenticationType property on IIdentity; you'll find that it always returns WebSSO, which basically means ADFS was responsible for the login.

Other interesting properties include AuthenticatingAuthority, the URI of the account partner that authenticated the client. And it's always a good idea to provide a Log Off button, so you'll find the SignOutUrl property useful in this regard.

The sample application for this article dumps out all of the public properties of SingleSignOnIdentity into a table, as shown in Figure 6. It also dumps out the collection of security properties, which basically shows all of the claims that were sent in their raw form. Before you can run this application, though, you'll need to set up ADFS.

Figure 6 Sample Application
Figure 6 Sample Application(Click the image for a smaller view)
Figure 6 Sample Application
Figure 6 Sample Application (Click the image for a larger view)

Getting Started with ADFS in the Lab

If you'd like to experiment with ADFS, you'll need to make time to set up an appropriate environment. I took notes as I built my claims-based B2B setup and came up with a reasonably quick way to put together a set of three Virtual PC images that represent an account partner with a single client machine along with a resource partner. I've put these notes up on my wiki. Some notes may seem brief, but they'll make sense if you've worked with Virtual PC and Windows. I walk you through setting up domains and using SYSPREP, which may be new to you. You can edit the wiki by double-clicking the page, so feel free to add comments for fellow readers; just try to be brief in the spirit of my original notes.

There is a sample application up on the wiki as well, including the config file you can use to test with. Just copy the default.aspx and web.config files into a virtual directory on the resource partner image; then you should be able to log into the client machine and point your browser at the Web application in the partner application to log into it.

Keith Brown is a cofounder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight's Applied .NET Security course, as well as several books, including the .NET Developer's Guide to Windows Security, available in print and on the Web. Learn more at www.pluralsight.com/keith.


Smart Storage: Protect Your Data Via Managed Code And The Windows Vista Smart Card APIs

Smart Storage

Protect Your Data Via Managed Code And The Windows Vista Smart Card APIs

Dan Griffin
This article is based on a prerelease version of Windows Vista. All information contained herein is subject to change.
This article discusses:
  • Windows smart card programming basics
  • Road map for a sample smart card app
  • Writing managed wrappers for smart card functionality
  • Transaction management for smart cards
This article uses the following technologies:
Windows Vista, C++, C#
Code download available at: SmartCards2006_11.exe (2079KB)
Contents
Windows Smart Card Programming
Smart Card Evolution
Sample App Roadmap
WinSCard API Wrapper
GetSmartCard Helper Routine
Card Module API Wrappers
Handling CardAcquireContext
Transaction Management
Card Module Interface Design Rationale
Using CLR Crypto
Dependencies and Testing

T
he smart card concept—basically, a credit card with an embedded microchip—is around 30 years old. But today's emphasis on security is driving companies and governments alike to take a fresh look at some old ideas.

Smart cards are a compelling alternative to the weak link of authentication systems: passwords. The industry is hungry for password-replacement technologies. Via an embedded cryptographic processor, smart cards provide a very secure and easy-to-use authentication mechanism.

However, smart card deployments introduce their own challenges. The industry needs better products for deployment and management of complex authentication technologies. In his keynote address at the RSA 2006 conference, Bill Gates demonstrated Microsoft Certificate Lifecycle Manager, a product that takes advantage of the APIs discussed in this article.

The industry is also beginning to respond to consumer privacy-protection demands. Consumers need a way to protect privileged information; for example, online banking in the US will soon require strong authentication. And down the road, consumers will use technologies such as Windows® CardSpace to select what personal information they reveal in a variety of online transactions. For example, during online banking, I may cryptographically prove that my Social Security Number (SSN) is bound to my identity, but there's no reason for me to share my credit card number. In contrast, I'll authorize my credit card number, but not my SSN, to be securely shared with an e-commerce site.

Microsoft has recognized the pivotal role of smart cards in its platform security strategy. Developers need to understand how smart card-aware applications work and what facilities the Windows operating system exposes to make life easier.

Windows Smart Card Programming

Windows has supported the Personal Computer/Smart Card (PC/SC) software stack for about a decade (see www.pcscworkgroup.com for more information). PC/SC functionality is exposed to applications via the Windows Smart Card (WinSCard) client API, implemented in winscard.dll and, to a lesser degree, scarddlg.dll. The client API allows host applications to communicate with smart cards indirectly via the Smart Card Resource Manager service, also known as SCardSvr.

The resource manager model is used because the smart card is a shared resource, and multiple applications may attempt to communicate with the card at any given time. Communication with the card must be serialized. Therefore, a simplified, database-like locking model is enforced by the resource manager. In PC/SC, holding the lock for a given smart card reader (also known as an Interface Device or IFD) is known as holding the transaction. The term "transaction" in this context causes some confusion, though. For example, there's no such thing as rollback or de-commit; PC/SC can't automatically undo changes made by the application within a transaction.

An alternate description of PC/SC can be found in the Platform SDK documentation for SCardEstablishContext, SCardBeginTransaction, and other WinSCard functions. A host application uses these routines to establish a pipe to the resource manager, obtain an exclusive lock on a smart card reader, send a sequence of commands to the card, and release the lock.

The PC/SC component stack is summarized in Figure 1. Note that the PC/SC client code is loaded in the application process. The resource manager is a system service, separate from the client. The reader device driver is the only component of the PC/SC stack that operates in kernel mode. The reader drivers are loaded exclusively by the resource manager to prevent applications from bypassing the transaction model. That's a critical aspect of the PC/SC security model. Finally, the APIs discussed in this article are available only to user mode callers. Unfortunately, there's no kernel mode resource manager.

Figure 1 PC/SC Component Stack
Figure 1 PC/SC Component Stack

The rest of the article will focus on the top portion of the stack: the host application and the PC/SC client API. I'll next discuss the limitations of this programming model and how it's evolving.

Smart Card Evolution

To date, the typical uses for smart cards on Windows have been authentication-related. For example, they can be used instead of a password to log on. These smart card authentication schemes are cryptography-based. As a result, adding support for a new smart card in Windows has historically required vendors to implement a plug-in known as a CSP (Cryptographic Service Provider), which implements the interface known as Crypto API 1.0. Unfortunately, CSPs are notoriously complicated to write. Furthermore, since the CSP interface was intended only for low-level cryptographic operations, it is not capable of exposing the rich new features of modern smart cards. These features include increased data storage capacity, high-bandwidth I/O, on-card applets, and new crypto algorithms. The lack of an application-level programming model for those capabilities has been a thorn in the side of hardware vendors looking for differentiation in the market and for programmers who want to write for as wide a variety of hardware as possible.

Thankfully, the situation is greatly improved in Windows Vista™ due to a new plug-in model known as the Card Module API. The great thing about the new API is that it allows smart-card-aware applications to be card-type-agnostic. Let's look at an example.

My sample is a Windows digital rights management (DRM) application that uses a smart card to store a digital media license. The digital license consists of a digitally signed, compressed XML blob. In interacting with the smart card over the lifetime of the license, the host application executes some simple commands on the card: create a file to store the license, write the file, read the file, and delete the file. Each command is equivalent to a different sequence of bytes. For example, if the application determines that the file known to the smart card as "02" (many smart cards have very simple file systems) contains the license blob, then a relevant command/byte-sequence could encode: "read the first 128 bytes of file 02." Each command is sent to the card via the WinSCard function SCardTransmit. I'll come back to that in a second.

One problem in generalizing a smart-card-aware application is that different vendors' cards use different command encodings. Granted, there has been some standardization, such as ISO 7816, and the ISO encoding for a basic operation such as reading a file may indeed work on a wide variety of cards. But generally this is not something that developers can count on. Hence, the DRM application in this example would have to be modified to support each new smart card type. In my experience, the application code that builds those smart card command byte sequences is messy and tough to maintain.

The new card module API addresses this card incompatibility issue by providing a familiar file-system-like interface, plus some additional routines to expose the crypto-related authentication requirements mentioned earlier. For example, if the application is using the old model of including per-card encodings for the command "read the first 128 bytes of file 02," the pseudocode might look something like Figure 2.

Figure 2 Pseudocode for Per-Card-Type Command Handling (Close Figure 2)
if (cardType == A) {
    // Build read-file transmit buffer for card type A
    ...

    // Send the command to the card
    SCardTransmit(
        cardHandle,          // PC/SC context handle
        pciInputPointerA,    // Protocol Control Information (in)
        transmitBufferA,     // Our read-file command
        transmitBufferLength,// Read-file command length
        pciOutputPointerA,   // Protocol Control Information (out)
        receiveBuffer,       // Data response from card
        receiveBufferLength);// Length of data response

    // Interpret the response from this card type
    ...
}
else if (cardType == B) {
    // Build read-file transmit buffer for card type A
    ...

    // Send the command to the card
    SCardTransmit(
        cardHandle,          // PC/SC context handle
        pciInputPointerB,    // Protocol Control Information (in)
        transmitBufferB,     // Our read-file command
        transmitBufferLength,// Read-file command length
        pciOutputPointerB,   // Protocol Control Information (out)
        receiveBuffer,       // Data response from card
        receiveBufferLength);// Length of data response

    // Interpret the response from this card type
    ...
}

In contrast, by using the new API, I can simply replace the whole block with a single function call:

CardReadFile(
    contextPointer,  // PC/SC context handles
    NULL,            // Directory in which requested file is located
    "license",       // File name to read
    &dataOut,        // Contents of the requested file
    &dataLengthOut); // Size of the file contents
If you're thinking that the new API looks much simpler than the old way, I agree. But don't worry; there's still some work to do!

So far, I've described some of the basics of the PC/SC software stack and the card module API. I'm almost ready to tackle the sample code of the DRM application. Since none of the smart-card-related interfaces I've described are exposed to Microsoft® .NET Framework code, wouldn't it be cool if I could combine the power of the card module API with the convenience of .NET?

I figured that the most generally accessible solution would be to provide code wrappers that exercise the necessary WinSCard and Card Module routines via P/Invoke (a mechanism for calling native DLL exports from managed code). Then I could demonstrate the whole application in managed code. I found out it would be easier to write some additional native wrapper code to simplify this task. In retrospect, implementing the managed interfaces without detailed knowledge of the native API would have been quite difficult.

I'll first outline what the sample application needs to do. Then I'll discuss the wrappers and P/Invoke interfaces in detail.

Sample App Roadmap

First, the application will locate and bind to an inserted smart card. If no smart card is inserted, the user will be prompted to insert one. These operations require the WinSCard API, so this portion of the application will depend heavily on P/Invoke and the native wrappers.

Next, I'll open a handle to a cryptographic key pair on the card. If a suitable existing key pair isn't available, I'll try to create a new one. Fortunately, the required crypto-related functionality is already exposed by the CLR, so this is easy. There is one potential hurdle for the crypto portion of the application: the new Base Smart Card Crypto Service Provider must be installed since it's the most reliable way to interface with card module-based cards.

I'll create some sample XML data to represent a digital media license. Then I'll encrypt that data with the crypto key pair. Again, the required XML and crypto-related functionality are already available to use via the CLR. I'll save the data and signature to the card via the card module. This will require new native wrappers and P/Invoke interfaces for the native routines. Next, since this application is intended to be a test, I'll do the reverse of the previous steps. That is, I'll read back the file I just wrote to the card. Finally, I'll decrypt the data via the CLR crypto routines.

Now that I've summarized the behavior I'd like to see in the application, let's categorize the functional requirements. These three building blocks are all I need: wrappers for the WinSCard API, wrappers for the Card Module API, and the existing CLR crypto routines. Once those are implemented and understood, the writing of the application itself is easy! Let's look at each building block in detail. I recommend that you download the sample C# code to follow along, although the text is peppered with comments about smart card programming fundamentals in order to keep things interesting.

WinSCard API Wrapper

The first thing the application does is call into the .NET Crypto API to find a suitable key pair. However, I'm going to discuss that portion after describing the lower-level smart card API calls that the application makes, in order to make a point about the overall architecture.

To interact directly with the smart card via the card module API, the first thing I need to do is acquire PC/SC handles for a suitable card. (Please note that each of the native routines referenced here is documented in the Platform SDK, so I won't provide detailed usage for every public routine that's consumed here.) To bind to a smart card from a managed application, I need to expose several native APIs.

The foundation for just about any smart card application on Windows is SCardEstablishContext. This PC/SC routine, exported from winscard.dll, establishes a handle that allows the application to interact with the smart card resource manager. The function is pretty simple, and for these easy ones I didn't need to implement an additional native wrapper. A P/Invoke interface was sufficient:

[DllImport("winscard.dll")]
public static extern Int32 SCardEstablishContext(
    UInt32 scope,
    [MarshalAs(UnmanagedType.AsAny)] Object reserved1,
    [MarshalAs(UnmanagedType.AsAny)] Object reserved2,
    ref IntPtr pcscContext);

The second PC/SC routine exercised by the sample application, SCardUIDlgSelectCardW, was by far the hardest one to expose in .NET code. There are two reasons for this. First, the function uses a single C-style struct as its parameter. (See OPENCARDNAME_EXW defined in winscard.h in the Platform SDK.) The struct includes, among a number of other fields, two Unicode character arrays that act as both input and output. Marshaling this struct correctly took some trial and error. I should note that this struct has, in turn, an optional embedded struct: OPENCARD_SEARCH_CRITERIAW, also in winscard.h. The nested struct is just as complicated as the first, if not more so. For this sample, I chose not to expose the advanced capabilities of SCardUIDlgSelectCardW that depend on the second struct.

The second challenge in exposing SCardUIDlgSelectCardW is that the first data member of OPENCARDNAME_EXW must be initialized to the size, in bytes, of the struct. I explored three options for initializing that field to the correct value. First, I tried to hardcode it—that is, determine how big the structure is (for example, via a debugger and a simple test application in native code) and always initialize the field to that value in managed code. It's nice to have portable code, though, and this struct will not be the same size in a 64-bit environment.

Second, you can use one or more of the CLR marshaling primitives, such as Marshal.SizeOf, to determine at run time how big the marshaled structure will be. Unfortunately, in my testing, that particular routine threw an exception indicating that the marshaled size of certain data members couldn't be determined. Moreover, unless this approach would produce the exact same value as the native sizeof, it's not a good idea.

Third, I implemented a native wrapper that accepts a partially initialized struct, sets the dwStructSize field, and then calls SCardUIDlgSelectCardW. Then I exposed the native wrapper to managed code via P/Invoke. You've probably already guessed that I chose this option. While I knew that I wanted to package the native wrapper as a separate DLL for flexibility, and adding the management overhead of an additional DLL to support a single API would have been a tough pill to swallow, I already knew that the Card Module routines were going to require a similar solution. In other words, I figured that supplying a separate native DLL was inevitable. I'll cover the card module routines in the next section.

In the sample code, MgSc.dll (an abbreviation for Managed Smart Card) exports the native wrapper routines. The code for the wrapper routines, including MgScSCardUIDlgSelectCardW, is in MgSc.cpp. The P/Invoke class for all of the native wrappers in that DLL is also called MgSc; it contains P/Invoke stubs that look similar to the following:

[DllImport("mgsc.dll", CharSet=CharSet.Unicode)]
public static extern Int32 MgScSCardUIDlgSelectCardW(
    [MarshalAs(UnmanagedType.LPStruct)] [In, Out]
    PCSCOpenCardNameWEx ocnwex);
After getting this API working, the rest seemed really easy!

GetSmartCard Helper Routine

Figure 3 shows an excerpt of the GetSmartCard helper routine found in the sample code. Notice that, if MgScSCardUIDlgSelectCardW is successful, it implies that the cardHandle member of the PCSCOpenCardNameWEx parameter is now initialized and refers to an inserted smart card. In other words, we're ready to party on that card.

Figure 3 GetSmartCard Helper Routine (Close Figure 3)
static Int32 GetSmartCard(
    [In, Out] ref MgScContext mgscContext,
    [In, Out] ref IntPtr pcscContext,
    [In, Out] ref IntPtr cardHandle)
{
    Int32 result = 0;
    PCSCOpenCardNameWEx ocnwex = new PCSCOpenCardNameWEx();
    bool cardIsLocked = false;
    string readerName;
    UInt32 readerNameLength = 0, cardState = 0, cardProtocol = 0;
    byte [] atr;
    UInt32 atrLength = 0;

    try
    {
        // Get a context handle from the smart card resource manager
        if (0 != (result = PCSC.SCardEstablishContext(
            PCSCScope.SCARD_SCOPE_USER, null, null, ref pcscContext)))
                throw new Exception("SCardEstablishContext");

        // Get a handle to a card, prompting the user if necessary
        ocnwex.flags = PCSCDialogUI.SC_DLG_MINIMAL_UI;
        ocnwex.pcscContext = pcscContext;
        ocnwex.shareMode = PCSCShareMode.SCARD_SHARE_SHARED;
        ocnwex.preferredProtocols = PCSCProtocol.SCARD_PROTOCOL_Tx;
        ocnwex.reader = new string(
            (char) 0, (int) readerAndCardBufferLength);
        ocnwex.maxReaderLength = readerAndCardBufferLength;
        ocnwex.card = new string(
            (char) 0, (int) readerAndCardBufferLength);
        ocnwex.maxCardLength = readerAndCardBufferLength;

        if (0 != (result = MgSc.MgScSCardUIDlgSelectCardW(ocnwex)))
            throw new Exception("SCardUIDlgSelectCardW");

        // Lock the card
        if (0 != (result = PCSC.SCardBeginTransaction(ocnwex.cardHandle)))
            throw new Exception("SCardBeginTransaction");

        cardIsLocked = true;

        // Get the ATR for the selected card
        if (0 != (result = PCSC.SCardStatusW(ocnwex.cardHandle, null,
            ref readerNameLength, ref cardState, ref cardProtocol,
            null, ref atrLength)))
                throw new Exception("SCardStatusW");

        readerName = new string((char) 0, (int) readerNameLength);
        atr = new byte [atrLength];

        if (0 != (result = PCSC.SCardStatusW(ocnwex.cardHandle,
            readerName, ref readerNameLength, ref cardState,
            ref cardProtocol, atr, ref atrLength)))
                throw new Exception("SCardStatusW");

        // Get a card module handle for this card
        mgscContext = new MgScContext();
        if (0 != (result = (int) MgSc.MgScCardAcquireContext(
            mgscContext, pcscContext, ocnwex.cardHandle,
            ocnwex.card, atr, atrLength, 0)))
                throw new Exception("MgScCardAcquireContext");

        cardHandle = ocnwex.cardHandle;
    }
    finally
    {
        if (0 != result)
        {
            // Something failed, so we need to cleanup.
            if (cardIsLocked)
                PCSC.SCardEndTransaction(ocnwex.cardHandle,
                    PCSCDisposition.SCARD_LEAVE_CARD);

            if ((IntPtr)0 != ocnwex.cardHandle)
                PCSC.SCardDisconnect(ocnwex.cardHandle,
                    PCSCDisposition.SCARD_LEAVE_CARD);

            if ((IntPtr)0 != ocnwex.pcscContext)
            {
                PCSC.SCardReleaseContext(ocnwex.pcscContext);
                pcscContext = (IntPtr)0;
            }
        }
    }
    return result;
}

The first thing I do in GetSmartCard with the returned card handle is call SCardBeginTransaction. This grants exclusive access to the smart card, preventing other applications (or even other card handles within this process) from interacting with the card. If the GetSmartCard routine returns successfully, the transaction is still held, and I'll keep the card locked up until I call the managed Crypto API routines. The reason I release the lock at that time is subtle as I'll explain in a moment.

The second thing I do with the card handle is use it to query PC/SC for the ATR (Answer to Reset) string of the selected smart card. The bytes of the ATR constitute a per-card type identifier that facilitates low-level protocol negotiation between the card and the smart card reader device driver. The card ATR is also used as the mapping, via the system registry, between the card and its corresponding card module (as well as between the card and its corresponding CSP, although those details are abstracted by Crypto API). In other words, in order to load the proper card module, I must first know the ATR of the card I want to talk to.

Since I already have the card handle, I call the SCardStatus API via P/Invoke to get the ATR. You can see in GetSmartCard that I call SCardStatus twice: once to query for the size of its two variable-length output parameters and again with buffers of sufficient size to accept those outputs. But if you take a look at the Platform SDK documentation for SCardStatus, you'll see that the API can optionally allocate the output buffers on behalf of the caller.

Take the ATR output buffer as an example. In native code, the caller can set the buffer-length input/output parameter, pcbAtrLen, to point to the value SCARD_AUTOALLOCATE, which is defined in winscard.h to be -1. The pbAtr parameter is then passed as an LPBYTE *, cast to an LPBYTE. The returned buffer is freed via SCardFreeMemory. The same semantics are available for a number of PC/SC functions. In fact, whenever I write PC/SC-related native code, I make heavy use of SCARD_AUTOALLOCATE for convenience. However, I chose not to expose that capability in the wrapper code due to the additional complexity of defining such chameleon-like behavior and getting it marshaled correctly.

Once I've got the ATR, I'm ready to map the selected card to its card module. I decided to abstract that operation into one of the wrapper functions that I implemented to expose the card module to managed code. That's the topic of the next section.

Card Module API Wrappers

The native card module interface is defined in cardmod.h, which is included in current Windows Vista releases of the Platform SDK. If you take a look at the CARD_DATA struct, defined near the end of the header file, you'll notice that there are a couple of complications in exposing the card module routines in .NET code. Similar to the memory-management-related challenge offered by SCARD_AUTOALLOCATE in some of the PC/SC routines, a few of the card module routines allocate dynamically sized output buffers on behalf of the caller. But, unlike SCARD_AUTOALLOCATE, the card module memory allocation behavior isn't optional. Take, for example, CardReadFile, which I'll need to use in the sample application when I read the digital license file from the smart card. I'll end up calling CardReadFile via a function pointer, the type of which is defined as follows.

typedef DWORD (WINAPI *PFN_CARD_READ_FILE)(
__in                       PCARD_DATA  pCardData,
__in                       LPSTR pszDirectoryName,
__in                       LPSTR pszFileName,
__in                       DWORD dwFlags,
__out_bcount(*pcbData)     PBYTE *ppbData,
__out                      PDWORD pcbData);

Now take a look at two members of the CARD_DATA struct. The first is the heap allocator callback to be used by the card module when creating output buffers. The second is the PFN_CARD_READ_FILE function pointer to be initialized by the card module during CardAcquireContext for subsequent use by the calling application.

// These members must be initialized by the CSP/KSP before
// calling CardAcquireContext.
...
PFN_CSP_ALLOC           pfnCspAlloc;
PFN_CSP_FREE            pfnCspFree;
...
// These members are initialized by the card module
...
PFN_CARD_READ_FILE      pfnCardReadFile;

In summary, during the CardReadFile call, the card module will do more or less the following: allocate a sufficiently large buffer via pfnCspAlloc, read the requested file from the card via SCardTransmit, set pcbData, and return. Once the caller is finished with the returned data buffer, it calls pfnCspFree.

Rather than crafting a set of P/Invokes, I knew it was going to be quicker to get things working with a native wrapper for each card module routine. For example, in the CardReadFile case, the .NET application calls the native wrapper once to determine the size of the output buffer and then again with a buffer of sufficient size. This is a significant performance tradeoff, since the application is reading the file from the smart card twice. It's likely that the situation could be improved by implementing delegates for the heap callbacks.

Handling CardAcquireContext

Let's return now to the flow of the sample digital license application. Recall that, via PC/SC routines, I've now got a card and its ATR. Based on the ATR, I look up the name of the corresponding card module DLL in what's known as the Calais database, which is the data stored under HKLM\Software\Microsoft\Cryptography\Calais in the system registry. "Calais" was chosen as a project codename when the PC/SC stack was originally being implemented at Microsoft. I like to think it's not totally coincidental that, by some accounts, the smart card was invented by a Frenchman.

The Calais database lookup is accomplished via SCardGetCardTypeProviderName, passing the SCARD_PROVIDER_CARD_MODULE value defined in cardmod.h. Instead of exposing that specific PC/SC routine to the managed world via a P/Invoke interface, I decided instead to roll it into MgScCardAcquireContext, the native code wrapper for CardAcquireContext. Why? Well, let's look forward a few (native code) operations. The next step is to take the DLL string name returned from the Calais database and pass it to LoadLibrary. The resulting HMODULE must be maintained for the lifetime of the context with the card module; I certainly don't want the card module DLL unloaded out from under me. Then I call GetProcAddress to find the CardAcquireContext export. Finally, I initialize the CARD_DATA struct with the native callbacks that the card module will be expecting, including the heap allocation routines.

In summary, there's little benefit in trying to get all of this native-code-related stuff working via .NET code when it's only going to be exercised when preparing to call CardAcquireContext. Better just to roll it into the native wrapper for that routine.

Consider what the card module does under the covers. The steps taken by a given card module in processing CardAcquireContext will depend on the type of card. If the card is based on the ISO command set, there may be no need for the card module to send any commands to the card at this time. Instead, the card module may verify that the specified ATR (input by the caller via the CARD_DATA struct) is indeed supported. It may then set up some internal context state and attach it to the pvVendorSpecific member of the caller's CARD_DATA. On the other hand, newer cards may be proxy- or virtual-machine-based. Sun's Java card is a prominent example. In that case, it's likely that the card module will send a command to the card via SCardTransmit to instantiate a card-side card module API handler applet.

Transaction Management

Let's return now to the sample code. If the GetSmartCard helper routine returns successfully, the transaction on the card is still held. In other words, the card handle that is output from GetSmartCard still has exclusive access to the card. There's more to this than meets the eye. Notice that the overall flow of the sample app is to first encrypt the data via managed Crypto API, then do all of the necessary card module work including writing and reading back the file, and finally decrypt the data via managed Crypto API.

This order of operations highlights a few important considerations of what I call transaction management. First, when calling the PC/SC and card module routines, I want to perform as many operations as needed while minimizing the number of separate transactions. In other words, I'll minimize the number of times I release and re-acquire the exclusive communications link with the smart card. By making all of those calls in a single transaction, I eliminate the possibility of another application intervening and leaving the card in an unknown state.

However, that strategy only goes so far; the second consideration is that Crypto API does its own transaction management. For example, after I read the encrypted file back from the card, I must call SCardEndTransaction before calling managed Crypto API to decrypt the file. Otherwise the call to RSACryptoServiceProvider.Decrypt would block forever on an internal SCardBeginTransaction deep within the CSP.

A third consideration of transaction management, and an important security consideration as well, is that the card must remain locked while authenticated. Via the card module interface, the card becomes authenticated when the correct PIN is transmitted via CardAuthenticatePin. I've exposed this through a P/Invoke interface for MgScCardAuthenticatePin. The PC/SC security model is such that applications that want to perform privileged operations on the card are expected to use the following sequence.

  1. Lock the card via SCardBeginTransaction.
  2. Authenticate to the card. I'm using CardAuthenticatePin to accomplish this.
  3. Perform the privileged operations. In this example, I first attempt to delete a file from the card and then attempt to write a file. Both operations require authentication.
  4. De-authenticate the card.
  5. Unlock the card via SCardEndTransaction.

This way, unauthorized applications are prevented from sending privileged commands to a card that has been authenticated on behalf of an authorized application.

Some cards don't support de-authentication directly. Instead, the application must reset the card via PC/SC. Resetting the card returns it to its initial, unauthenticated state. But resetting effectively "reboots" the smart card, which can adversely affect the performance of the next host application that attempts to communicate with the card. Therefore, the card module interface has been designed with an optional routine to avoid this problem and the resulting performance pitfalls.

In my native wrapper function MgScCardDeauthenticate, if the target card module doesn't implement the CardDeauthenticate function, the wrapper will return ERROR_CALL_NOT_IMPLEMENTED. If the de-authenticate call fails for any reason, the managed application will make a note to itself that the card must be reset before the current transaction is released.

if (0 != (result = (Int32) MgSc.MgScCardDeauthenticate(
    mgscContext, MgScUserId.User, 0)))
        cardDispositionOnRelease = PCSCDisposition.SCARD_RESET_CARD;
This is consistent with the transaction management security model that I described earlier.

Card Module Interface Design Rationale

The final details of the interaction between the application and the smart card are the simplest, and have already been mentioned briefly. Namely, to implement the storage-related logic of placing the digital license file on the card, I first call the native MgScCardDeleteFile via P/Invoke. For simplicity I don't check the return value, since if this is the first time executing this application with a given card the file won't exist, in which case the API is expected to return an error.

I next call the wrappers for CreateFile and WriteFile. There's an interesting lesson here on the design rationale for the card module API. Separate CardCreateFile and CardWriteFile routines offer a simple way to replicate the behavior of the CREATE_NEW and OPEN_EXISTING semantics offered by the familiar Windows CreateFile API, including the corresponding error conditions. This is important because it lowers the bar for programmers who now can interact with the card using the same patterns that they've already mastered for file I/O in general.

But unlike the Windows CreateFile function, CardCreateFile takes a byte count parameter as input. Some card modules may ignore that parameter, but it's a good idea for applications to pass in the lowest maximum size of the file to be created. The reason for this is that some cards must preallocate the entire file at creation time. On the other hand, you don't want to waste space. One potentially confusing ramification of this design is that CardReadFile may return the whole allocated size of the file (the potentially larger value passed to CardCreateFile), not just the potentially smaller amount that's been written via CardWriteFile.

After the encrypted license has been written to the card, I read it back via MgScCardReadFile. A production-quality application wouldn't perform this step, but I do it for demonstration purposes and to test the interface. The next step after that is to decrypt the file, again as a sanity check, and compare it to the original value.

Using CLR Crypto

Notice that the first thing I do in the sample code is bind to a smart-card-based RSA key pair via the .NET Crypto API. (See the documentation for class RSACryptoServiceProvider in the System.Security.Cryptography namespace.)

Although the abstractions and power provided by .NET are nice, I'm a firm believer in trying to understand what's happening under the covers. Since the .NET Crypto API is based on the native Crypto API, I can make some assumptions about this code, adapted from my GetKey helper routine:

CspParameters cspParameters = new CspParameters(
    1, "Microsoft Base Smart Card Crypto Provider");
cspParameters.Flags = CspProviderFlags.UseExistingKey;
cspParameters.KeyContainerName = "";
rsa = new RSACryptoServiceProvider(cspParameters);

I can assume that this code results in, at the minimum, calls to CryptAcquireContext and CryptGetUserKey. If this fails, I set cspParameters.Flags to zero and try again. That will result in a call to CryptAcquireContext followed by CryptGenKey.

Dependencies and Testing

I can make further assertions about what the CSP is probably doing under the covers. This will facilitate the creation of a component dependency diagram for the sample application. The CSP I'm calling is actually built on top of the card module interface; that's the primary rationale for writing a card module in the first place. And the fact that it's a smart card CSP implies that it's probably establishing a connection to the smart card in a similar way that the sample code does before calling directly into the card module routines.

In other words, before I make the explicit calls to SCardUIDlgSelectCardW, SCardBeginTransaction, SCardStatus, and CardAcquireContext that I discussed earlier, the CSP is making more or less the same calls internally. Especially in light of the amount of setup work required just to call CardAcquireContext, doesn't this duplication of effort seem undesirable? It would be nicer if the required context information for calling directly into the card module could be retrieved from Crypto API, or at least exposed via some shared code.

Figure 4 depicts these dependencies. The bottom layer of the diagram refers to everything in the smart card stack from the winscard.dll client API, over to the resource manager service, down to the driver, and out to the smart card.

Figure 4 SampleApp Dependencies
Figure 4 SampleApp Dependencies

Now that I've discussed some smart card fundamentals as well as the test application in detail, I hope you'll download the sample code and play around with it. To build and run the sample code requires a few specific items:

  • The cardmod.h header from a recent version of the Windows Vista Platform SDK.
  • A card-module-based smart card. That is, a smart card for which the vendor has supplied a card module.
  • A smart card reader. In some cases, such as certain USB keys, the smart card and reader is a single unit.
  • The Base Smart Card Crypto Provider. This can be downloaded via Windows Update. Click the Custom button then select Software, Optional on the left-hand side.

At the time of this article's publication, the latest Platform SDK release is Beta 2; it can be downloaded from www.microsoft.com/downloads. You can drop the headers into your existing Visual Studio® build environment or into an older version of the Platform SDK environment. I did the former and it worked fine.

I'll post updates to my blog at www.jwsecure.com. Reader comments and questions are welcome there as well. Have fun!

Dan Griffin is a software security consultant in Seattle, WA. He previously spent seven years at Microsoft on the Windows Security development team. Dan can be reached via www.jwsecure.com.


Extending SDL: Documenting And Evaluating The Security Guarantees Of Your Apps

Extending SDL

Documenting And Evaluating The Security Guarantees Of Your Apps

Mark Novak
This article discusses:
  • Analyzing the security of application features
  • A deeper look at security requirements
  • A top-down security design extension to SDL
  • Disclosing details to users and experts
This article uses the following technologies:
Security Development Lifecycle (SDL)
Contents
Security Analysis
Top-Down Security Design
Disclosing Details

T
he internal software design and development practice at Microsoft, called the Security Development Lifecycle (SDL), is reaching maturity and starting to be publicized and adopted outside the company. This is helped in part by the success of the book The Security Development Lifecycle by Michael Howard and Steve Lipner (Microsoft Press®, 2006). The success of SDL has been well documented—software developed in keeping with the requirements and recommendations that SDL puts forth is indeed better protected against attack. Yet, for all its documented success, software developed per SDL requirements still fails to answer a basic question many customers ask: what security features does the product deliver?

In this article, I'll present a case study of an extension to SDL which, if adopted, could translate into a much better flow of information between users and designers of the security features of software products. My approach calls for recognizing that security of software is not a compliance artifact. Rather, it is a set of key features that can and should be documented, evaluated, and used when making purchasing decisions. The security features—I like to call them security guarantees of the software—are deep and multileveled. At a high level understandable by most users, they should be described in the customer-facing sales literature. Lower-level technical details targeted at experts should be readily available for inspection by those curious to understand the technology.

Security Analysis

The analysis of the software necessary to dissect its security guarantees will be done, to be sure, by both malicious hackers and legitimate security researchers who will inevitably reverse-engineer it and draw their own conclusions. Unfortunately, in today's software marketplace most customers will only find out that their software is weak if a flaw is discovered, publicized, and exploited. This state of the software industry goes against the fundamental tenet called Kerckhoffs' Principle, also known as Shannon's Maxim, which states that implementation is public: the entire body of security is in the secret key that ensures access only by authorized parties. This is not to say that the software development community excessively relies on security by obscurity or that all software vendors must publish their technology as open source. The underlying problem I want to point out is that today's software is, by and large, developed and distributed in a manner that is most suited to the following two principles:

  • Utilizing sound software engineering practices during implementation
  • Patching vulnerabilities found by hackers after software has been deployed

There's a key element that's being overlooked, though. It relates to acknowledging that every security feature and every part of its implementation designed to ensure a more secure product must roll up to either an explicit feature or an implicit security guarantee made by the product. This makes sense, since it is universally recognized that everything about the software is built around its functional requirements. My first important rule is this:

Design and develop software that addresses the stated as well as implicit security requirements placed upon it by users.

As a security professional, I pay particular attention to the security guarantees of software. Recently I needed to choose a software package to back up my files. As with most users, the data I would like to back up from my laptop contains sensitive, personally identifiable, and financial information (such as my Microsoft® Money file). The data on my laptop is protected with Encrypting File System (EFS), so I am reasonably comfortable with the risk that it might get lost or stolen. The prospect of backing up my data to my beloved Network Attached Storage (NAS) device—a fabulous, expensive piece of hardware and a very attractive target to a potential burglar—is worrisome, however. I would like to ensure that the data stored on the NAS cannot be recovered by a thief. Therefore, software I purchase will need to meet my security requirements and also prove to me that it has done so.

I started out by searching on the Web for reviews and comparisons of different software packages. I discovered dedicated sites on the Internet that listed handy comparison tables. I was able to find information on literally dozens of features regarding backup functionality, such as disk imaging, types of supported backup (incremental, mirror), types of supported media (NAS, CD-ROM, and so on), drive spanning, registry backup, scheduling, and access to locked files.

However, on the topic of security, I found just two rows of information: backup password protection and backup password encryption. Let us go into a little bit of detail about why classifications like this, typical as they are of how vendors treat security features, would make a security-savvy person groan.

For starters, what exactly is "backup password protection"? The authors of the review site wrote: "Backup software allows you to add a password to your backup so you can restrict access." Every single software package reviewed had this so-called feature. But none of the packages explained what this really meant.

What measure of protection do I get if I password protect my backup to a NAS device? Most likely, the protection scheme looks like what you see in Figure 1. It appears the file will be protected as long as you're using the backup software to access it. In the illustration, the user of the laptop must supply the password to the backup software, which will perform the password check and retrieve the backup on the user's behalf. The attacker, however, does not need to bother with such gentlemanly contracts: he can either sniff the network as the file is being transferred or steal (or otherwise gain access to) the file server in order to access the file. Assuming the attacker has access to the unencrypted backup, the password protection arrangement offers no real additional security. At the risk of sounding cantankerous, I question why software developers even bother with such features.

Figure 1 Problems with Password Protecting Backup Files
Figure 1 Problems with Password Protecting Backup Files

On one hand, the password-protection solution makes matters no worse, so why worry? Step back for a moment and think of the users' perceptions. The notion of backup password protection implies a very different sense of the security guarantee than what is actually being delivered. To the uninitiated, password protection means that anyone who doesn't know the password will not be able to retrieve the data. In this case the backup password protection feature is misleading, which brings me to my next rule.

Avoid introducing features into the software that imply, but cannot deliver, defensible security guarantees.

With the password protection option thus discredited, what about the other option—backup encryption? If properly implemented, this can deliver a strong security guarantee, but many things can still go wrong. From the description of the functionality offered by backup software vendors I have evaluated, I couldn't determine if it would really work (and I chose not to reverse-engineer the software to find an answer).

In my job with the Secure Windows® Initiative team at Microsoft, I have long advocated that developers who design and implement logic involving cryptographic algorithms should first prove their ability to write correct cryptographic code. What follows is an incomplete list of potential problems I can see with improperly implemented backup encryption schemes.

Danger #1 Using Weak Cryptography An obvious problem with a weak algorithm, such as 40-bit RC4, is that it can be cracked by brute force in about one day on modern hardware, and that's without any fancy optimizations. Naturally, I would like to use the strongest possible cryptographic algorithm, such as 256-bit Advanced Encryption Standard (AES), and (one can dream!) that the software will seamlessly switch to even stronger cryptography as it becomes available in the future.

Danger #2 Using Strong Cryptography Incorrectly I am guessing that performing an incremental backup of a large set of files would require the backup software to access the contents of the existing backup in something other than sequential fashion. One easy (and even stupid) way of allowing random-access reading and writing of encrypted data is by turning off a default feature of most block ciphers called Cipher Block Chaining (CBC). This insecure mode, called Electronic Code Book (ECB), is illustrated in Figure 2. As you can see, the middle picture, encrypted with ECB, does not conceal much. I would expect a better grade of protection from my backup software!

Figure 2a Plaintext
Figure 2a Plaintext
Figure 2b ECB Encryption
Figure 2b ECB Encryption
Figure 2c CBC Encryption
Figure 2c CBC Encryption

Danger #3 Using Cryptography Inconsistently Would it be possible for an attacker to obtain the names of files and directories being backed up? Even with the files encrypted well, a lot can still be gleaned from the names of files and directories on the drive. If I stole an executive's backup file and found an encrypted file named Leveraged Buyout Plan for Contoso.xls, I may need to read no further—that alone reveals a great deal of information.

The list of possible security issues can be quite long. If the software has an auto-update feature (as most software products do), how easy would it be for an attacker to advertise a false update and install their code on the user's machine? Hopefully the software obtains its updates over a secure channel and also ensures that the update is properly signed by its maker before installing, but when was the last time you had any assurance of this being done correctly?

At this point I will issue my third and final rule, before I go into a proposal of how designers of software can address all three rules collectively.

Publish sufficient information about the design of security features to convince your users that the design is solid.

Granted, not everyone will be able to understand this technical information, so it doesn't necessarily belong on the front page of the sales pamphlet. Software vendors have no qualms about presenting users with voluminous end-user license agreements, but they often neglect to disclose equally important information about the software security design. Today, however, you have only two ways of evaluating security features of a product—to check sources such as CERT advisories for known vulnerabilities or to reverse-engineer the product (possibly violating the EULA in the process) to convince yourself that it is solid. Neither is a good alternative.

Top-Down Security Design

So what can we do about this? The solution I propose is tentatively called the top-down security design extension to SDL. The approach starts early in the development cycle, when functional requirements are being determined. In the case of creating a backup software product, the designers may list several requirements, not all of them dealing explicitly with security. The breakdown of functional requirements may look something like Figure 3.

Figure 3 Functional Requirements for Software (Close Figure 3)

Functional Requirement | Instances
Support for different backup types | Incremental Mirror Entire System
Support for different media types | Local file system NAS Online Web service CD and DVD
Support for encrypting backups | User’s choice of encryption strength and algorithm Support for key escrow
Support for online updates | On-demand
Support for backup scheduling | Scheduled

The next step is to examine this list critically for both explicit and implicit security guarantees. The best way to think about this step is by asking, "What security features would make the customer more likely to buy my product?" This is where security expertise comes in handy, and this is also where functional specification should meet threat modeling—a critical missing link today. Figure 4 expands the list of features by specifying security considerations for a subset of each feature. I did this by adding a new column, listing anticipated end-user requirements and high-level design considerations that enable them. The considerations are prioritized according to their importance to the overall security of the products. For now, I informally put them into three buckets: High, Medium, and Low. High means that the feature is a key functionality and must be present in the product. Medium means a valuable feature that a user should know about and have the option of using. Low means it's OK to skip due to an insufficient additional level of protection or lack of practicality.

Figure 4 Feature-Specific Security Considerations (Close Figure 4)

Functional Requirement | Instances | Security Considerations
Support for different backup types | Incremental, Mirror, Entire system | Medium When users restore backups, they can be sure that the data has not been tampered with. They know that sensitive data will not be compromised if it falls into the wrong hands.
Support for different media types | All types | Considerations are necessarily specific to the type of media involved.
 | Local file system | The backed-up data will be as secure as the original data. For instance, the software will warn users if they attempt to back up files to locations accessible by other users of the computer. High Enable users to ascertain that the destination is strongly ACL’d, especially if the computer has multiple users. Medium Support for encryption is possible, but only relevant if original data is EFS-protected since both are on the same file system.
 | NAS | User data will be safe against disclosure and tampering even if the attacker compromised the remote storage device or the network traffic. Medium Data is safe in transit against eavesdropping and tampering. This is less of a concern if it is already encrypted, depending on the algorithm chosen. High Data is encrypted to defend against NAS device or network transport being compromised. Low Use of ACLs is possible, but probably not very reliable since remote hardware may be under different administrative control.
 | Online Web service | Users know to which remote service they are talking. The service has access to ciphertext only and does not have the plaintext. Thus, servers are unable to hand over user data to others, even by court order. The software will help users choose a strong password. The service will not allow anyone but the authorized users to access, update, or delete their files. High Server authentication should be mandatory, so use transport-level security properly. High No data should leave the machine unencrypted in case backed-up data is ever compromised or handed over to law enforcement. High Strong authentication ensures that data is available only to the originating party.
 | CD and DVD | The data will be safe even if the CD or DVD containing a user’s backup is lost or stolen. Medium Users are made aware of the security implications of losing their backup media. High Opportunity is given to users to create encrypted backups.
Support for encryption of backups | User’s choice of cryptographic algorithm and strength of key | The algorithm chosen may be different depending on the target of the backup. For instance, different performance versus functionality trade-offs may have to be made for local file system versus online backup. High Ensure the key is correctly managed (for example, not stored with the backup file). Medium Provide user feedback on the strength of chosen password or key. High No metadata information (for example, name, attributes, or alternative streams) about backed-up files can be recovered by an attacker. The same strength of encryption is applied to contents as well as metadata. High Strength of cryptographic protection is not weakened because of the need for high-performance random access to data. LowNew algorithms are made available as security science evolves.
 | Support for key escrow | Users can recover files even if they forget the password used to encrypt them. High Key is impractical to recover from escrow without information known only to the user.
Support for online updates | Scheduled, On-demand | When updating backup software, users can be sure the updates came from a trusted party. High Attacker is not allowed to advertise false updates or install malicious code on user’s machine. High Updates are not installed unless signed by the original publisher. Low Updates are limited to the software; they do not alter other parts of the operating environment. High Attacker cannot tamper with scheduling algorithm to cause missed backups. High Attacker cannot downgrade the strength of security of backup operation (for example, by tampering with settings).
Support for scheduled backups | Scheduled, On-demand | Users are in complete control of their backup schedule. An attacker cannot tamper with backup settings to lessen the strength of the protections that users select. Medium Attacker cannot tamper with the scheduling algorithm to cause missed backups. High Attacker cannot downgrade the strength of the security of backup operation (for example, by tampering with settings).

The long list in Figure 4 is far from exhaustive, but it illustrates the point that security considerations and guarantees apply to many features of the software, even those that on the surface are not primarily security features. Listing requirements in conjunction with the associated security guarantees is the key to understanding a full range of protections afforded by the software. Now, if you were in the position of evaluating software packages, wouldn't you agree that being presented with such a list of security guarantees would make your purchasing decision easier? Could it possibly tip the scale in favor of one product over another? I think that it would.

Disclosing Details

I think it's important to mention a couple of additional points: the work does not stop here, and documenting the security guarantees need not necessarily mean disclosing your trade secrets.

Clearly, enumerating the security guarantees thus far is good, but it is used primarily for educating your sales force and producing marketing literature. As we go from the design phase into implementation, it is critical that any additional decisions don't invalidate requirements-level security guarantees already enumerated.

For instance, the scheduling of backups might be done by a dedicated service. This presents additional considerations: is this scheduling service properly isolated from other, potentially malicious, software running on the box? If it is co-located with other service processes, what additional elevation paths does that present? What additional avenues for attack does it open? For instance, someone with the ability to set the system time can now affect the scheduling of backups. Is this an important threat? Clearly, it should be considered and its effects documented, if for no other reason than to have this information for the future, should the security landscape change. What interprocess communication mechanisms are you opening up? How do you secure those channels? Are there existing best practices for securely implementing what you want?

At this point, you can also examine the proposed solution and decide to include additional security guarantees. For instance, backup software may be naturally positioned to detect system and other critical files that have been tampered with since the last backup and alert the user of the incidents.

Now let us consider how this approach coexists with trade secrets. As you further dissect security features—from requirements to design to implementation—more and more technical details emerge. Your cryptographic algorithm for high-performance incremental encrypted backup may indeed be something you would want to patent. Your key escrow protections are something you might want to keep secret or obfuscate (not a good choice, but that's more of a business decision). A security researcher or technical reviewer may decide that this is a critical piece of functionality and ask additional questions. There is nothing in this approach to security engineering that can preclude you from stipulating that the algorithm should be available to reviewers who are willing to sign a nondisclosure agreement. You might want to proactively hire security research firms that would be willing to review your solution and offer their public approval to the strength of your design and implementation. The customers could then place their trust in your product based on the reputation of the research firm, rather than on the first-hand knowledge of the inner workings of a solution.

In summary, the approach presented here is what I believe to be the missing link in SDL: tying secure design and implementation of software directly to the functional specifications of security features in a way that can be understood by customers at any level of depth they require (and that your licensing terms allow). I believe this is a promising step forward in secure design of software.

Mark Novak is a Security Development Lead at Microsoft. He joined Microsoft in 1995 and has spent the last several years focusing on different aspects of software security. Mark can be reached at markpu@microsoft.com. Mark would like to thank Adam Shostack for his suggestions and review of this article.


SQL Security: New SQL Truncation Attacks And How To Avoid Them

SQL Security

New SQL Truncation Attacks And How To Avoid Them

Bala Neerumalla
This article discusses:
  • Delimiting identifiers and strings
  • Useful T-SQL functions
  • Truncation and modification attacks
  • Finding bugs and vulnerabilities
This article uses the following technologies:
SQL Server
Contents
Delimiting Identifiers and Strings
T-SQL Functions
SQL Injection Vulnerabilities
Modification by Truncation
SQL Injection by Truncation
Detecting Injection by Truncation

E
xploits using SQL injection have drawn a lot of attention for their ability to get through firewalls and intrusion detection systems to compromise your data layers. Whether it's a first-order or second-order injection, if you look at the basic code pattern, it is similar to any other injection issue where you use untrusted data in the construction of a statement. Most developers have started mitigating these vulnerabilities in Web front ends by using parameterized SQL queries in conjunction with stored procedures at the back end, but there are some instances where developers still use dynamically constructed SQL, like in the construction of Data Definition Language (DDL) statements based on user input or for apps written in C/C++.

In this article I will discuss some new ideas that can result in either modifying SQL statements or injecting SQL code even if the code has escaped the delimiting characters. I will start with some best practices for constructing delimited identifiers and SQL literals, and then I'll show you new ways attackers can inject SQL code in order to help you protect your applications.

Delimiting Identifiers and Strings

In SQL Server™, there are two kinds of string variables: SQL identifiers that uniquely identify SQL objects like tables, views, and stored procedures, and literal strings that represent data. The way you delimit SQL identifiers is different from the way you delimit data strings. We'll discuss the best practices for constructing dynamic SQL that needs to use these data variables.

You will need to use delimited identifiers if keywords are used for the SQL object names or if special characters are contained in the object names. Say you want to drop a login with the name my_dbreader. To do so, you execute the following statement:

DROP LOGIN my_dbreader

What if you are trying to drop a login that uses DROP, also a keyword, as its name? If you use the following SQL statement, SQL Server will return an incorrect syntax error.

DROP LOGIN DROP
What if you want to drop a login with a name like my][dbreader? This will also throw an incorrect syntax error.

In both examples, because the login name is either a keyword or contains special characters, you will need to provide some beginning and ending markings so that SQL Server can identify the object name inside the SQL statement.

You can use either double quotes or square brackets as the delimiters for SQL identifiers, but you can only use double quotes when the QUOTED_IDENTIFIER setting is on, which is a connection-based setting. To keep things simple, it's a good practice to always use square brackets.

To successfully drop the login name DROP, you can use square brackets to construct your SQL statement:

DROP LOGIN [DROP]
But what about the following statement?
DROP LOGIN [my][dbreader]
In this particular case, since the login name my][dbreader contains the delimiting characters, SQL thinks that [my] is the login name as it's enclosed within square brackets. Because [dbreader] followed the login name, this doesn't form a proper SQL statement and causes a syntax error. You can fix this problem by escaping the right square bracket with another right square bracket. So if you execute the following statement, SQL Server will successfully drop the login my][dbreader:
DROP LOGIN [my]][dbreader]
The escaping mechanism is simply doubling up the occurrences of right square brackets. You don't need to do anything with other characters, including left square brackets.

Preparing delimited literals is similar to preparing delimited SQL identifiers, the main difference being the delimiting character that needs to be used. Let's look at few examples before coming up with similar rules to form a delimited string literal.

Suppose you want to create a login name dbreader with P@$$w0rd as the password. You could use the following SQL statement:

CREATE LOGIN [dbreader] WITH PASSWORD = 'P@$$w0rd'
In this statement, P@$$w0rd is string data delimited by single quotes so SQL knows where the string begins and ends. But what happens if the string data contains single quotes? SQL Server will throw an error as the statement will not be valid:
CREATE LOGIN [dbreader] WITH PASSWORD = 'P@$$'w0rd'
You will need to escape all the occurrences of single quotes within the string to form a valid SQL statement:
CREATE LOGIN [dbreader] WITH PASSWORD = 'P@$$''w0rd'
When you execute this statement, SQL Server will create a login dbreader with P@$$'w0rd as the password.

You can also use double quotes as the delimiter, but as I mentioned previously, the success of this approach is completely dependent on whether the QUOTED_IDENTIFIER setting is on. As a result, it's a much better idea to always use single quotes as the delimiters for string literals.

T-SQL Functions

As you can see, the rules for handling identifiers and strings are relatively simple, and you can delimit them manually if you know the strings beforehand. But what if you are constructing dynamic T-SQL statements based on user input? You will need automatic ways to do this. Two T-SQL functions that can help you prepare delimited strings are QUOTENAME and REPLACE.

QUOTENAME returns a Unicode string with the delimiters added to make the input string a valid identifier. The QUOTENAME function uses this syntax:

QUOTENAME ( 'string' [ , 'delimiter' ] )
You pass QUOTENAME a string to be delimited and a one-character string to use as the delimiter. The delimiter can be a square bracket or a single or double quotation mark.

This function is mainly designed for preparing delimited SQL identifiers, which is why it only accepts the type sysname, which is nvarchar(128) in SQL Server. You can also use this function to prepare delimited SQL literal strings, but because of the argument length restriction it only works for strings of 128 characters or less, (this is where the REPLACE function will be of use). Figure 1 shows how sp_addlogin uses QUOTENAME to prepare delimited login names and password strings. As you can see, because both @loginname and @passwd are sysnames, the QUOTENAME function can be used to prepare a delimited SQL identifier and a delimited literal. So even if someone passes @loginname = 'my[]dbreader' and @passwd = 'P@$$''w0rd', there won't be any SQL injection opportunities as QUOTENAME properly escapes the delimiting characters:

create login [my[]]dbreader] with password = 'P@$$''w0rd'

Figure 1 Delimiting Strings with QUOTENAME (Close Figure 1)
create procedure sys.sp_addlogin
    @loginame       sysname
   ,@passwd         sysname = Null
   ,@defdb          sysname = ‘master’
   ,@deflanguage    sysname = Null
   ,@sid            varbinary(16) = Null
   ,@encryptopt     varchar(20) = Null
AS
    -- SETUP RUNTIME OPTIONS / DECLARE VARIABLES --
    -- some code ----
    set @exec_stmt = ‘create login ‘ + quotename(@loginame, ‘[‘)

    if @passwd is null
        select @passwd = ‘‘

    if (@encryptopt is null)
        set @exec_stmt = @exec_stmt + ‘ with password = ‘ +
            quotename(@passwd, ‘‘‘‘)
    else
    -- some code
GO

The REPLACE function replaces all occurrences of a given string with a specified replacement string. Unlike QUOTENAME, there are no length restrictions on the arguments it accepts:

REPLACE ( 'string1' , 'string2' , 'string3' )

REPLACE takes three strings: string1 is the expression to be edited, string2 is the item within string1 to be replaced, and string3 is the item to be used in place of string2. Any of the string expressions can be of character or binary data.

For preparing delimited SQL literals, you can use REPLACE to double up the occurrences of single quotes, but you will need to add the delimiters manually (beginning and ending single quotes). Figure 2 shows how sp_attach_single_file_db uses this function to prepare an escaped physical name of a file. Since @physname is nvarchar(260), you cannot use QUOTENAME to prepare the delimited literal, which is the reason for using REPLACE. So even if someone passes a string with single quote they cannot break out of the SQL statement and inject any SQL code.

Figure 2 Delimiting Strings with REPLACE (Close Figure 2)
create procedure sys.sp_attach_single_file_db
    @dbname sysname,
    @physname nvarchar(260)
as
    declare @execstring nvarchar (4000)
    -- some code --
    select @execstring = ‘CREATE DATABASE ‘
        + quotename( @dbname , ‘[‘)
        + ‘ ON (FILENAME =‘
        + ‘‘‘‘
        + REPLACE(@physname,N’’’’,N’’’’’’)
        + ‘‘‘‘
        + ‘ ) FOR ATTACH’
    EXEC (@execstring)
    -- some code --
GO

SQL Injection Vulnerabilities

Let's assume that we have an application that implements password management in a user-level table (you should never store passwords in clear text, but I've simplified this for illustrative purposes) and supports changing passwords using a stored procedure sp_setPassword (see Figure 3).

Figure 3 Changing a Password (Close Figure 3)
CREATE PROCEDURE sp_setPassword
    @username varchar(25),
    @old varchar(25),
    @new varchar(25)
AS

DECLARE @command varchar(100)

SET @command=
    ‘update Users set password=‘‘‘ + @new +
    ‘‘‘ where username=‘‘‘ + @username +
    ‘‘‘ AND password=‘‘‘ + @old + ‘‘‘‘

EXEC (@command)
GO

A quick look at the stored procedure shows that none of the parameters are escaped for single quotes and, as such, this is vulnerable to SQL injection attacks. An attacker can pass a few specific arguments and modify the SQL statement to this:

update Users set password='NewP@ssw0rd'
where username='admin' --' and password='dummy'
The result sets the password for the admin account (or any known account) without requiring the actual password. In T-SQL, you can fix this code by using either REPLACE or QUOTENAME. Figure 4 shows the corrected code using the REPLACE function.

Figure 4 Using REPLACE to Avoid Injection (Close Figure 4)
CREATE PROCEDURE sp_setPassword
    @username varchar(25),
    @old varchar(25),
    @new varchar(25)
AS

-- Declare variables.
DECLARE @command varchar(100)

-- Construct the dynamic SQL
SET @command=
    ‘update Users set password=‘‘‘ + REPLACE(@new, ‘‘‘‘, ‘‘‘‘‘‘) + ‘‘‘‘ +
    ‘ where username=‘‘‘ + REPLACE(@username, ‘‘‘‘, ‘‘‘‘‘‘) + ‘‘‘‘ +
    ‘ AND password = ‘‘‘ + REPLACE(@old, ‘‘‘‘, ‘‘‘‘‘‘) + ‘‘‘‘

-- Execute the command.
EXEC (@command)
GO

As you can see, REPLACE will double up all the occurrences of single quotes present in the parameters. So if an attacker passes the same arguments, the statement will instead become:

update Users set password='NewP@ssw0rd'
where username='admin''--' and password='dummy'
This will not be susceptible to typical SQL injection issues.

Modification by Truncation

If you look closely at the stored procedure shown previously, you will notice that the @command variable can only hold 100 characters, but a REPLACE function on each variable of 25 characters can return 50 characters when all the characters are single quotes. SQL Server 2000 SP4 and SQL Server 2005 SP1 silently truncate the data if the variable doesn't have big enough buffers. This provides attackers an opportunity to truncate the command string.

In this example, if someone can truncate the command right after the expression username='username', then it's possible to change the password of a known user account without knowing the victim's current password.

Let's assume that an attacker knows that a user with the name administrator exists in the Web application (this could be any user account). The attacker will need to supply a new password with a length of 41 characters in order to make the command long enough to be properly truncated—41 characters because out of the 100 characters for the command, 27 characters are consumed by the update statement, 17 by the where clause, 13 by "administrator", and 2 single quotes surrounding the new password.

The attacker can only pass 25 characters for the new password. However, he can get around this by passing single quotes, which get doubled by the REPLACE function. So by passing 18 single quotes, 1 capital letter, 1 symbol, 2 lowercase letters, and 1 digit, the attacker will be able to truncate the command right after the where username='administrator' expression. If the attacker passes ''''''''''''''''''!Abb1 for the @new parameter and administrator as the username parameter, then @command will become:

update Users set password=
'''''''''''''''''''''''''''''''''''''!Abb1' where username='administrator'

Figure 5 uses QUOTENAME instead of REPLACE. The only difference between the previous example and this example is that, in the previous example, the developer has added the single quote delimiters for the username, new password, and old password, and in the current example, the QUOTENAME function adds them. Since there is no change to the user-supplied data, this can be exploited by the same attack string that is used for the previous example. Figure 6 is a shortened version of a C/C++ function written in a mid-tier application to achieve the same functionality. It is vulnerable to the same attack.

Figure 6 Truncation Problems in C++ (Close Figure 6)
DWORD ChangePassword(char* psUserName, char* psOld, char* psNew)
{
    char* psEscapedUserName = NULL;
    char* psEscapedOldPW = NULL;
    char* psEscapedNewPW = NULL;
    char szSQLCommand[100];
    HRESULT hr=0;

    // Input Validation
    ...

    // Calculate and allocate the new buffer with length
    // userdatalen*2 + 1
    // Escape all single quotes with double quotes
    ...

    //Construct the query
    hr = StringCchPrintf(szSQLCommand, sizeof(szSQLCommand)/sizeof(char),
        "Update Users set password=‘%s’ where username=‘%s’"
        "AND password=‘%s’,
        psEscapedNewPW, psEscapedUserName, psEscapedOldPW);

    if (S_OK != hr)
    {
        // handle error cases
    }

    // Execute and return
}
Figure 5 Using QUOTENAME to Avoid Injection (Close Figure 5)
CREATE PROCEDURE sp_setPassword
    @username varchar(25),
    @old varchar(25),
    @new varchar(25)
AS
-- Declare variables.
DECLARE @command varchar(100)

-- In the following statement, we will need 43 characters
-- to set an administrator password without knowing its current password.
-- 100 - 26 - 16 - 15 = 43 (26 for update stmt, 16 for where clause,
-- 15 for ‘administrator’). But @new only takes 25 characters, which we
-- can get around by using single quotes. So one can pass the following
-- parametes and set admin password. @new = 18 single quotes, 1 Capital
-- letter, 1 symbol, 2 small case letters, 1 digit
-- @username = administrator
-- @command becomes
-- update Users set password=‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘‘!Abb1’
-- where username=‘administrator’
SET @command= ‘update Users set password=‘ + QUOTENAME(@new,’’’’) +
‘ where username=‘ + QUOTENAME(@username,’’’’) + ‘ AND password = ‘ + QUOTENAME(@old,’’’’)

-- Execute the command.
EXEC (@command)
GO

SQL Injection by Truncation

Figure 7 shows one more variant of the same problem, but uses separate variables to fix first-order SQL injection issues. As you can see, this code stores the escaped strings into separate variables, and @command has enough buffers to hold the whole string. @escaped_username, @escaped_oldpw and @escaped_newpw are declared as varchar(25), but they would need to hold 50 characters in the case where all of the characters in @username, @old, and @new are 25 single-quote characters. This gives the opportunity to truncate the escaped character string.

Figure 7 Using Separate Variables to Avoid Injection (Close Figure 7)
CREATE PROCEDURE sp_setPassword
    @username varchar(25),
    @old varchar(25),
    @new varchar(25)
AS
-- Declare variables.
DECLARE @escaped_username varchar(25)
DECLARE @escaped_oldpw varchar(25)
DECLARE @escaped_newpw varchar(25)
DECLARE @command varchar(250)

SET @escaped_username = REPLACE(@username, ‘‘‘‘, ‘‘‘‘‘‘)
SET @escaped_oldpw = REPLACE(@old, ‘‘‘‘, ‘‘‘‘‘‘)
SET @escaped_newpw = REPLACE(@new, ‘‘‘‘, ‘‘‘‘‘‘)

SET @command =
    ‘update Users set password=‘‘‘ + @escaped_newpw + ‘‘‘‘ +
    ‘ where username=‘‘‘ + @escaped_username + ‘‘‘‘ +
    ‘ AND password = ‘‘‘ + @escaped_oldpw + ‘‘‘‘
EXEC (@command)
GO

An attacker can pass 123...n' as the new password, where n is 24th character, make @escaped_newpw also 123...n' (the second single-quote character returned by the REPLACE function gets truncated), and make the final query look like the following, which he can exploit by injecting code through username field:

update users set password='123...n''
where username='<SQL Injection here using Username>
This code pattern is more dangerous, as this provides the opportunity to inject SQL code rather than just truncate existing SQL.

Figure 8 provides another example of the same variant that uses the QUOTENAME function instead of REPLACE. Since QUOTENAME adds the delimiters, the payload will be different, but it's still vulnerable to SQL injection attacks.

Figure 8 Using QUOTENAME with Separate Variables (Close Figure 8)
ALTER PROCEDURE sp_setPassword
    @username varchar(25),
    @old varchar(25),
    @new varchar(25)
AS
-- Declare variables.
DECLARE @quoted_username varchar(25)
DECLARE @quoted_oldpw varchar(25)
DECLARE @quoted_newpw varchar(25)
DECLARE @command varchar(250)

-- In the following statements, all the variables can only hold
-- 25 characters, but quotename() will return 52 characters when all
-- the characters are single quotes.
SET @quoted_username = QUOTENAME(@username, ‘‘‘‘)
SET @quoted_oldpw = QUOTENAME(@old, ‘‘‘‘)
SET @quoted_newpw = QUOTENAME(@new, ‘‘‘‘)

-- By passing the new password as 123...n where n is 24th character,
-- @quoted_newpw becomes ‘123..n
-- Observe carefully that there is no trailing single quote as it gets
-- truncated.
-- So the final query becomes something like this
-- update users set password=‘123...n where username=‘ <SQL Injection
-- here using Username>
SET @command= ‘update Users set password=‘ + @quoted_newpw +
              ‘ where username=‘ + @quoted_username +
              ‘ AND password = ‘ + @quoted_oldpw
EXEC (@command)
GO

In this case, the code is storing the delimited strings into separate variables and @command has enough buffers to hold the whole command string. Just like the previous example, the problem lies in the quoted variables @quoted_username, @quoted_oldpw, and @quoted_newpw. They are declared as varchar(25), but they will need 52 characters in the case where all the characters in @username, @old, and @new are 25 single-quote characters. (QUOTENAME will also add the beginning and ending delimiter.) This gives the attacker an opportunity to truncate the delimited character string.

An attacker can pass 123...n (where n is the 24th character) as the new password, make @escaped_newpw also '123...n (beginning single quote is added by QUOTENAME), and make the final query to look like this, which he can exploit by injecting code through username field:

update users set
password='123...n where
username=' <SQL Injection here using Username>
Figure 9 is a shortened version of this code in C/C++ to achieve the same functionality. Again, it is vulnerable in the same way.

Figure 9 Variable Truncation Issues in C++ (Close Figure 9)
DWORD ChangePassword(char* psUserName, char* psOld, char* psNew)
{
    char szEscapedUserName[26];
    char szEscapedOldPW[26];
    char szEscapedNewPW[26];
    char szSQLCommand[250];

    // Input Validation

    // Escape User supplied data
    Replace(psUserName, "’", "’’", szEscapedUserName,
            sizeof(szEscapedUserName));
    Replace(psPassword, "’", "’’", szEscapedOldPW,
            sizeof(szEscapedOldPW));
    Replace(psPassword, "’", "’’", szEscapedNewPW,
            sizeof(szEscapedNewPW));

    // Construct the query
    StringCchPrintf(szSQLCommand, sizeof(szSQLCommand)/sizeof(char),
        "Update Users set password=‘%s’ where username=‘%s’"
        "AND password=‘%s’,
        szEscapedNewPW, szEscapedUserName,szEscapedOldPW);

    // Execute and return
}

Even though I used T-SQL code for demonstration purposes, in reality you don't need to use dynamic SQL for Data Manipulation Language (DML) statements, so most of the apps that contain DML code will not be vulnerable to these issues.

Let's look at another example that constructs a dynamic DDL statement based on user input, as in Figure 10. Just like earlier examples, the following statements have truncation issues:

set @escaped_oldpw = quotename(@old, '''')
set @escaped_newpw = quotename(@new, '''')
They can be exploited by passing @new = '123...' where the 127th character (no single quotes) begins @old = '; SQL Injection', which will make the SQL statement look like this:
alter login [loginname]
with password = '123... old_password = '; SQL
Injection

Figure 10 Creating a Dynamic DDL Statement (Close Figure 10)
create procedure sys.sp_password
    @old sysname = NULL,        -- the old (current) password
    @new sysname,               -- the new password
    @loginame sysname = NULL    -- user to change password on
as
    -- SETUP RUNTIME OPTIONS / DECLARE VARIABLES --
    set nocount on
    declare @exec_stmt nvarchar(4000)
    declare @escaped_oldpw sysname
    declare @escaped_newpw sysname

    set @escaped_oldpw = quotename(@old, ‘‘‘‘)
    set @escaped_newpw = quotename(@new, ‘‘‘‘)

    set @exec_stmt = ‘alter login ‘ + quotename(@loginame) +
        ‘ with password = ‘ + @escaped_newpw + ‘ old_password = ‘ +
        @escaped_old

    exec (@exec_stmt)

    if @@error <> 0
        return (1)

    -- RETURN SUCCESS --
    return  (0)    -- sp_password

Even though stored procedures seem to be the more likely place where these issues show up, not all of them result in security vulnerabilities. Let's see which ones need to be reviewed carefully.

In SQL Server all stored procedures get executed under the context of the caller by default. So even if a procedure has SQL injection issues, a malicious local user who has execute permission on the procedure will not be able to elevate his privileges, and the code injected will get executed under his context. But if you have an internal maintenance script that gets executed as the owner or as some other specific user (by using the EXECUTE AS feature), then the caller has the ability to execute code under a different user context and elevate himself to that user's privileges.

All truncation issues are definitely bugs, but they may not necessarily be security vulnerabilities. But it's better to fix them as you don't know who might find and use them in future.

There are other steps you can take to mitigate injection vulnerabilities in your SQL code. First, avoid using dynamic SQL for DML statements in stored procedures. If you can't avoid dynamic SQL, then use sp_executesql. Second, as illustrated by the examples shown in this article, you need to calculate buffer lengths properly. And finally, in C/C++ code, check string operation return values and see if the string has been truncated or not, failing appropriately if it has been. See the sidebar "Techniques for Detecting Vulnerabilities" for a summary of the steps you can take.

Detecting Injection by Truncation

Detecting SQL injection by truncation issues using automated tools requires a good understanding of all the code patterns that will result in truncation opportunities. You can employ different string data to target different particular code patterns. In the following scenarios, assume that n is the length of the input buffer.

For detecting QUOTENAME delimiting issues, let's start by assuming QUOTENAME (or a similar function for C/C++ apps) is used to prepare a delimited identifier or literal and that the delimited string buffer size is less than 2*n + 2. To catch these issues when the delimited string buffer length equals n, pass long strings of nondelimiting characters. The trailing delimiter will get truncated and you will have an injection opportunity using some other input variable.

To catch these issues when the delimited buffer length is odd, pass long strings of single-quote characters (or right square brackets or double quotes). Since QUOTENAME will double up all the occurrences of delimiters and adds a beginning delimiting character, the trailing delimiter gets truncated as the escaped string buffer can only hold an odd number of characters.

To catch these issues when the delimited buffer length is an even number, pass strings like 1', 1'', 1''', 1'''' and so on, incrementing the number of single quotes (or right square brackets) on each iteration. Because QUOTENAME will double up all the occurrences of single quotes, you will have an even number of single quotes in the return string and, along with a beginning delimiter and 1, you will have an even number of characters. As a result, the trailing delimiter gets truncated.

You can also detect these issues when REPLACE (or a similar function in C/C++ apps) is used to prepare the escaped string and when the escaped string buffer size is less than 2*n. To catch these issues when the escaped string buffer length equals n, pass strings like 1', 12', 123' and 123...n', incrementing the length of the input string on each iteration. In this case, if you hit the right length, then the REPLACE function will double up the last single quote character. Because the escaped string variable doesn't have enough buffer space, the last single quote gets truncated and gets stored as passed, providing an opportunity to break out of the SQL statement.

To catch issues with REPLACE when the escaped buffer length is odd, pass incremental lengths of single quote character strings like ', '', ''' and ''''...' (or simply pass long strings of single quote characters). In this case, REPLACE will double up all the occurrences of single quotes. However, because there is an odd-length buffer size, the last single quote gets truncated providing the opportunity to break out of the statement.

To catch these issues when the escaped buffer length is an even number, pass strings like 1', 1'', 1''', 1'''' and so on incrementing the number of single quotes (or right square brackets) on each iteration. The return value without the beginning 1 will contain an even number of characters, so the entire return value will have an odd number of characters. Because the buffer length is even, the trailing single quote gets truncated, providing the opportunity to break out of SQL statement.

Techniques for Detecting Vulnerabilities

Using Code Reviews If you are performing code reviews, here are some techniques for detecting issues in your SQL statements.

Detecting First or Second-Order SQL Injections

  • Identify the APIs used to execute dynamic SQL statements.
  • Review whether any data validation is performed on the data used in the dynamic SQL statement.
  • If data validation is not performed, review whether the data is escaped for delimiting characters (single quotes for string literals and right square brackets for SQL identifiers).

Detecting SQL Modification by Truncation Issues

  • Review the buffer lengths used that are to store the final dynamic SQL statement.
  • Calculate the maximum buffer required to hold the SQL statement when the input is maxed out and see if the buffer used to hold the SQL statement is big enough.
  • Pay special attention to the return values from QUOTENAME or REPLACE functions; if the length of input data is n characters, these functions will return 2*n + 2 or 2*n when all the input characters are delimiting characters.
  • For C/C++ applications, check whether the return values from APIs like StringCchPrintf that are used to prepare the SQL statement are checked for insufficient buffer errors.

Detecting SQL Injection by Truncation Issues

  • Review the buffer lengths that are used to store the delimited strings or escaped strings.
  • If n is the length of the input string, then you will need 2*n + 2 for storing the return value from QUOTENAME and 2*n for storing the return value from REPLACE.
  • For C/C++ applications, check whether the return values from replace equivalent functions are checked for insufficient buffer errors.

Using Black Box Methods If you have automated tools or smart fuzzers, then here are some techniques for detecting issues in your SQL statements.

Detecting SQL Injection Issues

  • Send single quotes as the input data to catch the instances where the user input is not sanitized and used as string literals inside a dynamic SQL statement.
  • Use right square brackets (the ] character) as the input data to catch instances where the user input is used as part of a SQL identifier without any input sanitization.

Detecting Truncation Issues

  • Send long strings, just as you would send strings to detect buffer overruns.

Detecting SQL Modification by Truncation Issues

  • Send long strings of single quote characters (or right square brackets or double quotes). These will max out the return values from REPLACE and QUOTENAME functions and might truncate the command variable used to hold the SQL statement.

Bala Neerumalla works as a security software developer at Microsoft. He specializes in finding application security vulnerabilities.


Editor's Note: Healthy Printing

Editor's Note
Healthy Printing

We were contemplating this month’s Editor’s Note, preliminarily titled "Why, Oh Why, Can’t Bluetooth Ever Just Work Right? Why? WHY?" when we were distracted by an incoming e-mail message from a reader. He asked why the magazines are mailed to him in plastic bags. "A printed version is bad enough for the planet, but to wrap it in a bag too?"

First of all, we commend the reader for having the courage to write to us in the first place. We can be a bit scary—some people send letters to us and are never heard from again. Second, he raises a good point—what is this magazine (and its sister publication, TechNet Magazine) doing to help conserve natural resources? We gave our printers a call.

MSDN Magazine sends the magazine in a bag when there’s extra material to go with it—a CD, poster, or some other sort of insert. As you’ve probably noticed, we give you a lot of content for your buck, and sometimes it’s printed so that it’s big enough to fit on your wall and be seen from a distance. If there’s nothing but the magazine one month, then it’s not mailed in a plastic bag. And you don’t have to throw the bag away—it’s made from recyclable materials.

The paper we use is partially recycled. Our printer tells us that 100% bleached, recycled paper is actually worse for the environment (the bleaching process is quite harsh), and it’s not sturdy enough for our printing press (so we’d waste a lot more paper). The portion of the paper that’s not recycled comes from new growth that’s managed by the paper mills, so they’re not clearing old forest for our paper.

The ink we use is soy-based. Way back in the old days, ink used to be petroleum-based. Newspapers eventually realized that this was not the best option, either for the pocketbook or the environment. Soy-based ink was developed as an alternative. Unlike petroleum inks, soy-based inks are made from soybean oil, a renewable resource. Not only does soy ink have no reported effects on the environment, it can actually be beneficial—half the soybeans grown in the United States don’t require irrigation, so they may put less of a burden on the environment than most other vegetable-based oils. Soy-based ink can often provide brighter, crisper colors than the old petroleum-based ink as well. (At least this is what Wikipedia told us, and they’re never wrong!)

Many readers have noticed that our paper is heavier than most other magazines—that’s by design. High-quality inks and paper provide a publication that’s easier to read and that stands up to repeated flipping of pages. Having good paper is important for us because we want to make sure the magazine isn’t just some throwaway. If you get a daily newspaper, in a week you have a big stack of newsprint that you have to stick in a blue bin and leave by the curb for pickup. You can only hope that it’s being routed to a proper recycling plant.

People who receive MSDN Magazine, on the other hand, tend to keep it for a long time, years on average. We design our topic coverage to be relevant now, and even more so a year or two down the road. Since people hold onto their issues for a long time, we build each issue to last, both physically and in terms of content. When’s the last time you picked up a three-year-old issue of People Magazine because you remembered a celebrity gossip piece that has just become relevant to your work?

Of course, not every aspect of MSDN Magazine is recycled. We strive for 100% new-growth content, harvested from the finest authors and experts in the field. We flash-freeze the content and rush it to press so that you’ll be proud to present it on your dinner table or, more likely, your office desk.

We will note with some disdain, however, that no one seems to be able to tell us why Bluetooth just never works quite right for us. Maybe we’ll save that topic for another Editor’s Note.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Paul Andrew, Tom Archer, Kevin Boske, Jonathan Caves, Tom Christian, Derek Del Conte, Joe Duffy, Shawn Farkas, Dan Hartop, Jim Johnson, Brian Jones, Don McCrady, Cyrus Najmabadi, Joe Self, Ayman Shoukry, Dharma Shukla, and Joel West.

Excel, IntelliSense, Internet Explorer, Microsoft, Microsoft Press, MSDN, PowerPoint, Visual Basic, Visual C++, Visual Studio, Windows, Windows NT, WinFX, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN MAGAZINE DOES NOT MAKE ANY REPRESENTATION OR WARRANTY, EXPRESS OR IMPLIED WITH RESPECT TO ANY CODE OR OTHER INFORMATION HEREIN. MSDN MAGAZINE DISCLAIMS ANY LIABILITY WHATSOEVER FOR ANY USE OF SUCH CODE OR OTHER INFORMATION. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Technology LLC. CMP Technology LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Synchronize Files, Rich Textboxes, and More

Toolbox
Synchronize Files, Rich Textboxes, and More
Scott Mitchell

Contents
Compare Files and Folders
Create Demo and Support Videos
Rich Textboxes in Your Web App
The Bookshelf

Compare Files and Folders

When working within a team, developers often need to compare the code on their file system against the code checked into the source control provider, or perhaps to examine the differences between two different versions of a given file or set of files in source control. Or maybe you have hundreds of data files in a folder and need to determine if and how they've changed since the last backup.

Programs designed to find differences in files have existed since the dawn of computers, with the UNIX diff program being one of the most widely known and used (Windows® has long included a similar program called WinDiff). Unfortunately, WinDiff and many of the packaged diff-like utilities lack more advanced capabilities, such as comparing ZIP files, or entire folders of files, or files residing on a remote FTP server, and then synchronizing differences.

Beyond Compare (Click the image for a smaller view)

Beyond Compare (Click the image for a larger view)

One tool that offers rich comparison and synchronization features is Beyond Compare version 2.4.1 by Scooter Software. When starting the tool, you are greeted with the Compare Wizard, which lists the common uses of Beyond Compare—comparing two folders or two files, comparing and merging source code changes, synchronizing with an FTP site, and so on—and quick steps to get started with each. Alternatively, you can bypass the wizard and specify the folders, files, or FTP sites to compare. These choices, among others, can be saved as a project to quickly repeat the comparison or synchronization at a later point in time.

With the targets for comparison selected, Beyond Compare lists the results in a two-pane interface, using colors and highlighting to indicate which items are the same, which ones differ, and which ones exist in one source but not the other. Clicking on a file opens a new window that displays the differences (if any) between the files in the two sources. After viewing the differences, you can opt to synchronize the sources.

Beyond Compare's power comes with its comparison rules, which permit you to indicate how the comparison engine determines whether two files differ. For example, if two files are identical except for white space, should they be flagged as different? What if they're identical except for the casing of the text? Beyond Compare provides a number of built-in rules designed for use when comparing code from different programming languages—there are predefined rules to aid in comparing C++/C#, XML, Visual Basic®, HTML, comma-delimited files, and so on. And if none of the built-in rules suffice, you can always create your own comparison conditionals.

Price: $30 for a single-user license.

www.scootersoftware.com

Create Demo and Support Videos

They say a picture is worth a thousand words, so what better way to pitch your product to prospective buyers or provide support to existing users than through the power of video? A software package designed specifically for turning on-screen demos into videos is Camtasia Studio version 3.1.2 by TechSmith.

Camtasia Studio allows you to collect, edit, and arrange various multimedia clips that can include on-screen recordings of your product in action; audio content, either from a microphone or from an existing audio file; and live or prerecorded video content.

Creating the on-screen recordings is a snap. Simply indicate whether to capture the entire screen, a particular window or screen region and whether or not to record audio input. Once you're ready to start recording, press F9 and interact with your program as you normally would—Camtasia will record the mouse movements and screen changes.

Camtasia (Click the image for a smaller view)

Camtasia (Click the image for a larger view)

Once you've completed the on-screen recording, press F10 to add the clip to Camtasia Studio, where it can be edited, if needed. When you have your completed video and audio recording, the final step is to export the clip to a playable video format. When exporting, you can specify various compression, size, and quality-related properties that impact the file size. (A one-minute 800x600 video with audio, saved to a Flash file with the default settings, requires close to 1.5MB.)

Novice video and audio creators will find that they can easily create impressive videos in minutes with Camtasia Studio and its helpful wizards. For example, Camtasia Studio adds a Microsoft® PowerPoint® toolbar icon that can record a PowerPoint presentation with just a click of the mouse.

For more experienced content creators, Camtasia Studio offers an array of editing and enhancement options. Add hotspots to more clearly show where and when the mouse has been clicked, zoom into regions on the screen, add callouts and annotations to the video, and draw on the screen with pens. In short, Camtasia Studio's price and features make it appealing to a wide range of users.

Price: $299 for a single-user license.

www.techsmith.com

Rich Textboxes in Your Web App

Many of today's Web apps are being designed to include rich data collection UIs so that they appear and behave more like desktop applications. With many Web sites having their content created by visitors, the standard textbox is quickly becoming obsolete.

When including the user's input as part of the Web app's content, it behooves the designer to provide a rich textbox, one that allows the user to indicate the text's font, formatting, and so on. This is remarkably easy with FCKeditor, a free, open-source rich textbox created by Frederico Caldeira Knabben.

FCKeditor provides a rich data entry interface with customizable toolbars and styles, support for spell checking, tables, predefined content templates, the ability to embed images and Macromedia Flash files—a user experience not unlike any word processor.

At its heart, FCKeditor is a collection of HTML and JavaScript pages that render the rich textbox in Microsoft Internet Explorer®, Firefox, and Netscape browsers on Windows, Mac, and Linux operating systems (there is plain-text only support for the Opera and Safari browsers). FCKeditor includes samples and integration packages for incorporating the FCKeditor HTML pages and scripts into server-side technologies.

On the browser, users can enter their content using either the rich editor or through the Source view, where they may include any HTML markup for formatting. Upon submitting the form, the HTML content corresponding to the user's content is available via the FCKeditor's Value property.

For many Web application scenarios, the standard HTML textbox fails to offer the features or pizzazz that a rich text editor can supply. With FCKeditor, incorporating rich text data collection is a fast and easy process, regardless of the server-side technology being used.

Price: Free, includes source code.

www.FCKeditor.net

The Bookshelf

The Microsoft Enterprise Library, first released in January 2005, is composed of seven application blocks—Configuration, Data Access, Caching, Exception Handling, Logging and Instrumentation, Security, and Cryptography. Each of these application blocks was designed from the ground-up to offer a consistent interface and configuration experience for the developer and to provide functionality that can easily be extended or customized.

While the application blocks are designed so that an architect can opt to use just the ones needed, the move from individually packaged, independent application blocks to a larger, unified library has made understanding, installing, and utilizing the Enterprise Library a more daunting task. Effective Use of Microsoft Enterprise Library (Addison Wesley, 2006), by Len Fenster, helps demystify the Enterprise Library application blocks, explaining their intended functionality, illustrating how they can be used, and showing how to extend their functionality.

(Click the image for a smaller view)

(Click the image for a larger view)

The book is composed of nine content-packed chapters and three appendices, with each application block thoroughly explored in its own chapter. Each chapter begins with an examination of the application block's purpose, followed by a look at its class structure. The author then walks through how to configure an application to use the application block; numerous source code samples illustrate how to work with the application block from code. The book is a must read for those who are using the Enterprise Library or are in the midst of evaluating it.

Price: $49.99.

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


Basic Instincts: Server-Side Generation of Word 2007 Docs

Basic Instincts
Server-Side Generation of Word 2007 Docs
Ted Pattison

Code download available at: BasicInstincts2006_11.exe (170KB)
Contents
Word 2007 Document Internals
Generating Your First DOCX File
Generating DOCX Files on the Server

Until now, writing and deploying server-side applications that read, modify, and generate documents used by Microsoft® Office applications has presented a challenge. The older binary format used by Microsoft Word, Excel®, and PowerPoint® was introduced in 1997 and has remained the default file format up through the Office 2003 release. However, this binary file format has proven to be too tricky to work with. The vast majority of production applications that read and write Office documents do so by going through the object model of the hosting Office application.

Applications and components that use the object model of applications such as Word or Excel run much better on the desktop than they do in server-side scenarios. Anyone who has spent time writing the extra infrastructure code required to make an Office desktop application behave reliably on the server will tell you it's a less than desirable solution. That's because Office desktop applications such as Word and Excel were never designed to run on the server, and they require a custom utility program to terminate and restart them whenever they encounter a modal dialog that requires human intervention.

The ability to read and write Office documents without going through the object model of the hosting Office application becomes far more desirable in a server-side scenario. Office 2000 and Office 2003 introduced some modest capabilities for creating Excel workbooks and Word documents using XML. This advancement introduced the possibility of writing portions of an Office document using an XML library such as the one contained in the Microsoft .NET Framework provided through the System.Xml namespace.

In the 2007 Microsoft Office system, Microsoft has taken this idea much further by adopting the Office Open XML file formats for the documents used by Microsoft Office Word, Excel, and PowerPoint 2007. The Office Open XML file formats are exciting to ASP.NET and SharePoint® developers because they provide the ability to read, write, and generate a Word document, an Excel workbook, or a PowerPoint presentation on the server without requiring the system to run an Office desktop application on the server.

The Office Open XML file formats are on their way to becoming a published European Computer Manufacturers Association (ECMA) standard. You can read more about the ongoing standardization process, download the latest draft of the Office Open XML file formats specification, and find other great online resources at openxmldeveloper.com. My goal in this month's column is to introduce the programming required to create simple Word documents within an ASP.NET application and pass them back to users who will be able to open them directly with Word 2007.

Word 2007 Document Internals

Let's begin by examining the structure of a simple Word document based on the Office Open XML file formats. As you will see, the Office Open XML file formats are based on standard ZIP file technology. Each top-level file is saved as a ZIP archive, which means you will be able to open a Word document just as you would any other ZIP file and snoop around inside using the ZIP file support built into Windows® Explorer.

Note that the 2007 Microsoft Office system applications such as Word and Excel have introduced file extensions for documents that use the new format. For example, the .docx extension is used for Word documents stored in the Office Open XML file formats while the more familiar .doc extension continues to be used for Word docs stored in the binary format.

To see what I'm talking about, create a new document in Word 2007 and add "Hello Word" as the text. Save the document using the default format to a new file named Hello.docx and close Word. Next, locate Hello.docx using Windows Explorer and rename it Hello.zip. Open Hello.zip and see the structure of folders and files that Word has created inside (see Figure 1).

Figure 1 DOCX File is a ZIP Archive
Figure 1 DOCX File is a ZIP Archive(Click the image for a smaller view)
Figure 1 DOCX File is a ZIP Archive
Figure 1 DOCX File is a ZIP Archive (Click the image for a larger view)

The top-level file (Hello.docx) is known as a package. Since a package is implemented as a standard ZIP archive, it automatically provides compression and it makes its contents instantly accessible to many existing utilities and APIs on both Windows platforms and other platforms alike.

Inside a package there are two kinds of internal components: parts and items. In general, parts contain content and items contain metadata describing the parts. Items can be further subdivided into relationship items and content-type items. A part is an internal component containing content that is persisted inside the package. The majority of parts are simple text files that are serialized as XML with an associated XML schema. However, parts can also be serialized as binary data when necessary, for example when a Word document contains a graphic image or a media file.

A part is named using a Uniform Resource Identifier (URI) that contains its relative path within the package file combined with the part file name. For example, the main part within the package for a Word document is /word/document.xml. Here are a few more typical part names you will find inside the package for a simple Word document:

  • /[Content_Types].xml
  • /_rels/.rels
  • /docProps/app.xml
  • /docProps/core.xml
  • /word/_rels/document.xml.rels
  • /word/document.xml
  • /word/fontTable.xml
  • /word/settings.xml
  • /word/styles.xml
  • /word/theme/theme1.xml

The Office Open XML file formats use relationships to define associations between a source and a target part. A package relationship defines an association between the top-level package and a part. A part relationship defines an association between a parent part and a child part. Relationships are important because they make these associations discoverable without having to examine the content within the parts in questions. That means relationships are independent of content-specific schema and are, therefore, faster to resolve. There is an additional benefit in that you can establish a relationship between two parts without having to modify either one of them.

Relationships are defined in internal components known as relationship items. A relationship item is stored inside the package just like a part, although a relationship item is not actually considered a part. For consistency, relationship items are always created inside folders named _rels.

For example, a package contains exactly one package-relationship item named /_rels/.rels. The package-relationship item contains XML elements to define package relationships such as the one between the top-level package for a .docx file and the internal part /word/document.xml, as shown here:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="../package/2006/relationships ">
  <Relationship Id="rId1"
      Type="../officeDocument/2006/relationships/officeDocument "
      Target="word/document.xml"/>
</Relationships>
As you can see, a Relationship element defines a name, a type, and a target part. You should also observe that the type name for a relationship is defined using the same conventions used to create XML namespaces.

In addition to a single package-relationship item, a package can also contain one or more part-relationship items. For example, you define relationships between /word/document.xml and child parts inside a package-relationship item located at the URI /word/_rels/document.xml.rels. Note that the Target attribute for a relationship in a part-relationship item is a URI relative to the parent part and not the top-level package.

Every part inside a package is defined in terms of a specific content type. A content type is metadata which defines a part's media type, a subtype, and a set of optional parameters. Any content type used within a package must be explicitly defined inside a component known as a content type item. Each package has exactly one content type item named /[Content_Types].xml. An example of content type definitions defined inside /[Content_Types].xml is shown in Figure 2. You should observe from this figure that relationship items are like parts in that they are also defined with an associated content type.

Figure 2 Content Types Define Package (Close Figure 2)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Types xmlns=
  "http://schemas.openxmlformats.org/package/2006/content-types">

  <Default Extension="rels" ContentType=
    "application/vnd.openxmlformats-package.relationships+xml"/>

  <Default Extension="xml" ContentType="application/xml"/>

  <Override PartName="/word/document.xml"
    ContentType="application/vnd.openxmlformats-
                 officedocument.wordprocessingml.document.main+xml"/>
</Types>

Content types are used by the consumer of a package to interpret how to read and render the content within its parts. As you can see from Figure 2, a default content type is typically associated with a file extension, such as .rels or .xml. Override content types are used to define a specific part in terms of a content type that is different from the default content type associated with its file extension. For example, Figure 2 shows how /word/document.xml is associated with a content type that is different from the default content type used for files with the .xml extension.

Generating Your First DOCX File

While there are several existing libraries you can use to read and write to ZIP files, you should use the new packaging API that is part of the WindowsBase.dll assembly that ships with the .NET Framework 3.0 whenever possible because the packaging API is aware of the Office Open XML file formats. For example, there are convenient methods that make it easy to add relationship elements to a relationship item and to add content type elements to a content type item. The packaging API makes things easier because you never have to touch these elements and items directly.

This month's code uses the version of WindowsBase.dll that is installed along with the WinFX® Runtime Components February community technology preview (CTP). Note that WinFX Runtime Components were renamed to the .NET Framework 3.0 with the June CTP release. You can use either of these two versions or any later release of the .NET Framework 3.0. To test the code we are going to write in this column, you will also need to install Word 2007 Beta 2 or later. (Changes to Office since Beta 2 may require some modifications to the code presented here.)

Once you have installed the .NET Framework 3.0, which includes WindowsBase.dll, you can start programming against the packaging API in a Visual Studio® 2005 project by adding a reference, as shown in Figure 3.

Figure 3 Add Reference to WindowsBase.dll
Figure 3 Add Reference to WindowsBase.dll(Click the image for a smaller view)
Figure 3 Add Reference to WindowsBase.dll
Figure 3 Add Reference to WindowsBase.dll (Click the image for a larger view)

The classes that make up the packaging API are contained inside the namespace System.IO.Package. When working with packages, you will also be frequently programming against older and (hopefully) familiar classes in the System.IO namespace and the System.Xml namespace. Examine the code in Figure 4 that shows the skeleton for creating a new package.

Figure 4 Creating a New Package (Close Figure 4)
Imports System
Imports System.IO
Imports System.IO.Packaging
Imports System.Xml

Class GenerateDOCX

  Shared Sub Main()

    ‘*** Create a new package
    Dim pack As Package = Package.Open("c:\Data\Hello.docx", _
                                       FileMode.Create, _
                                       FileAccess.ReadWrite)

    ‘*** Write code here to create parts and add content

    ‘*** Close package
    pack.Close()

  End Sub

End Class

The System.IO.Packaging namespace contains the Package class, which exposes a shared method named Open to be used to create new packages and open existing packages. As with other classes that deal with file I/O, a call to Open should always be complemented with a call to Close.

Once you have created the new package, the next step is to create one or more parts and to serialize content into them. In my example in this month's column, we will follow the guidelines for a "Hello World" application, which will require the creation of a single part named \word\document.xml. We can create a part by calling the CreatePart method on an open Package object and passing parameters for a URI and a string-based content type:

'*** Create main document part (document.xml) ...
Dim uri As Uri = New Uri("/word/document.xml", UriKind.Relative)
Dim partContentType As String = "application/vnd.openxmlformats" & _
    "-officedocument.wordprocessingml.document.main+xml"
Dim part As PackagePart = pack.CreatePart(uri, partContentType)

'*** Get stream for document.xml
Dim streamPart As New StreamWriter(part.GetStream(FileMode.Create, _
                                                  FileAccess.Write))

The call to CreatePart passes a URI based on the path /word/document.xml and the content type that's required by the Office Open XML file formats for the part in a word-processing document that contains the main story. Once we have created a part, we must serialize our content into it using standard stream-based programming techniques. The code you have just seen opens a stream on the part by calling the GetStream method and uses the resulting Stream object to initialize a StreamWriter object.

The StreamWriter object is going to be used to serialize the "Hello World" XML document into document.xml. Take a look at the following XML; it represents the simplest of XML documents that can be serialized in document.xml:

<?xml version="1.0" encoding="utf-8"?>
<w:document xmlns:w=
    "http://schemas.openxmlformats.org/wordprocessingml/2006/3/main">
  <w:body>
    <w:p>
      <w:r>
        <w:t>Hello Word 2007</w:t>
      </w:r>
    </w:p>
  </w:body>
</w:document>

Note that all the elements in this XML document are defined in the schemas.openxmlformats.org/wordprocessingml/2006/3/main namespace as required by the Office Open XML file formats. The XML document contains a high-level document element. Within the document element there is a body element that contains the main story.

Within the body element there is a <p> element for each paragraph. Within the <p> element is an <r> which defines a run. A run is a region of elements that share the same set of characteristics. Within the run is a <t> element which defines a range of text, in this case "Hello Word 2007".

Now it's time to generate this XML document with code using the XmlDocument class from the System.Xml namespace. If you take a look at Figure 5 you'll see how the code creates these elements within the proper structure and using the appropriate namespace. You should also note how this code writes the XML document into the stream using the StreamWriter object and then closes the stream and flushes the serialized content into the Package object.

Figure 5 Generating the XML Document (Close Figure 5)
‘*** Define string variable for Open XML namespace for nsWP:
Dim nsWP As String = &quot;http://schemas.openxmlformats.org&quot; &amp; _
                      &quot;/wordprocessingml/2006/3/main&quot;

‘***  Create the start part, set up the nested structure ...
Dim xmlPart As XmlDocument = New XmlDocument()
Dim tagDocument As XmlElement
tagDocument = xmlPart.CreateElement(&quot;w:document&quot;, nsWP)
xmlPart.AppendChild(tagDocument)
Dim tagBody As XmlElement
tagBody = xmlPart.CreateElement(&quot;w:body&quot;, nsWP)
tagDocument.AppendChild(tagBody)
Dim tagParagraph As XmlElement
tagParagraph = xmlPart.CreateElement(&quot;w:p&quot;, nsWP)
tagBody.AppendChild(tagParagraph)
Dim tagRun As XmlElement
tagRun = xmlPart.CreateElement(&quot;w:r&quot;, nsWP)
tagParagraph.AppendChild(tagRun)
Dim tagText As XmlElement
tagText = xmlPart.CreateElement(&quot;w:t&quot;, nsWP)
tagRun.AppendChild(tagText)

‘*** Insert text into part as a Text node
Dim nodeText As XmlNode
nodeText = xmlPart.CreateNode(XmlNodeType.Text, &quot;w:t&quot;, nsWP)
nodeText.Value = &quot;Hello Word 2007&quot;
tagText.AppendChild(nodeText)

‘*** Write XML to part and close stream
xmlPart.Save(streamPart)

‘*** Close stream and flush XML content into package
streamPart.Close()
pack.Flush()

Now we are finished writing the XML content into document.xml. The final step is to create a relationship between the package and document.xml by calling the CreateRelationship method of the Package object. This is easy as long as you know the correct string value for the relationship type and you can come up with a unique name (such as rId1) for the relationship being created:

'*** Create the relationship part
Dim relationshipType As String = _
    "http://schemas.openxmlformats.org" & _
    "/officeDocument/2006/relationships/officeDocument"
pack.CreateRelationship(uri, TargetMode.Internal, _
    relationshipType, "rId1")
pack.Flush()

'*** Close package
pack.Close()

You should also observe the call to Flush after the call to CreateRelationship. This call forces the packaging API to update the package relationship item with the proper Relationship element. The final call to the Package object's Close method completes the package serialization and releases the file handle on Hello.docx.

Now you have seen all the steps to generate a simple .docx file from a console application written in Visual Basic® 2005, all of which is available here. Now let's write the code to generate a .docx file on the server side from within an ASP.NET app.

Generating DOCX Files on the Server

The first thing to consider when modifying the code from the console application to run on a Web server is that you probably want to avoid having to save the package out as a physical file within the file system of the host computer. Instead, it will be faster to simply create the package in a memory stream within the ASP.NET worker process. Then you can write it back to the client using the OutputStream object of the ASP.NET Response object.

Let's start by changing the code to use a MemoryStream object instead of a physical file. Examine the code in Figure 6, which has been added to a server-side event handler inside an ASP.NET 2.0 page. Note that the first parameter to Package.Open has changed from the string-based file path to a MemoryStream object. This approach will allow you to reuse the same code for creating the package and its parts as we did earlier. However, you don't have to worry about creating and naming an OS-level file. This approach may also provide faster response times and better throughput in high-traffic scenarios.

Figure 6 Using a MemoryStream Object (Close Figure 6)
‘*** Create in-memory stream as buffer
Dim bufferStream As New MemoryStream()

‘*** Create new package in memory stream
Dim pack As Package = Package.Open( _
  bufferStream, FileMode.Create, FileAccess.ReadWrite)

‘*** This calls same code shown in Hello World example
WriteContentToPackage(pack)

‘*** Save/close package object leaving DOCX file in MemoryStream
pack.Close()

‘*** (1) SET UP HTTP HEADERS FOR RESPONSE
‘*** (2) WRITE PACKAGE CONTENT INTO RESPONSE BODY

The code you have just seen creates a MemoryStream object and then serializes a .docx file into it just as you would serialize a .docx file into a physical file. The code inside the custom WriteContentToPackage method was taken directly from the code inside the console application shown earlier. However, now it's creating a package and serializing it into a buffer in memory instead of into a physical file.

Once you have written the package into the MemoryStream object and called Close on the package object, your work with the packaging API is complete. All that's left to do is to set up the appropriate HTTP headers and write the package content into the body of the response that is being sent back to the client.

Let's start with the HTTP headers. You should call methods on the ASP.NET response object to clear any existing headers and to add a content-disposition header specifying that the response contains an attachment with a file name of Hello.docx:

Response.ClearHeaders()
Response.AddHeader("content-disposition", _
                   "attachment; filename=Hello.docx")

Next, you must set up the encoding and MIME content type for the HTTP response and then write the binary content for the package into the body of the HTTP response. This can be accomplished using the code in Figure 7. Note that the last two calls to Response.Flush and Response.Close are required to make sure the entire package is completely written into the HTTP response.

Figure 7 Response.ClearContent (Close Figure 7)
Response.ClearContent()
Response.ContentEncoding = System.Text.Encoding.UTF8
Response.ContentType = &quot;application/vnd.ms-word.document.12&quot;

‘*** Write package to response stream
bufferStream.Position = 0
Dim writer As New BinaryWriter(Response.OutputStream)
Dim reader As New Bina NO RESPONSE YET | ryReader(bufferStream)
writer.Write(reader.ReadBytes(bufferStream.Length))
reader.Close()
writer.Close()
bufferStream.Close()

‘*** flush and close the response object
Response.Flush()
Response.Close()

As long as the client's machine has been configured to understand the MIME content type associated with a .docx file, the user will be presented with the usual options: open the document within Word or save it to the local hard drive.

If the user clicks on the Open button, the .docx file is opened automatically in Word as shown in Figure 8. It is interesting to note that up to this point the package has never been saved as a physical file to the file system. It's been stored only in memory both on the Web server and on the client desktop computer. If the user closes the document without saving it, it is as if it never existed at all. This makes this approach ideal for generating templates for letters, memos, customer lists, and any type of document you can think of.

Figure 8 Document Opens in Word 2007
Figure 8 Document Opens in Word 2007(Click the image for a smaller view)
Figure 8 Document Opens in Word 2007
Figure 8 Document Opens in Word 2007 (Click the image for a larger view)

Send your questions and comments for Ted to instinct@microsoft.com.

Ted Pattison is an author, trainer and SharePoint MVP who lives in Tampa, Florida. Ted is writing a book titled Inside Windows SharePoint Services 3.0 for Microsoft Press and he delivers advanced SharePoint 2007 Training to professional developers through his company, Ted Pattison Group (http://www.TedPattison.net).


Test Run: Using Excel For Test Data

Test Run
Using Excel For Test Data
Dr. James McCaffrey

Code download available at: TestRun2006_11.exe (171KB)
Contents
The Library under Test
The Test Case Data File
Creating the Test Harness
Storing Results
Wrapping Up

If you are writing test automation, there are four complementary approaches you can take: buy and use a commercial test framework (including Visual Studio® 2005 Team System, which has some neat new features), use an open source test framework, write custom heavyweight automation (generally more than four pages of code), or write lightweight automation (generally under four pages of code). Most of my Test Run columns focus on techniques for lightweight automation. When writing lightweight test automation, one of your best choices for test case data and test results storage is to use Microsoft® Excel®.

In this month's column I will show you how to use Excel for test automation storage. Even though this is aimed at developers with beginning .NET coding skills, intermediate and advanced programmers should still find the information I present here interesting and rather useful.

Suppose you are writing a cribbage game like the one shown in Figure 1. In cribbage, your hand consists of five cards—four in your hand and one shared by you and your opponent. Your point score is determined by a number of factors, such as the number of pairs in your hand and the number of card combinations you hold that sum to 15. Each pair is worth 2 points and each set of cards that sum to 15 is worth 2 points. Face cards count as 10, aces count as 1, and all other cards count as their point value. For the hand shown in Figure 1, the value of the pairs is 2 because there is a single pair of 7s. The value of the 15s is 6 because there are three combinations of cards that sum to 15: the 7 of diamonds and 8 of clubs; the 7 of clubs and 8 of clubs; the 7 of diamonds, 7 of clubs, and ace of hearts.

Figure 1 App Under Test
Figure 1 App Under Test

Behind the scenes, the game UI calls into a CribbageLib class library that you've written. This library houses classes to represent a Card object and a Hand object, as well as methods and properties, such as Hand.ValueOfPairs and Hand.ValueOf15s.

In this column, I will present techniques for using Excel to hold test case data and store test results, as illustrated by the program shown in Figure 2. The lightweight test harness begins by checking to see if the Excel spreadsheet that holds test case data exists. (I'll explain shortly how to set up an Excel spreadsheet so that you can programmatically read from it.) Next, the harness probes into the Excel data to determine the number of test cases and then reads all the Excel data into an in-memory DataTable object. (I'll discuss the alternative approach of reading one test case at a time in the last section of this column.) The harness creates a new Excel spreadsheet to hold test case results, and then executes each test case. The harness determines a pass/fail result for each test, prints the result to the console, and saves the result to the Excel spreadsheet. The complete source code for the test harness and the library under test are in the code download that accompanies this column.

In the sections that follow, I will briefly describe the class library under test so you will understand how to set up appropriate Excel spreadsheets for storage. I will present the code for the simple test harness that produced the results shown in Figure 2. I will explain the code in detail so you will be equipped to modify and extend the code to use Excel to meet your own needs. And I will conclude with a very brief discussion of how and when to use Excel for test storage, looking at the pros and cons of this approach when compared to other types of test storage (SQL Server™ databases, text files, and XML files in particular). I believe you'll find the ability to use Excel for lightweight test storage a useful addition to your software testing, development, and management tool kit.

Figure 2 Using Excel for Test Data Storage
Figure 2 Using Excel for Test Data Storage(Click the image for a smaller view)
Figure 2 Using Excel for Test Data Storage
Figure 2 Using Excel for Test Data Storage (Click the image for a larger view)

The Library under Test

In order to test a software component with automation, you must know at a bare minimum the component's interfaces. The code in Figure 3 shows the structure of my CribbageLib class library. Let me emphasize that I am deliberately not adhering to standard .NET coding guidelines so I can keep the size of my library under test small.

Figure 3 CribbageLib Library Structure (Close Figure 3)
using System;
namespace CribbageLib
{
  public class Card
  {
    public int rank;
 // 0 = Ace, 1 = Deuce, . . , 12 = King
    public int suit;
 // 0 = Clubs, 1 = Diamonds, 2 = Hearts, 3 = Spades
    public int value;
 // Ace = 1, Deuce = 2, . . , Jack/Queen/King = 10
    public string image;
 // "Ac", "9h", etc.

    public Card(string c)
    {
      // create Card from string c
    }

    public override string ToString()
    {
      return image;
    }
  } // class Card

  public class Hand
  {
    Card[] cards;
    public Hand(Card c1, Card c2, Card c3, Card c4, Card c5)
    {
      cards = new Card[5];
      cards[0] = c1;
      cards[1] = c2;
      cards[2] = c3;
      cards[3] = c4;
      cards[4] = c5;
    }

    public override string ToString()
    {
      return "{ " + cards[0].ToString() + " " +
             cards[1].ToString() + " " + cards[2].ToString() + " " +
             cards[3].ToString() + " " + cards[4].ToString() + " }";
    }

    public int ValueOf15s { get { /* return point value */ } }

    public int ValueOfPairs { get { /* return point value */ } }
  } // class Hand
} // ns CribbageLib

The CribbageLib library contains a Card class to represent a single card. I use int fields rank, suit, and value to represent the rank (such as ace, deuce, three), the suit (club, diamond, heart, spade), and value in cribbage (aces are worth 1 point, deuces 2, face cards are 10 points). The Card constructor accepts a string (such as "Ac" for the ace of clubs), parses the two input characters, and stores values for the three corresponding member fields. For convenience when writing a ToString() method, I also store a string field named image, which is just the input argument. The Hand class is essentially an array of five Card objects, representing the four cards a player holds plus the one common card shared by both players. The Hand constructor accepts five Card objects.

The heart of my CribbageLib class library under test consists of the Hand.ValueOf15s and Hand.ValueOfPairs properties. These properties return the point value for the number of 15s in a cribbage hand and the point value for the number of pairs in a hand. Although the purpose of my CribbageLib library is merely to provide a sample to test, the ValueOf15s and the ValueOfPairs properties are quite interesting little problems in their own right. I've actually seen these used as hiring interview questions. However, as far as this column is concerned, if you want to use Excel for lightweight test storage, you don't need to know the implementation details of the properties. You can treat these properties as black boxes that have expected results for a particular input.

The Test Case Data File

You typically create your Excel data files manually when using Excel test case data storage for lightweight test automation, (I will demonstrate how to create Excel files programmatically in just a moment.) Manually creating an Excel spreadsheet for test case data is like creating a spreadsheet for any other purpose, but with a couple of added steps. Because my test harness uses OLE DB technology to access Excel programmatically, I need to supply column header labels to identify each column so I can write queries. I also need to create a named area in the spreadsheet to correspond to a SQL table name.

For this example, I want to store a test case ID number (like "00001"), a test case input (like "Ah7d7c8cJs") to represent a cribbage hand, a value to indicate which class property I am sending my input to (like "ValueOf15s"), and an expected result (like "6"). So, I launch Excel, manually type in appropriate column names ("caseID", "input", "method", and "expected"), and manually enter my test case data. To assign a virtual table name to my spreadsheet, I select the column header cells (cells A1 through D1 in this case) and then type a table name into the Excel Name Box field in the upper left-hand corner of the spreadsheet. In this case, I named my table tblTestCases, as you can see in Figure 4.

Figure 4 Test Case Data in Excel
Figure 4 Test Case Data in Excel(Click the image for a smaller view)
Figure 4 Test Case Data in Excel
Figure 4 Test Case Data in Excel (Click the image for a larger view)

Although it is not strictly necessary to do so, I also rename the worksheet to the same name as the virtual table name. When programmatically accessing Excel data, the name in the Name Box field (rather than the worksheet name) is used to determine which data to reference. But as you'll see in a moment, when you programmatically create an Excel spreadsheet, the worksheet name receives the same name as that assigned to the Name Box. So renaming the worksheet of Excel test data that has been manually created is more consistent than leaving the worksheet named Sheet1.

If you look closely at the Excel spreadsheet in Figure 4, you will notice that I entered all the test case data as text. For instance, the expected result for the first test case is 6 formatted as text rather than 6 formatted as a numeric type. This is indicated by the small green triangle in cell D2. I recommend you store all test case data as text and then programmatically convert non-string data to the appropriate data type in the test harness. The Excel data model is not the same as the .NET data model. It is generally easier to perform all data type conversions in the harness rather than trying to get data types as close as possible in the spreadsheet.

Creating the Test Harness

For clarity, I've combined the code used to perform several different tasks with Excel test case data into a single test harness program. Figure 5 shows the structure of the test harness that generated the output shown in Figure 2.

Figure 5 Test Harness Structure (Close Figure 5)
using System;
using System.IO;
using System.Data;
using System.Data.OleDb;
using CribbageLib;
namespace CribbageLibTest
{
  class Program
  {
    static void Main(string[] args)
    {
      try
      {
        Console.WriteLine("\nBegin lightweight test harness with " +
            "Excel storage demo\n");
        Console.WriteLine("Verifying Excel test case data exists");
        if (!File.Exists("..\\..\\..\\TestCases.xls"))
          throw new Exception("Test case data not found");
        else Console.WriteLine("Found test case data");
        Console.WriteLine("Determining number of test cases");
        ... // probe Excel test case data code here
        Console.WriteLine("Number of test cases = " + count);
        Console.WriteLine("Reading all test case data from Excel " +
            "into memory");
        ... // read all test case data code here
        Console.WriteLine("All test case data now in memory");
        Console.WriteLine("\nCreating Excel test case results file");
        ... // create Excel code here
        Console.WriteLine("Results file is " + resultsFile);
        Console.WriteLine("\nExecuting CribbageLib tests\n");
        ... // connect to Excel test results data here

        string caseid, input, method, result;
        int actual, expected;
        Console.WriteLine("caseID  input method  expected  case result");
        Console.WriteLine("===========================================");
        for (int row = 0;
row <
dt.Rows.Count;
++row) // main test loop
        {
          ... // read test case, execute, print result,
              // save result to Excel
        }

        Console.WriteLine("\nAll test results stored to Excel\n");
        Console.WriteLine("End Excel as test storage demo\n");
      }
      catch (Exception ex)
      {
        Console.WriteLine("Fatal error: " + ex.Message);
      }
      Console.ReadLine();
    } // Main()
  } // class Program
} // ns CribbageLibTest

I begin my test harness by adding a project reference to the CribbageLib.dll component that houses my library under test. Then I add a using statement for the namespaces in the library so that I can reference the Card and Hand classes without having to fully qualify their names. In addition, I add a using statement to the System.Data.OleDb namespace, which contains classes that can be used to connect to, access, and manipulate OLE DB data sources, including Excel spreadsheets. I also add a using statement to the System.Data namespace so I can easily instantiate and use a DataTable object to act as an in-memory data store for my test case data from Excel. Because this column is essentially a tutorial, for simplicity I organize my test harness into a single Main method—you, however, may want to consider making your harnesses more modular.

I begin by printing a start message to console, then use the static File.Exists method to verify that the Excel test case data is located where I expect it to be. (To simplify this example, I have removed most of the error-checking code you would need in a production environment.) Once I know my test case data exists, I probe the Excel spreadsheet to find out how many rows of data there are:

int count;
string probeConnStr = "Provider=Microsoft.Jet.OLEDB.4.0;" +
                      "Data Source=testCases.xls;" +
                      "Extended Properties=\"Excel 8.0;HDR=YES\"";
using(OleDbConnection probeConn = new OleDbConnection(probeConnStr))
{
    probeConn.Open();
    string probe = "SELECT COUNT(*) FROM [tblTestCases$A1:A65536] " +
                    "Where caseID IS NOT NULL";
    using(OleDbCommand probeCmd = new OleDbCommand(probe, probeConn))
    {
        count = (int)probeCmd.ExecuteScalar();
    }
}

Here, I create a connection string that specifies the appropriate OLE DB provider, location, and auxiliary information. You have to be careful with the syntax. Notice that in the Extended Attributes property I use a \" sequence to embed a double quote character into the connection string. The HDR=YES attribute indicates that my Excel worksheet has an initial header row. The "Excel 8.0" part does not directly refer to the version of the Excel program on my computer—it refers to the Jet database ISAM (Indexed Sequential Access Method) format that is installed. You can check the ISAM version on your machine by viewing the system registry setting HKEY_LOCAL_MACHINE\Software\Microsoft\Jet\4.0\ISAM Formats. Next I create an OleDbConnection object, passing the connection string to the constructor. I call the Open method to establish a connection to my Excel spreadsheet. I then craft a select string that will return the number of non-NULL rows in my spreadsheet, which will be the number of test cases since the header row does not contribute to the return value. This, of course, assumes that I do not have any empty rows in my Excel test case data. Notice the somewhat unusual syntax in the SELECT statement, which refers to my Excel data. I surround my virtual table with square bracket characters and I append a $ character to the virtual table name. I specify a range of A1:A65536 because 65536 is the maximum number of rows supported in an Excel worksheet. Finally, I use the ExecuteScalar method of the OleDbCommand class so I can capture the return value of the number of test cases into a variable named count. And I complete the probing code by closing the OleDbConnection.

Now I'm ready to reconnect to my Excel spreadsheet and read all my test case data from Excel into memory like this:

string tcdConnStr = "Provider=Microsoft.Jet.OLEDB.4.0;" +
                    "Data Source=testCases.xls;" +
                    "Extended Properties=\"Excel 8.0;HDR=YES\"";
using(OleDbConnection tcdConn = new OleDbConnection(tcdConnStr))
{
    tcdConn.Open();
    int lastRow = count + 1;
    string select = "SELECT * FROM [tblTestCases$A1:D" + lastRow + "]";
    OleDbDataAdapter oda = new OleDbDataAdapter(select, tcdConn);
    DataTable dt = new DataTable();
    oda.FillSchema(dt, SchemaType.Source);
    oda.Fill(dt);
}
In this snippet, I create a new connection string. I could use the same string variable as I used to probe the number of test cases; I use a different string variable only for clarity and to make it easier for you to modularize my code if you wish. Next I create a new OleDbConnection object. Again, I could have reused the connection object I used for the test case count probe. After opening the connection, I compute the last row of test case data by adding 1 to the number of actual test cases in the Excel spreadsheet—the extra 1 accounts for the header row. Then I build a SELECT string by concatenating the first part of the select with the variable holding the number of the last row that contains test case data.

In order to fill my in-memory DataTable object with test case data, I create an OleDbDataAdapter. Then I instantiate a new DataTable object to hold all my test case data. I use the OleDbDataAdapter's FillSchema method to configure my DataTable attributes to conform to my Excel data source. I then call the Fill method, which actually transfers my test case data from Excel into memory. And I finish by closing the OleDbConnection. Very clean and simple.

Storing Results

Now that I've gotten all my test case data into memory, I am ready to create a new Excel spreadsheet to store my test case results. The code is similar to my other Excel tasks except this time I use an OleDbCommand object (see Figure 6).

Figure 6 Creating a Spreadsheet to Store Results (Close Figure 6)
string stamp = DateTime.Now.ToString("s");
stamp = stamp.Replace(":", "-");
string resultsFile = "testResults" + stamp + ".xls";
string tcrConnStr = "Provider=Microsoft.Jet.OLEDB.4.0;" +
                    "Data Source=" + resultsFile + ";" +
                    "Extended Properties=\"Excel 8.0;HDR=YES\"";
using(OleDbConnection tcrConn = new OleDbConnection(tcrConnStr))
{
    tcrConn.Open();
    string create = "CREATE TABLE tblResults (caseID char(5),
                     Result char(4), WhenRun DateTime)";
    using(OleDbCommand createCmd = new OleDbCommand(create, tcrConn))
    {
        createCmd.ExecuteNonQuery();
    }
}

I begin by fetching the system date and time so that I can create a time-stamped results file name. I pass an "s" argument to the ToString() method so I will get a date-time string in a format that is sortable (such as 2006-07-23T16:56:44). But because the : character is not valid in a file name, I use the String.Replace method to replace all : characters with hyphens.

My connection string is similar to the two others I've already used, except that I embed the time-stamped file name into it. I then open the connection to a new Excel file. The file does not actually exist yet, so you can think of this as a kind of virtual connection.

Next I craft a SQL-like CREATE string specifying a virtual table name of tblResults. Notice that I can specify the data type for each column. You have to be a bit careful here because many SQL data types do not map exactly to Excel data types. An alternative approach is to simply design your results file so that all the columns are varchar type data. Then, if you need to perform some numeric analysis of your test results in Excel (say, computing an average or a maximum value), you can manually format the columns you are analyzing into the appropriate Excel type (such as Number or Percentage).

In this example, I store the test case ID, the test result as "pass" or "fail", and a DateTime variable that holds the time at which the test case was executed. I finish by instantiating an OleDbCommand object and using the ExecuteNonQuery method. As I mentioned earlier, this process creates an Excel spreadsheet that contains a worksheet named tblResults, which has a Named Area tblResults.

Now that my test case data is in memory and my Excel results file has been created, I am ready to execute my tests and store my results. I begin by creating a connection to my newly created Excel results file, as shown here:

Console.WriteLine("\nExecuting CribbageLib tests\n");
string insertConnStr = "Provider=Microsoft.Jet.OLEDB.4.0;" +
                       "Data Source=" + resultsFile + ";" +
                       "Extended Properties=\"Excel 8.0;HDR=YES\"";
using(OleDbConnection insertConn = new OleDbConnection(insertConnStr))
{
    insertConn.Open();
    OleDbCommand insertCmd = new OleDbCommand();
    insertCmd.Connection = insertConn;

The next thing I do is set up an INSERT command, which I will use to insert a test result after determining whether a test case has passed or failed.

Next I set up my test harness variables and then print a simple output header. I declare string variables to hold my test case ID, my test case input, the method (or Property in this example) under test, and an indicator of the result ("pass" or "fail"). I declare int variables expected and actual to hold the expected return value from the method under test (which is stored in my in-memory DataTable) and the actual result returned when the method under test is called by the test harness. This is done like so:

string caseid, input, method, result;
int actual, expected;
Console.WriteLine("caseID  input method  expected  case result");
Console.WriteLine("===========================================");

Now the main test loop iterates through each row of the DataTable object, as shown in the following:

for (int row = 0;
row <
dt.Rows.Count;
++row)

Inside the main test loop, I first perform a rudimentary check to make sure I have a valid test case. Then I fetch each column of the current row of the DataTable and store into the variables with more meaningful names that I declare just outside the loop:

object o = dt.Rows[row]["caseid"];
if (o == null) break;
caseid = (string)dt.Rows[row]["caseid"];
input = (string)dt.Rows[row]["input"];
method = (string)dt.Rows[row]["method"];
expected = int.Parse( (string)(dt.Rows[row]["expected"]) );
Notice that this is the point at which I do my type conversions. Remember that I originally stored all my test case data as type Text in my Excel spreadsheet and read all the data as type String into my DataTable object. Now I can call the method under test and retrieve the actual value:
CribbageLib.Hand h = new Hand(new Card(input.Substring(0, 2)),
                              new Card(input.Substring(2, 2)),
                              new Card(input.Substring(4, 2)),
                              new Card(input.Substring(6, 2)),
                              new Card(input.Substring(8, 2)));
if (method == "ValueOf15s") actual = h.ValueOf15s;
else if (method == "ValueOfPairs") actual = h.ValueOfPairs;
else throw new Exception("Unknown method in test case data");

I use the String.Substring method to parse out each pair of characters, such as 7c, which represent a Card object. The first integer argument to Substring is a zero-based index value of where in the string to begin extracting the substring. The second integer argument is the number of characters to extract (as opposed to the ending index as you might have thought). I branch on the value of variable method and call the method under test, capturing the actual return value. Finally, to finish my main test loop, I determine the date and time of my test case execution, determine the test result, build up an INSERT string, print my result to console, and insert the result into the Excel results worksheet:

DateTime whenRun = DateTime.Now;
result = (actual == expected) ? "Pass" : "FAIL";

Console.WriteLine(caseid + "   " + h.ToString() + " " +
                  method.PadRight(15, ' ') +
                  expected.ToString().PadRight(8, ' ') + result);
string insert = "INSERT INTO tblResults (caseID, Result, WhenRun)
                 values ('" + caseid + "', '" + result +
                 "', '" + whenRun + "')";
insertCmd.CommandText = insert;
insertCmd.ExecuteNonQuery();

I use the PadRight method—this is an old trick—to line up my output in columns. The insert statement is a bit ugly because of the embedded single quotes, commas, and paren characters—you just have to be careful here. Because I am invoking an INSERT command, I use the ExecuteNonQuery method to actually perform the insert operation.

Wrapping Up

As I mentioned, lightweight test automation is a complement to, not a replacement for, other test automation approaches. If you do find yourself in a scenario where lightweight automation is appropriate, Excel makes an excellent test data storage choice. The two major alternatives for such storage are plain text files and XML documents. Text files are particularly simple but compared to Excel storage, they have the significant disadvantage that spurious control characters and whitespace characters are difficult to detect. Meanwhile, XML documents can be overkill for lightweight test automation storage. XML's hierarchical structure makes reading and writing the data somewhat more difficult than reading and writing data stored in Excel.

Additionally, Excel has some neat advantages that are useful in this sort of scenario. First, if you use Excel to store test case data, you can employ a strategy of "open test case data entry." By this, I mean you can place the Excel file in a public area and allow everyone on your team to add test cases. A few years ago I was writing a poker library. I found an old clunker machine, set up an Excel test case data spreadsheet on it, placed a deck of cards next to the machine, and encouraged everyone to enter test cases. It was very successful and the process uncovered some critical bugs. Since most people are comfortable with Excel, you'll probably get more participation in this sort of open project than if you use some other data storage. The second notable thing about Excel is that if you save your results to a worksheet, you've got a built-in way to do some lightweight test result analysis using the many math functions.

The new Microsoft Office 2007 system will enhance your ability to use Excel for test data storage. The new Office XML Formats for Excel, Word, and PowerPoint® stand to bring several programmatic interoperability improvements to software development and testing. Because the new Excel .xlsx file format is based on XML, it will provide improved file management capabilities and an improved ability to recover damaged files. And because the Office XML Formats use ZIP compression technology, test data files will be significantly smaller.

For more information on the new Microsoft Office 2007 file formats, see Ted Pattison's Basic Instincts column in this issue of MSDN®Magazine and also check out microsoft.com/office/preview

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products, including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Data Points: Revisiting System.Transactions

Data Points
Revisiting System.Transactions
John Papa

Code download available at: DataPoints2006_11.exe (172KB)
Contents
In a Nutshell
Current Transactions and Enlistment
Transactional Settings
Disposal
Lightweight Transactions
Nesting
Wrapping It Up

The System.Transactions namespace of the Microsoft .NET Framework makes handling transactions much simpler than previous techniques. Some time ago I wrote a Data Points column about how System.Transactions worked with Beta 1 of the Microsoft® .NET Framework 2.0 and SQL Server™ 2005. Of course, during the release process some features were added and some were dropped; several of the TransactionScopeOptions changed.

Since then, I've received many questions about System.Transactions and have decided it's time to revisit it. So let's look at how it works now, how to use it, when it makes sense, and when it does not. Along the way you'll see how to take advantage of .NET constructs to make your use of the namespace more efficient. I'll also demonstrate best practices with transactions. All of the samples can be found in the download at the MSDN®Magazine Web site.

In a Nutshell

Let's begin with how to turn two database commands into a transaction by building a wrapper around them. It's easy. Just reference the System.Transactions.dll, wrap the code you want to be transactional within a using statement that creates a TransactionScope, and invoke the Complete method at the end of the transaction.

Figure 1 shows a transaction being created and wrapping itself around multiple database queries. If either of the SqlCommand objects throws an exception, control of flow will jump out of the TransactionScope's using statement block and the TransactionScope will dispose of itself and roll back the transaction. Because the code employs the using statements, the SqlConnection and the TransactionScope objects will be disposed of automatically. So, by adding only a few lines of code, you now have a transactional model that handles exceptions, cleans up after itself, and manages the committing or rolling back of the commands.

Figure 1 A Simple Transaction (Close Figure 1)
// Create the TransactionScope
using (TransactionScope ts = new TransactionScope())
{
    using (SqlConnection cn2005 = new SqlConnection(someSql2005))
    {
        SqlCommand cmd = new SqlCommand(sqlUpdate, cn2005);
        cn2005.Open();
        cmd.ExecuteNonQuery();
    }

    using (SqlConnection cn2005 = new SqlConnection(anotherSql2005))
    {
        SqlCommand cmd = new SqlCommand(sqlDelete, cn2005);
        cn2005.Open();
        cmd.ExecuteNonQuery();
    }

    // Tell the transaction scope that the transaction is in
    // a consistent state and can be committed
    ts.Complete();

    // When the end of the scope is reached, the transaction is
    // completed, committed, and disposed.
}

The example that you see in Figure 1 has a lot of moving parts. TransactionScope encompasses all resource manager connections that automatically enlist in transactions. You can also set different options on the TransactionScope, although the example in Figure 1 uses defaults.

Current Transactions and Enlistment

At the heart of System.Transactions is the TransactionScope class. When it is instantiated, it creates a current transaction (also known as an ambient transaction) that any resource manager can enlist itself in. For example, if an instance of TransactionScope has been created and a connection is opened to a resource manager that defaults to automatic enlistment in transactions, the connection will join the transaction scope.

You can examine whether or not a transaction scope exists at any point in your code by looking at the System.Transactions.Transaction.Current property. If this property is null, then no current transaction exists. When a resource manager opens its connection to its resource, it looks to see if a transaction exists. If the resource manager has been set to automatically enlist itself in a current transaction, then it will join the transaction. One of the attributes of the SQL Server connection string is auto-enlist. By default, auto-enlist is set to true, so any active transaction will be joined. You can change the default by explicitly adding auto-enlist=false to the connection string, like so:

Server=(local)\SQL2005;Database=Northwind;
Integrated Security=SSPI;auto-enlist=false
This is the magic of System.Transactions. I did not change any of the ADO.NET code in Figure 1, yet it still takes advantage of the TransactionScope. All I had to do was create a TransactionScope and the SqlConnection object enlisted in the active transaction when the connection was opened.

Transactional Settings

If you want to deviate from the default settings of the TransactionScope class, you can set the isolation level and the transaction's timeout period on the TransactionScope object by creating a TransactionOptions object. The TransactionOptions class has an IsolationLevel property, which can change the default isolation level of serializable to another level such as ReadCommitted or even the new Snapshot level introduced with SQL Server 2005. (Keep in mind that the isolation level is merely a suggestion. Most database engines will try to use the suggested isolation level, but may opt for another.) The TransactionOptions class also has a TimeOut property that can be used to deviate from the default of one minute.

The sample in Figure 1 used the default TransactionScope object with its default constructor. This sets the isolation level to serializable, the transaction's timeout period to one minute, and its TransactionScopeOptions setting to Required. If you want to alter these settings, there are seven additional overloaded constructors for the TransactionScope that you can use. In Figure 2, I have listed the TransactionScopeOptions enumerators settings. These enumerators allow you to control how nested transactions will respond to each another. The code in Figure 3 actually changes the settings.

Figure 3 Changing Transactional Settings (Close Figure 3)
// Create the TransactionOptions object
TransactionOptions tOpt = new TransactionOptions();

// Set the Isolation Level
tOpt.IsolationLevel = IsolationLevel.ReadCommitted;

// Set the timeout to be 2 minutes
// Uses the (hours, minutes, seconds) constructor
// Default is 60 seconds
tOpt.Timeout = new TimeSpan(0, 2, 0);

string cnString = ConfigurationManager.ConnectionStrings[
    "sql2005DBServer"].ConnectionString);

// Create the TransactionScope with the RequiresNew transaction
// setting and the TransactionOptions object I just created
using (TransactionScope ts =
    new TransactionScope(TransactionScopeOption.RequiresNew, tOpt))
{
    using (SqlConnection cn2005 = new SqlConnection(cnString)
    {
        SqlCommand cmd = new SqlCommand(updateSql1, cn2005);
        cn2005.Open();
        cmd.ExecuteNonQuery();
    }

    ts.Complete();
}
Figure 2 TransactionScopeOptions Enumerators (Close Figure 2)

TransactionScopeOptions | Description
Required | If a transaction exists already, this transaction scope will join it. Otherwise, it will create its own transaction.
RequiresNew | This transaction scope will create its own transaction.
Suppress | If within a currently active transaction scope, this transaction scope will not join the ambient transaction nor will it create its own. Use this option when some code needs to be left out of the transaction.

Disposal

The key to using System.Transactions is understanding how and when the transaction ends. If a TransactionScope object is not properly disposed of, the transaction will remain open until either the object has been collected by the garbage collector or the timeout period has elapsed. One danger in leaving the transaction open is that it will lock the resource manager's resources while it is active. To better understand this, take a look at the following code sample:

TransactionScope ts = new TransactionScope();
SqlConnection cn2005 = new SqlConnection(cnString);
SqlCommand cmd = new SqlCommand(updateSql1, cn2005);
cn2005.Open();
cmd.ExecuteNonQuery();
cn2005.Close();
ts.Complete();
This code will create an instance of the TransactionScope object, and when the SqlConnection is opened it will join the transaction. If all goes well, the command will execute, the connection will close, the transaction will complete, and it will be disposed of. If an exception is thrown, the control of flow will skip over the closing of the SqlConnection and the disposal of the TransactionScope, causing the transaction to remain open longer than desired. The key is to make sure the TransactionScope is disposed of properly so the transaction either commits or rolls back quickly. There are two easy ways to handle this: a try/catch/finally block or the using statement. You could declare the objects outside of the try/catch/finally block, add the code to a try block to create the objects and execute the command, and put the disposal of the TransactionScope and the SqlConnection in the finally block. This will ensure that the transaction is closed out in a timely manner.

I often prefer the using statement as it implicitly creates a try/catch block under the covers for you. The using statement guarantees that the TransactionScope will be disposed of even if an exception is thrown in the middle of the code block. It makes sure that the Dispose method is called on the TransactionScope whenever the block is exited. This is important because just prior to disposal of the TransactionScope, the transaction is completed. When the transaction is completed, the TransactionScope evaluates whether or not the Complete method has been called. If it has, then the transaction is committed; otherwise, it is rolled back. The earlier code could be rewritten as follows:

using (TransactionScope ts = new TransactionScope())
{
    using (SqlConnection cn2005 = new SqlConnection(cnString)
    {
        SqlCommand cmd = new SqlCommand(updateSql1, cn2005);
        cn2005.Open();
        cmd.ExecuteNonQuery();
    }
    ts.Complete();
}
Notice that I used the using statement with both the TransactionScope and the SqlConnection object. This ensures both objects would be disposed of properly and quickly if an exception is thrown. If no exception is thrown, the objects are disposed of when the end of the using statement's code bock is reached (its last curly brace).

Lightweight Transactions

One of the greatest features of System.Transactions is its support of lightweight transactions. Lightweight transactions do not involve the Microsoft distributed transaction coordinator (DTC) unless they need to. If the transaction is local, it will remain lightweight. If the transaction becomes distributed and involves a second resource manager, then the lightweight transaction will be promoted to a fully distributed transaction and will indeed involve the DTC.

Involving the DTC, while necessary in distributed scenarios, is expensive. So if you don't strictly need it, it's best to avoid it. Fortunately, SQL Server 2005 supports lightweight transactions; previous versions of SQL Server do not (therefore, all transactions with SQL Server 2000 will always be upgraded to distributed transactions). Let's take a look at a few examples of the benefits of lightweight transactions.

In the example in Figure 4, I create a TransactionScope that executes two commands against a single SQL Server 2000 database. When the connection to SQL Server 2000 is opened, it joins the TransactionScope; because SQL Server 2000 does not support lightweight transactions, the DTC is involved—even though this is obviously not a distributed scenario (since I'm only hitting one database and it's on the same connection).

Figure 4 No Lightweight Support (Close Figure 4)
using (TransactionScope ts = new TransactionScope())
{
    using (SqlConnection cn2000 = new SqlConnection(cnString2000))
    {
        cn2000.Open();

        SqlCommand cmd1 = new SqlCommand(updateSql1, cn2000);
        cmd1.ExecuteNonQuery();

        SqlCommand cmd2 = new SqlCommand(updateSql2, cn2000);
        cmd2.ExecuteNonQuery();
    }
    ts.Complete();
}

In the next example (shown in Figure 5) I create a TransactionScope that executes two commands against a single SQL Server 2005 database. Since SQL Server 2005 supports lightweight transactions, this will become a distributed transaction only if a second resource manager is involved in it.

Figure 5 Lightweight Support (Close Figure 5)
using (TransactionScope ts = new TransactionScope())
{
    using (SqlConnection cn2005 = new SqlConnection(cnString2005))
    {
        cn2005.Open();

        SqlCommand cmd1 = new SqlCommand(updateSql1, cn2005);
        cmd1.ExecuteNonQuery();

        SqlCommand cmd2 = new SqlCommand(updateSql2, cn2005);
        cmd2.ExecuteNonQuery();
    }
    ts.Complete();
}

So when should you use System.Transactions? If you are going to be using a distributed transaction, System.Transactions makes a lot of sense. Also, if your resource manager supports lightweight transactions, System.Transactions is the way to go. But using System.Transactions to wrap all database commands is not necessarily a best practice.

For example, let's assume your application executes multiple commands against a database that does not support lightweight transactions. Business rules dictate that these operations need to be wrapped in a transaction to keep them atomic. If the commands within the transaction are hitting a single database, then ADO.NET transactions would be more efficient than System.Transactions since an ADO.NET transaction will not invoke the DTC in this case. If you have places in your application that truly need to be distributed, then System.Transactions is a good option.

So what resource managers does System.Transactions work with? Actually, it works with all of them, but only those resource managers that support lightweight transactions will take full advantage of promotable transactions.

Nesting

Earlier I mentioned the TransactionScopeOptions enumerator and how you can set it to be Required (the default value), RequiresNew, or Suppress. This comes into play when you have nested methods and transactions. For example, let's assume Method1 creates a TransactionScope, executes a command against a database, and then calls Method2. Method2 creates a TransactionScope of its own and executes another command against a database. There are several ways you might want to handle this. You might want Method2's transaction to join Method1's transaction or you may want Method2 to create a separate transaction of its own. This is where the TransactionScopeOptions enumerator becomes valuable. Figure 6 illustrates nested transactions.

Figure 6 Nesting Transactions (Close Figure 6)
private void Method1()
{
    using (TransactionScope ts =
        new TransactionScope(TransactionScopeOption.Required))
    {
        using (SqlConnection cn2005 = new SqlConnection())
        {
            SqlCommand cmd = new SqlCommand(updateSql1, cn2005);
            cn2005.Open();
            cmd.ExecuteNonQuery();
        }
        Method2();
        ts.Complete();
    }
}

private void Method2()
{
    using (TransactionScope ts =
        new TransactionScope(TransactionScopeOption.RequiresNew))
    {
        using (SqlConnection cn2005 = new SqlConnection())
        {
            SqlCommand cmd = new SqlCommand(updateSql2, cn2005);
            cn2005.Open();
            cmd.ExecuteNonQuery();
        }
        ts.Complete();
    }
}

Here the inner transaction (Method2) will create a second TransactionScope instead of joining the outer transaction (from Method1). Method2's TransactionScope was created with the RequiresNew setting, which tells the transaction to create its own scope and not join an existing one. If you wanted this transaction to join the existing one, you could have left the default setting or set the option to Required.

When transactions join a TransactionScope (because they use Required), they all must vote to Complete the transaction successfully for it to commit. If any of the transactions within the same TransactionScope do not vote to Complete by not calling ts.Complete, then when the outer TransactionScope is disposed of, it will roll back.

Wrapping It Up

It is important to get in and out of your transactions quickly since transactions lock valuable resources. Best practices dictate that you create a transaction just before you need it, open your connection immediately before you need to execute commands on them, execute your action queries, and complete and dispose of the transaction as soon as possible. You also should avoid performing any nonessential, non-database code during the transaction to avoid keeping the resources locked any longer than necessary.

One of the greatest features of the lightweight transaction is that it can determine if it needs to promote itself to a distributed transaction. As I have demonstrated, System.Transactions can be beneficial when used appropriately. The key is knowing how and when.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa, C# MVP and MCSD.NET, has been working with Microsoft distributed architectures for over 10 years. He has enterprise experience architecting and developing with ASP.NET as well as Web Forms using both C# and Visual Basic .NET. John has authored or coauthored several books and speaks at industry conferences. Reach him at codebetter.com/blogs/john.papa.


CLR Inside Out: Investigating Memory Issues

CLR Inside Out
Investigating Memory Issues
Claudio Caldato and Maoni Stephens

Contents
Tools of the Trade
GC Performance Counters
Windows Performance Counters
Verifying an OOM Exception in a Managed Process
Determining What Caused an OOM Exception
Measure Managed Heap Size
What If Objects Survive?
Is Fragmentation a Problem on Your Managed Heap?
Measuring Time Spent on Garbage Collection
Investigating High CPU Usage

Uncovering and correcting memory issues in managed applications can be difficult. Memory issues manifest themselves in different ways. For example, you may observe your application's memory usage growing unboundedly, eventually resulting in an Out Of Memory (OOM) exception. (Your application may even throw out-of-memory exceptions when there is plenty of physical memory available.) But any one of the following may indicate a possible memory issue:

  • An OutOfMemoryException is thrown.
  • The process is using too much memory for no obvious reason that you can determine.
  • It appears that garbage collection is not cleaning up objects fast enough.
  • The managed heap is overly fragmented.
  • The application is excessively using the CPU.

This column discusses the investigation process and shows you how to collect the data you need to determine what types of memory issues you are dealing with in your applications. This column does not cover how to actually fix problems you find, but it does give some good insights as to where to start.

We'll begin with an overview of the most useful performance counters that can be used to investigate managed memory issues. Then we'll cover the tools that are commonly used for the investigation and will continue with a list of common managed memory issues and how to investigate them.

But before we get started, you should familiarize yourself with some fundamental concepts:

  • Garbage collection in the Microsoft® .NET Framework. For more information, see these two blog entries: blogs.msdn.com/156626.aspx and blogs.msdn.com/234273.aspx.
  • How virtual memory works in Windows®. This includes the concepts of reserving memory and committing memory.
  • Using the Windows Debuggers (WinDbg and CDB).

Tools of the Trade

Before we start, we should spend a moment discussing some tools you will typically use to diagnose memory-related issues.

Performance Counters You will usually want to start with the performance counters. They let you collect essential data that you can use to determine where exactly a problem might be. The most useful performance counters are those that report on the .NET Framework, though there are some others also worth looking at.

Debugger Here we'll use WinDbg, which is available as part of the Debugging Tools for Windows. The Son of Strike extension (SOS), available in SOS.dll, is used for debugging managed code in WinDbg. After you start the debugger and attach it to a managed process (or load a crash dump), you can load the SOS.dll by typing:

.loadby sos mscorwks
If you are debugging an app that uses a different version of mscorwks.dll and as a result this command fails, find the SOS.dll for the mscorwks.dll version the application is using and run this command:
.load <path_to_sos>\sos.dll
SOS.dll is installed with the .NET Framework under %windir%\microsoft.net\framework\<.NET version>. The SOS.dll extension provides a lot of useful commands for examining managed heaps. For documentation on all of its commands, see SOS Debugging Extension (SOS.dll) .

Windows Task Manager Taskmgr.exe is handy for spotting higher-than-expected memory usage and for checking the trend of some simple process metrics over a period of time. Note that there are two metrics in taskmgr that are frequently misinterpreted: Mem Usage and VM Size (Virtual Memory Size). Mem Usage represents the process working set (just like the Process\Working Set performance counter). It does not indicate the committed bytes. VM Size reflects committed bytes by the process (just like the Process\Private Bytes performance counter). VM Size can provide a first hint of whether you are dealing with memory leaks (if your application has a leak, VM Size will grow over time).

Memory Dumps Most of the investigation techniques explained in this column rely on memory dumps. There are two ways to use the debugger—you can attach it to a running process or use it to analyze a crash dump. While the first option offers a direct view into what is going on in your application while it is running, this technique is not always feasible.

Memory dumps have the advantage of decoupling the data collection phase from the actual problem investigation phase. Say you want to diagnose issues on a production server; it may be easier to analyze the dump offline using a different machine.

The .dump /ma dmpfile.dmp command in the debugger can be used to create a full memory dump. Make sure you always capture a full dump when investigating memory issues since minidumps do not contain all the information you need.

The ADPlus tool, which is included in the Debugging Tools for Windows, can be of great help when collecting crash dumps. See John Robbins's Bugslayer column from March 2005 for more information.

Throughout this column, we'll assume that the dump file is already loaded into the debugger (crash dumps can be loaded using the File | Open crash dump command), or that the debugger is already attached to the process and the execution is stopped on a breakpoint.

GC Performance Counters

The first step in every investigation is to collect the relevant data and formulate a hypothesis for where the problem may be. Performance counters are usually the place to start. The .NET Framework Performance Console provides access to counters that provide useful information about the garbage collector (GC) and the garbage collection process. Note that the .NET Memory performance counters are updated only when a collection occurs; they are not updated according to the sampling rate used in the Performance Monitor application.

You should start by checking the % Time in GC. This indicates the percentage of time spent in the GC since the end of the last collection. If you see a really high value (say 50 percent or more), then you should look at what is going on inside the managed heap. If, instead, % Time in GC does not go above 10 percent, it usually is not worth spending time trying to reduce the time the GC spends in collections because the benefits will be marginal.

If you think your application is spending too much time performing garbage collections, the next performance counter to check is Allocated Bytes/sec. This shows the allocation rate. Unfortunately, this counter is not highly accurate when the allocation rate is very low. The counter may indicate 0 bytes/sec if the sampling frequency is higher than the collection frequency since the counter is only updated at the beginning of each collection. This does not mean there were no allocations; rather, the counter was not updated because no collection happened in that particular interval. Knowing how long garbage collection takes is an important consideration, so we'll take a closer look at % Time in GC a bit later.

If you suspect that you are allocating a lot of large objects (85,000 bytes or bigger), check the Large Object Heap (LOH) size. This is updated at the same time as Allocated Bytes/sec.

A high allocation rate can cause a lot of collection work and as a result might be reflected in a high % Time in GC. A mitigating factor is if objects usually die young, since they will typically be collected during a Gen 0 collection. To determine how object lifecycles impact collection, check the performance counters for Gen collections: # Gen 0 Collections, # Gen 1 Collections, and # Gen 2 Collections. These show the number of collections that have occurred for the respective generation since the process started. Gen 0 and Gen 1 collections are usually cheap, so they don't have a big impact on the application's performance. Gen 2 collectors however, can be very expensive.

As a rule of thumb, a healthy ratio between generation collections is one Gen 2 collection for every ten Gen 1 collections. If you are seeing a lot of time spent in garbage collection, it could be that Gen 2 collections are being done too often. You should check this ratio to ensure there aren't too many Gen 2 collections relative to the number of Gen 1 collections.

You may find that % Time in GC is high but the allocation rate is not high. This can happen if many of the objects you allocate survive garbage collection and are promoted to the next generation. The promotion counters—Promoted Memory from Gen 0 and Promoted Memory from Gen 1—can tell you if the promotion rate is an issue. You want to avoid a high promotion rate from Gen 1. This is because you may have a lot of objects that live long enough to be promoted to Gen 2 but don't live long enough to stay in Gen 2. And once promoted to Gen 2, these objects become more expensive to collect than if they had died in Gen1. (This phenomenon is called midlife crisis. Take a look at blogs.msdn.com/41281.aspx for more details.) The CLR Profiler can help you find out which objects live too long.

High values for the Gen 1 and Gen 2 heap sizes are usually associated with high values in the promotion rate counters. You can check the GC heap sizes by using Gen 1 heap size and Gen 2 heap size. There is a Gen 0 heap size counter, but it doesn't measure Gen 0 size. Instead, it indicates the budget of Gen 0—meaning how much you can allocate in Gen 0 before the next Gen 0 collection will be triggered.

In all scenarios where you use a lot of objects that need finalization—for instance, objects that rely on COM components for some processing—you can take a look at the Promoted Finalization-Memory from Gen 0 counter. This tells you the amount of memory that cannot be reused because the objects that use the memory need to be added to the finalization queue and therefore cannot be collected immediately. IDisposable and the using statements in C# and Visual Basic® can help reduce the number of objects that end up in the finalization queue, thus reducing the associated cost.

More data on the heap sizes can be found using # Total committed Bytes and # Total reserved Bytes. These counters indicate the total committed and reserved memory, respectively, on the GC heap at that time. (The value of the total committed bytes is slightly larger than actual Gen 0 heap size + Gen 1 heap size + Gen 2 heap size + Large Object Heap size.) When the GC allocates a new heap segment, the memory is reserved for that segment but will be committed only when needed. Thus, the total reserved bytes can be bigger than the total committed bytes.

It's also worth checking if the application is inducing too many collections. The # Induced GC counter tells you how many collections have been induced since the process started. In general it is not recommended that you induce GC collections. In most cases, if # Induced GC indicates a high number, you should consider it a bug. Most of the time people induce GCs in hopes of cutting down the heap size, but this is almost never a good idea. Instead, you should find out why your heap is growing.

Windows Performance Counters

So far, we've looked at some of the most useful .NET Memory counters. However, you shouldn't overlook the value of other counters. There are various Windows Performance counters (which can also be viewed through perfmon.exe) that provide useful information for investigating memory issues.

The Available Bytes counter, listed under the Memory category, reports the available physical memory. This gives a good indication of whether you are running low on physical memory. If the machine is running low on physical memory, paging will either be happening or will soon begin to happen. This is useful data for diagnosing OOM issues.

The % Committed Bytes in Use counter, which also appears under the Memory category, provides the ratio of commit charge to commit limit. If this value is very high (say, more than 90 percent) you may begin to see commit failures. This is a clear indication that the system is under memory pressure.

Under the Process category, the Private Bytes counter indicates the amount of memory being used that can't be shared with other processes. You should monitor this if you want to see how much memory your process uses. If you are experiencing a memory leak, the private bytes will grow over time. This counter also gives a good indication of how your application impacts the entire system—using a lot of private bytes has a big impact on the machine since the memory cannot be shared with other processes. This is vital in some scenarios, such as terminal service, where you need to maximize the memory shared across user sessions.

Verifying an OOM Exception in a Managed Process

Performance counters can give you a clear indication of whether you are dealing with a memory issue. In most of the cases, though, a memory issue is detected only when you get an Out Of Memory exception in your application. So you need to find out if you are in fact dealing with an OOM exception that is caused by managed code.

After you've loaded SOS.dll, type the following command in the debugger:

!pe
This is the short form for !PrintException. Without an argument, it will print the last managed exception on the thread (if there is one). An example of a managed OOM exception is shown in Figure 1.

Figure 1 OutOfMemoryException for WinDbg (Close Figure 1)
0:010>!pe
Exception object: 39594518
Exception type: System.OutOfMemoryException
Message: <none>

InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    1A7BF848 789336E9 System.MulticastDelegate.CombineImpl(
                      System.Delegate)
    1A7BF87C 78930AC4 System.Delegate.Combine(System.Delegate,
                      System.Delegate)
    ... [omitted]
    1A7BFA9C 789B92A3 System.Threading._IOCompletionCallback.
                      PerformIOCompletionCallback(UInt32, UInt32,
                      System.Threading.NativeOverlapped*)

StackTraceString: <none>
HResult: 8007000e

If there is no managed exception on the current thread, you'll have to find out which thread the OOM is coming from. To find out, type the following into the debugger:

~*kb
Here, kb is short for Display Stack Backtrace. It lists all the threads with their call stacks (see Figure 2). In the output, look for the thread with the stack that has exception calls. The easiest way is to look for mscorwks::RaiseTheException.

Figure 2 Output of Display Stack Backtrace (Close Figure 2)
28adf5e4 7c822124 77e6baa8 000000c0 00000000 ntdll!KiFastSystemCallRet
28adf5e8 77e6baa8 000000c0 00000000 00000000
    ntdll!NtWaitForSingleObjec t+0xc
28adf658 77e6ba12 000000c0 ffffffff 00000000
    kernel32!WaitForSingleObjec tEx+0xac
28adf66c 791f2dde 000000c0 ffffffff 791f2d8e
    kernel32!WaitForSingleObjec t+0x12
...
28adf704 7c82ee84 28adfa9c 28adfbb8 28adf7bc ntdll!ExecuteHandler2+0x26
28adf7ac 7c82eda4 28add000 28adf7bc 00010007 ntdll!ExecuteHandler+0x24
28adfa8c 77e55dea 28adfa9c 25796140 e0434f4d ntdll!RtlRaiseException+0x3d
28adfaec 7924511d e0434f4d 00000001 00000000 kernel32!RaiseException+0x53
28adfb44 7923918f 5b61f2b4 00000000 5b61f2b4
    mscorwks!RaiseTheException+0xa0
...

The argument of the RaiseTheException function in mscorwks is the managed exception object. You can use !pe to dump it. In addition, !pe has a –nested option that will dump any nested exceptions in addition to the top-level ones.

Another way to find the thread that caused the OOM is to use the !threads command from SOS. The last column of the table displayed will contain the most recently thrown managed exception for the respective thread.

If you don't find an OOM exception using these techniques, there are no managed OOMs and the exception you're dealing with is being thrown by native code. In that case, you need to focus on the native code that your application uses (and that discussion is outside the scope of this column).

Determining What Caused an OOM Exception

After you've verified that you are dealing with an OOM case, you should check what caused the OOM. There are two legitimate cases for managed OOMs—either the process is running out of virtual memory or there is not enough physical memory available in order to commit.

The GC needs to allocate memory for its segments. When the GC decides that it needs to allocate a new segment, it calls VirtualAlloc to reserve space. If there isn't a contiguous free block large enough for a segment, the call fails and the GC cannot accommodate the new memory request.

In the debugger, the !address command shows you the largest free region of virtual memory. The output will look something like the following:

0:119>!address -summary
... [omitted]
Largest free region: Base 54000000 - Size 03b60000
0:119>? 03b60000
Evaluate expression: 62259200 = 03b60000
If the largest free block of virtual memory for the process is less than 64MB on a 32-bit OS (1GB on 64-bit), the OOM could be caused by running out of virtual memory. (On a 64-bit OS, it is unlikely that the application will run out of virtual memory space.)

The process can run out of virtual space if virtual memory is overly fragmented. It's not common for the managed heap to cause fragmentation of virtual memory, but it can happen. For example, it might occur if the app creates a lot of temporary large objects, causing the LOH to constantly acquire and release segments.

The !eeheap –gc SOS command will show you where each garbage collection segment starts. You can correlate this with the output of !address to determine if the virtual memory is fragmented by the managed heap.

The following are some other typical conditions that can lead to fragmented virtual memory:

  • Many small assemblies get loaded and unloaded all the time.
  • Lots of COM DLLs loaded because of COM interop.
  • Assemblies and COM DLLs are not both loaded in the managed heap. A common scenario that can lead to this is when an ASP.NET site has been compiled with the <debug> config flag enabled. This leads to every page being compiled in its own assembly, possibly fragmenting the virtual memory space enough to lead to OOMs.

Reserving memory does not require the OS to provide physical memory. Physical memory is assigned only when it is committed by the GC. If the system is running very low on physical memory, OOM exceptions should be expected. A simple way to check whether you are running low on physical memory is to open Windows Task Manager and look at the Commit Charge section on the Performance tab.

Figure 3 shows that the system has commited a total of 1981304KB and that the commit limit is 2518760KB. When total is close to limit, the system is running out of commit charge.

Figure 3 Viewing Commit Charge in Task Manager
Figure 3 Viewing Commit Charge in Task Manager(Click the image for a smaller view)
Figure 3 Viewing Commit Charge in Task Manager
Figure 3 Viewing Commit Charge in Task Manager (Click the image for a larger view)

The GC doesn't commit a whole segment all at once. Instead, it commits the segment in chunks as needed. (Note that how much the managed heap committed is indicated by the # Total committed Bytes counter, not # Bytes in all Heaps. This is because the Gen 0 size that is included in # Bytes in all Heaps is not the actual memory used in Gen 0, but rather its budget.)

You can use a user mode profiler, such as the CLR Profiler, to find out which objects are causing so much memory usage. In some cases, though, the overhead of running a profiler is not acceptable—for instance, when the problem needs to be debugged on a production server. In those cases, an alternative is to take memory dumps and then analyze them using the debugger. So let's take a look at how to use the debugger to analyze the managed heap.

Measure Managed Heap Size

When measuring managed heap size, the first thing you need to know is when to measure. Do you do this right before, right after, or during garbage collection? The best time to measure heap size consistently is right at the end of a Gen 2 collection because it collects the whole heap.

To look at objects at the end of a Gen 2 GC, set the following breakpoint in the debugger (for server GC, simply replace WKS with SVR):

bp mscorwks!WKS::GCHeap::RestartEE "j
(dwo(mscorwks!WKS::GCHeap::GcCondemnedGeneration)==2)
'kb';'g'"

Now that you have stopped at the end of a Gen 2 GC, the next step is to look at the objects on the managed heap. These objects are the survivors and you want to find out why they survived.

The !dumpheap –stat command performs a complete dump of the objects on the managed heap. (Thus, when you have a big heap, !dumpheap may take a while to finish.) The list that !dumpheap produces is sorted by memory used by type . This means you can start analyzing from the last few lines since they are the objects that take up the most space.

In the example in Figure 4, strings take up most of the space. If strings are the issue, the problem is often easy to solve. The content of the strings may tell you where they come from.

Figure 4 !dumpheap Output (Close Figure 4)
0:000>!dumpheap -stat
... [omitted]
2c6108d4   173712     14591808 MyClass.XtraGrid.Views.Grid.
                               ViewInfo.GridCellInfo
00155f80      533     15216804      Free
7a747c78   791070     15821400 System.Collections.Specialized.
                               ListDictionary+DictionaryNode
7a747bac   700930     19626040 System.Collections.Specialized.
                               ListDictionary
2c64e36c    78644     20762016 MyClass.MyEditor.ViewInfo.TextEditViewInfo
79124228   121143     29064120 System.Object[]
035f0ee4    81626     35588936 Toolkit.TlkOrder
00fcae40     6193     44911636 MyStrategyObject.TimeOver[]
791242ec    40182     90664128 System.Collections.Hashtable+bucket[]
790fa3e0  3154024    137881448 System.String
Total 8454945 objects

You can also look at strings in buckets. For example, you can check all strings that are between 150 to 200 in size, as shown in Figure 5. A lot of the strings in this sample are very similar. So instead of having so many strings, it is more efficient to store the common part ("PendingOrder-") and the numbers separately.

Figure 5 Viewing Strings of Specific Lengths (Close Figure 5)
0:000>!dumpheap -type System.String -min 150 -max 200
Using our cache to search the heap.
Address  MT            Size   Gen
1874c4f0 790fa3e0      160    2 System.String 15.59.03:
**** Transaction Completed. Account: 3344 Quantity: 50
18758218 790fa3e0      180    2 System.String PendingOrder-93344
187582cc 790fa3e0      180    2 System.String PendingOrder-12421
1875af38 790fa3e0      160    2 System.String 15.59.03:
**** Transaction Complete. Account: 7722 Quantity: 2
1875be20 790fa3e0      152    2 System.String PendingOrder-10889
1875bf74 790fa3e0      152    2 System.String PendingOrder-10778
... [omitted]

We have seen numerous scenarios where the managed heap contains the same string repeated thousands of times. The result is a big working set where much of the memory is consumed by strings. In this situation, it is often better to use string interning.

For other types that are not as obvious as string, you can use !gcroot to find out why these objects are alive (see Figure 6). Beware that the !gcroot command can take a long time if the object graph is very big.

Figure 6 Using !gcroot to Determine Why Objects Live (Close Figure 6)
0:000>!gcroot 1875bf74
Note: Roots found on stacks may be false positives.
Run "!help gcroot" for more info.
ebx:Root:19011c5c(System.Windows.Forms.Application+ThreadContext)->
19010b78(DemoApp.FormDemoApp)->
19011158(System.Windows.Forms.PropertyStore)->
... [omitted]
1c3745ec(System.Data.DataTable)->
1c3747a8(System.Data.DataColumnCollection)->
1c3747f8(System.Collections.Hashtable)->
1c376590(System.Collections.Hashtable+bucket[])->
1c376c98(System.Data.DataColumn)->
1c37b270(System.Data.Common.DoubleStorage)->
1c37b2ac(System.Double[])
Scan Thread 0 OSTHread 99c
Scan Thread 6 OSTHread 484
... [omitted]

Besides what survived on the managed heap, what gets allocated in Gen 0 is also part of your process's committed memory. If Gen 0 is allowed to grow big before the next garbage collection occurs, you may also observe committed memory being high due to this issue. This happens a lot more often on 64-bit Windows than on 32-bit. The !eeheap –gc SOS command will show you how big Gen 0 is.

What If Objects Survive?

Sometimes developers believe that their objects should be dead, but the GC doesn't seem to clean them up. The most common causes for this are:

  • There are still strong references to the objects.
  • The objects were not dead the last time their generation was collected.
  • The objects are dead, but a collection for the generations in which those objects live has not been triggered yet.

For the first and the second cases, you can use !gcroot to check whether there is a strong reference keeping the object alive. One possibility people often overlook is that an object may be held alive by the finalize queue while the finalizer thread is blocked because it cannot call into a single-threaded apartment (STA) thread that is not pumping messages to run the finalizer (see support.microsoft.com/kb/828988 for more details). You can determine if this is the problem by adding the following code:

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
This fixes the problem because WaitForPendingFinalizers pumps messages. However, once you confirm this is the issue, you should instead use Thread.Join since WaitForPendingFinalizers is quite heavyweight.

You can also verify whether this is the problem by running this SOS command:

!finalizequeue
And looking at the number of objects that are ready for finalization—not the number of "finalizable objects." When a finalizer is blocked, the finalizer thread shows which finalizer is currently being run, if any. (See Figure 7 for a sample finalization queue.)

Figure 7 Finalization Queue (Close Figure 7)
0:003> !finalizequeue
SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
generation 0 has 22 finalizable objects (002952f0->00295348)
generation 1 has 0 finalizable objects (002952f0->002952f0)
generation 2 has 0 finalizable objects (002952f0->002952f0)
Ready for finalization 0 objects (00295348->00295348)
Statistics:
MT      Count TotalSize Class Name
7911815c  1     20      Microsoft.Win32.SafeHandles.SafePEFileHandle
791003f4  1     20      Microsoft.Win32.SafeHandles.SafeFileMappingHandle
79100398  1     20      Microsoft.Win32.SafeHandles.SafeViewOfFileHandle
790fe034  2     40      Microsoft.Win32.SafeHandles.SafeFileHandle
790fb330  1     52      System.Threading.Thread
79111420  1     80      System.IO.FileStream
7910f304  15   300      Microsoft.Win32.SafeHandles.SafeRegistryHandle
Total 22 objects
0:000>!gcroot 1875bf74
Note: Roots found on stacks may be false positives.
Run "!help gcroot" for more info.

An easy way to get to the finalizer thread is by looking at the output of !threads-special. The stack shown in Figure 8 shows the state the finalizer thread is usually in—it's waiting for an event to indicate that there are finalizers to be run. When a finalizer is blocked, you will see that finalizer being run.

Figure 8 Finalizer Thread Waiting for Finalizer (Close Figure 8)
0:005>!threads -special
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                              PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
   0    1  9b0 00181320      a020 Enabled  00000000:00000000 0014c260     0 MTA
   3    2  c18 0018b078      b220 Enabled  00000000:00000000 0014c260     0 MTA (Finalizer)

       OSID     Special thread type
    2    cd0    DbgHelper
    3    c18    Finalizer
    4    df0    GC SuspendEE
0:003>~3s;k
eax=00000000 ebx=00dffd20 ecx=00152ba0 edx=00000000 esi=00dffd24 edi=7ffdf000
eip=7ffe0304 esp=00dffcd4 ebp=00dffd7c iopl=0     nv up ei pl zr na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000        efl=00000246
SharedUserData!SystemCallStub+0x4:
7ffe0304 c3               ret
ChildEBP RetAddr
00dffcd0 77f4372d SharedUserData!SystemCallStub+0x4
00dffcd4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
00dffd7c 77e4b0e4 KERNEL32!WaitForMultipleObjectsEx+0x11a
00dffd94 7a01dd9a KERNEL32!WaitForMultipleObjects+0x17
00dffdb4 7a01e695 mscorwks!WKS::WaitForFinalizerEvent+0x7a
...
00dfff14 7a0983ef mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb
00dfffb8 77e4a990 mscorwks!Thread::intermediateThreadProc+0x49
00dfffec 00000000 KERNEL32!BaseThreadStart+0x34

The third cause shouldn't be an issue. Normally, unless you manually induce garbage collection, a collection is only triggered when the GC considers it to be productive. This means an object might be dead but its memory will not be reclaimed right away. But when the physical memory pressure on the system is high, the GC becomes more aggressive.

Is Fragmentation a Problem on Your Managed Heap?

When you investigate memory issues, fragmentation is a major concern. It's important because you want to know just how much space is wasted on the managed heap. The amount of fragmentation for a managed heap is indicated by how much space free objects take on the heap. You can use the !dumpheap command to find out how much free memory is available on the managed heap, like so:

0:000>!dumpheap -type Free -stat
total 230 objects
Statistics:
      MT    Count    TotalSize   Class Name
00152b18      230    40958584          Free
Total 230 objects
In this sample, the output indicates that there are 230 free objects for a total of about 39MB. Thus, the fragmentation for this heap is 39MB.

When you try to determine if fragmentation is a problem, you need to understand what fragmentation means for different generations. For Gen 0, it is not a problem because the GC can allocate in the fragmented space. For Gen 1 and Gen 2, fragmentation can be a problem. In order to use the fragmented space in Gen 1 and Gen 2, the GC would have to collect and promote objects in order to fill those gaps. But since Gen 1 cannot be bigger than one segment, Gen 2 is what you typically need to worry about.

Excessive pinning is usually the cause of too much fragmentation. In the .NET Framework 2.0 a lot of work was done to reduce fragmentation caused by pinning (see the blog entry at blogs.msdn.com/476750.aspx for more information on GC improvements in the .NET Framework 2.0), but you still may see high levels of fragmentation if the app simply pins too much. You can use an SOS command, !gchandles, to see the number of pinned handles (see Figure 9). And you can use !objsize to find out which objects are pinned, as shown in Figure 10.

Figure 10 Determining Which Objects Are Pinned (Close Figure 10)
0:003> !objsize
Scan Thread 0 OSTHread 1a9cDOMAIN(0024F0F0):HANDLE(WeakSh):2212fc: sizeof(01231d30) = 52 (0x34) bytes (System.Threading.Thread)
DOMAIN(0024F0F0):HANDLE(Pinned):2213e8: sizeof(02234260) =
    4096 (0x1000) bytes (System.Object[])
DOMAIN(0024F0F0):HANDLE(Pinned):2213ec: sizeof(02233250) =
    4108 (0x100c) bytes (System.Object[])
DOMAIN(0024F0F0):HANDLE(Pinned):2213f0: sizeof(02233030) =
    620 (0x26c) bytes (System.Object[])
DOMAIN(0024F0F0):HANDLE(Pinned):2213f4: sizeof(02232020) =
    5276 (0x149c) bytes (System.Object[])
DOMAIN(0024F0F0):HANDLE(Pinned):2213f8: sizeof(0123118c) =
    12 (0xc) bytes (System.Object)
DOMAIN(0024F0F0):HANDLE(Pinned):2213fc: sizeof(02231010) =
    102632 (0x190e8) bytes (System.Object[])
Figure 9 Checking the Number of Pinned Handles (Close Figure 9)
0:002> !gchandles
GC Handle Statistics:
Strong Handles: 16
Pinned Handles: 4
Async Pinned Handles: 0
Ref Count Handles: 0
Weak Long Handles: 0
Weak Short Handles: 1
Other Handles: 0
Statistics:
      MT    Count    TotalSize    Class Name
790f9d10      1          12       System.Object
790fb760      1          28       System.SharedStatics
791077ec      1          48       System.Reflection.Module
790fc894      2          48       System.Reflection.Assembly
790fad68      1          72       System.ExecutionEngineException
790facc4      1          72       System.StackOverflowException
790fac20      1          72       System.OutOfMemoryException
790fb9c0      1         100       System.AppDomain
790fb330      2         104       System.Threading.Thread
790fd91c      4         144       System.Security.PermissionSet
790fae0c      2         144       System.Threading.ThreadAbortException
79124314      4        8744       System.Object[]
Total 21 objects

Fragmentation in the LOH is by design because we don't compact the LOH. This does not mean allocating on the LOH is the same as malloc using the NT heap manager! Due to the nature of how the GC works, free objects that are adjacent to each other are naturally collapsed into one big free space, which is available for large object allocation requests.

Measuring Time Spent on Garbage Collection

Developers often want to know how much time the GC takes for each collection. This is usually important data in a soft real-time scenario where there are some constraints on, for instance, the response time that the application must respect. And, of course, this is an important consideration, since too much time spent on garbage collection means CPU time taken away from actual processing.

The easiest way to get an idea of how much time garbage collection takes is to look at the % Time in GC performance counter. This counter is updated at the end of a collection and indicates the ratio of the time spent in the GC just finished divided by the time since the end of the last GC. If no collection happens during the sampling interval, the counter is not updated and you get the same value you got last time. Since you know the sampling interval in the performance monitor application (the default sampling interval in PerfMon is 1 second), you can roughly calculate the time.

Figure 11 gives some sample garbage collection data. In it, you'll see that Gen 0 collections occurred in the second and the third intervals. Since we don't know exactly when the collection happened during these intervals, this technique isn't 100 percent accurate. But it is useful for estimating time spent in the GC.

Figure 11 Sample GC Data (Close Figure 11)

Timestamp (Interval) | # Gen 0 Collections | # Gen 1 Collections | # Gen 2 Collections | % Time in GC
1 | 9 | 3 | 1 | 10
2 | 10 | 3 | 1 | 1
3 | 11 | 3 | 1 | 3
4 | 11 | 3 | 1 | 3

Considering this sample, here's the worst case scenario for the eleventh GC. Say the tenth Gen 0 collection finished at the beginning of the second interval and the eleventh Gen 0 collection finished at the end of the third interval. That means the time between the end of the two collections is about two sampling intervals, or two seconds. The % Time in GC counter shows 3 percent, and so the eleventh Gen 0 collection took 3 percent of 2 seconds (or 60ms).

Investigating High CPU Usage

When a collection is happening, CPU usage is supposed to be high so the GC can complete as quickly as possible. Figure 12 shows an example of when CPU usage is pretty much always caused by collections. All the spikes in the % Process Time counter correspond directly to a change in % Time in GC. Obviously, this should never happen in reality since, in addition to the GC using the CPU, other processes will also use the CPU. To figure out what else is consuming CPU cycles, you can use a CPU profiler to look at the functions that consume the most CPU time.

Figure 12 When CPU Usage is Caused by Collections
Figure 12 When CPU Usage is Caused by Collections(Click the image for a smaller view)
Figure 12 When CPU Usage is Caused by Collections
Figure 12 When CPU Usage is Caused by Collections (Click the image for a larger view)

If indeed you are seeing too much CPU power being consumed by the GC, there are too many collections taking place or collections are taking too long. Consider the case when collections are triggered by allocations. The allocation rate is the major factor that determines how often collections are triggered.

Figure 13 Less Accurate Data when Collections Are Spread Out
Figure 13 Less Accurate Data when Collections Are Spread Out

When a collection begins, the value of the Allocated Bytes/sec counter is updated by adding the number of allocated bytes in Gen 0 and the LOH. Since the counter is expressed as a rate, the actual number you see is the difference between the last two values divided by the time interval. For example, Figure 13 illustrates what happens if the sampling rate is 1 second and a collection occurs only after some intervals have passed. When the second collection occurs, the performance counter is updated like so:

Allocation = 850-250=600KB
Alloc/Sec = 600/3=200KB/sec

Figure 14 More Accurate Data if Collection Occurs Frequently
Figure 14 More Accurate Data if Collection Occurs Frequently

Even though the application keeps allocating, the performance counter doesn't reflect that because it doesn't get updated until the next collection. If a collection happens more frequently, you get a better picture (see Figure 14).

A common mistake is to consider the allocation rate as the measure of how long garbage collections take—how long a collection takes is decided by how much memory the GC has to go through. Since the GC only goes through survived objects, a long collection means many objects survived. If you encounter this, use the technique we discussed earlier to determine why you have so many survivors.

Send your questions and comments to clrinout@microsoft.com.

Claudio Caldato is a Program Manager for Performance and Garbage Collector on the Common Language Runtime team at Microsoft.

Maoni Stephens is the main developer working on the Garbage Collector in the Common Language Runtime team at Microsoft. Before joining CLR, Maoni spent several years in the Operating System group at Microsoft.


Cutting Edge: A Tour of Windows Workflow Activities

Cutting Edge
A Tour of Windows Workflow Activities
Dino Esposito

Code download available at: CuttingEdge2006_11.exe (170KB)
Contents
Out-of-the-Box Activities
Controlling Flow through Conditions
Loops and Repeated Activities
Synchronization Blocks
Exception Handling
Transactions
Calling Out Other Code
Summary

Most enterprise applications contain business processes that developers and architects find natural to express through workflows. Ultimately, a workflow is a program that captures the interactions among real-world entities. The workflow waits for stimulus from the external world, and the arrival of that stimulus could take arbitrarily long periods of time.

A pillar of the upcoming Microsoft® .NET Framework 3.0, Windows® Workflow Foundation provides the programming model and runtime engine for building workflow-enabled applications on a Windows platform equipped with the .NET Framework 2.0 or higher, running at least Windows XP Service Pack 2 (SP2) or Windows Server® 2003.

Don Box and Dharma Shukla wrote a Windows Workflow Foundation primer in the January 2006 issue of MSDN®Magazine. In recent Cutting Edge columns (March 2006 and April 2006), I addressed a very common type of workflow application where the execution of the workflow is conditioned by human intervention (see June 2003 Cutting Edge column and June 2004 Cutting Edge column). From a software perspective, the need for a human operator mostly means asynchronous event handling, state persistence, and the availability of a shared server. Windows Workflow Foundation supports virtually any scenario where human operators are involved.

Each workflow is a declarative program wherein each program statement is represented in terms of a component called an activity. One of the biggest myths about Windows Workflow Foundation is that all the constituent activities are linked either sequentially or by state-machine transition. In fact, Windows Workflow Foundation virtualizes the execution model in terms of an activity automata. This enables you to write composite activities which can capture any type of control flow pattern, ranging from fancy joins and merges to state machines to graphs to sequences to interleavings to non-local exits, and so on. In short, it allows you to model the control flow patterns that exist in the real world, through high-fidelity composite activities. However, Windows Workflow Foundation ships with built-in activities for only two patterns: sequential and state-machine workflows.

A sequential workflow has clear starting and ending points and execution proceeds from start to finish, one step after the next, following one possible route. State-machine workflows are expressed as a set of states, each of which may contain activities and is triggered by an event. You define an initial and a completed state and execution proceeds from state to state following all defined transitions until the completed state is reached.

As I mentioned, the building block of the workflow is the activity. Building such a workflow means combining activities to create the desired model for the problem to face. Windows Workflow Foundation comes with a large set of built-in activities, but it is also completely extensible through custom activities. Understanding goals and capabilities of the out-of-the-box activities is key to making full sense of the true potential of the platform. This column will provide a tour of the standard workflow activities you get when you install the Windows Workflow Foundation runtime and the Visual Studio® 2005 designer.

Out-of-the-Box Activities

Windows Workflow Foundation is part of the .NET Framework 3.0 (formerly code-named WinFX®). Even though it's still a beta, I found the latest Community Technology Preview (CTP) to be very stable; a number of companies are extensively using it to develop applications. As a result, a number of tools and custom activities are already available and a useful support community is growing on newsgroups and forums such as wf.netfx3.com. If you have, or will soon have, some interest in workflow applications, be aware that it's never too early to get properly acquainted.

To find the latest download, go to Windows Workflow Foundation. Once it's installed, start Visual Studio 2005, open a Visual Basic® or C# workflow project, and take a look at the contents of the toolbox. Depending on the type of project you select, it should look like one of the toolboxes in Figure 1.

The contents of the toolbox are context-sensitive and change based on the type of workflow you select. In particular, the left-hand toolbox in Figure 1 also includes the state activities required when you create a state-machine workflow. The toolbox window on the right only includes activities for a sequential workflow. The set of predefined activities can be split into a few categories based on the goals and expected behavior. Figure 2 shows a possible categorization: control flow, execution, events, Web services, and state machines.

Figure 2 Categories of Workflow Activities (Close Figure 2)

Flow | Execution | Events | Web Services | State
IfElse | TransactionScope | EventDriven | WebServiceInput | SetState
While | Sequence | Listen | WebServiceOutput | State
CAG | Compensate | HandleExternalEvent | WebServiceFault | StateInitialization
Policy | SynchronizationScope | EventHandlingScope | InvokeWebService | StateFinalization
Replicator | Code | Delay |  |
Throw | Parallel |  |  |
Suspend | InvokeWorkflow |  |  |
Terminate | CallExternalMethod |  |  |
FaultHandler | CompensatableSequence |  |  |
 | CompensatableTransactionScope |  |  |
Figure 1 Workflow Activities in Toolbox
Figure 1 Workflow Activities in Toolbox(Click the image for a smaller view)
Figure 1 Workflow Activities in Toolbox
Figure 1 Workflow Activities in Toolbox (Click the image for a larger view)

Control flow activities govern the sequence in which the graph activities execute. The list includes conditional blocks as well as constructs for implementing loops. Execution activities include constructs to terminate or suspend a workflow, throw an exception, execute internal or external code, or spawn another workflow. Most workflows need synchronization with the host environment—typically a Windows Forms or ASP.NET application. The activities in the events group let the workflow stop waiting for external events, handle received events, or just wait before proceeding with the next step. Finally, the Web services and state groups target two special features of a workflow—exposing the internal engine as a Web service and as a state machine. Let's examine the key activities in the framework, proceeding by groups.

Controlling Flow through Conditions

When conditions are specified in a workflow, the runtime engine evaluates them and then acts based on the result. Two of the core condition-based activities are IfElse and Policy. The IfElse activity works like a classic If statement in a high-level programming language. It may contain any number of condition-based branches and also a default branch to fall into if no other condition is met. The Policy activity, instead, represents a collection of rules. In Windows Workflow Foundation, a rule consists of a condition and one or more resulting actions. Think of a rule as an If-Then-Else statement, where the condition corresponds to the Boolean guard of the If block, and the actions define the behavior of the Then and Else clauses. Let's get to know more about each of these activities and then compare them.

When added to a workflow, the IfElse activity looks like Figure 3. By default, it has two branches, and new branches can be added by right-clicking and selecting Add Branch from the context menu. When the runtime engine reaches an IfElse activity it begins evaluating the condition of the various branches proceeding from the left to right. The condition of each branch is determined and the first branch that evaluates to true is run. You can alter the evaluation order of the branches by moving branches around through the commands in the activity's context menu. To enable a branch, you need to specify a valid condition. You specify a condition in either of two ways—through an expression or through a piece of code.

Figure 3 IfElse Activity in Action
Figure 3 IfElse Activity in Action(Click the image for a smaller view)
Figure 3 IfElse Activity in Action
Figure 3 IfElse Activity in Action (Click the image for a larger view)

If you opt for an expression, set the branch in the designer and give a public home to the Condition entry in the Visual Studio 2005 Properties box. Next, expand the expression editor and enter an expression that touches on workflow members and evaluates to a Boolean value. For example, if the workflow class features a public property named MinimumLength you can set the condition as follows:

this.MinimumLength >= 8
The editor shown in Figure 4 fully supports IntelliSense® and provides access to private and public members of the workflow. This means that you call out a Boolean function as well. Suppose you have a property declared like this:
Public Property MinimumLength() As Integer
   Get
      Return _minLength
   End Get
   Set(ByVal value As Integer)
       _minLength = value
   End Set
End Property

Figure 4 Condition Editor for IfElse Activities
Figure 4 Condition Editor for IfElse Activities(Click the image for a smaller view)
Figure 4 Condition Editor for IfElse Activities
Figure 4 Condition Editor for IfElse Activities (Click the image for a larger view)

When you try to read the value of MinimumLength, the runtime ends up invoking the get accessor of the property, which returns the value of the private member _minLength. The final branch in an IfElse activity can lack a condition. In this case, it will work as the Else branch of the IF workflow statement.

The second way to specify a condition is through ad hoc code. In this case, you add a method to the workflow class, as follows:

Sub EvalCondition(ByVal sender As Object, _
                  ByVal e As ConditionalEventArgs)
   e.Result = ... ' Boolean value
End Sub
The function must match the signature that accepts an object and a ConditionalEventArgs class and returns void. The Result member of the ConditionalEventArgs class will be set to the Boolean value that represents the evaluation of the condition.

IfElse activities can be nested to express complex logic, but the design of the logic remains hardcoded in the workflow. This is good and bad at the same time. It's a good thing because it allows you to design the workflow exactly as you want and integrate it with the rest of the activities. It's a bad thing when you need to create a set of rules to initialize portions of the workflow's state. Having a bunch of IfElse activities only to assign values to internal members of the workflow is clearly overkill.

You should use a combination of IfElse activities when you really need to control the flow of the workflow and orchestrate various blocks of the work. If what you need is just a sequence of programmatic If statements with simple code attached to branches, then you're better off using a Policy activity. A Policy activity is a collection of rules. The code associated with each rule, though, is limited to setting a property on a workflow or calling a workflow method or a static method on types in referenced assemblies.

The activity is associated with a sorted collection of rules that you define through the editor. Each rule has a priority and an Active property. The combination of the two properties determines whether the rule should be evaluated and with which priority. In addition, a rule must specify a reevaluation behavior—Always (default) or Never. If set to Always, the rule is reevaluated as needed, otherwise it is evaluated only the first time and never changes whatever happens to the workflow's state.

Instead, the rule set as a whole is subject to forward chaining. Simply put, forward chaining refers to the ability of the actions of one rule to cause other dependent rules to be reevaluated. For example, an action that updates the value of a field that another rule tests requires a reevaluation of the involved rules (unless reevaluation is disabled on the rule). There are three types of forward chaining.

Implicit chaining indicates that the runtime engine will figure out which fields are modified by some actions and automatically reevaluate them. This mechanism succeeds as long as actions work on properties explicitly. What if you have an action like the following:

IF this.MinimumLength <= 8 THEN RefreshInternalState()
Who knows what the RefreshInternalState method is going to do? The method might touch on workflow properties that are involved with other rules in the policy activity. By using attributes on the method declaration, you can explicitly indicate the behavior of the method:
<RuleWrite("PasswordLevel")> _
Public Sub RefreshInternalState()
    Me.PasswordLevel = 1
End Sub

The RuleWrite attribute indicates that the method is going to modify the specified property; likewise, the RuleRead attribute indicates that the method reads from the specified properties. In this way, unequivocal information is provided to the runtime engine to keep the rule set in sync.

Finally, you can write actions that explicitly call out updates on involved properties. Here's an example:

IF this.MinimumLength <= 8 THEN
    RefreshInternalState()
    Update("PasswordLevel")
The Update method in the rule action orders the reevaluation of all rules that include the specified property.

Loops and Repeated Activities

The native set of activities provides classic While activity as well as a Replicator activity that has some points in common with a classic For loop. The While activity accepts a condition and evaluates it at the beginning of each iteration. If the condition is true, the activity runs the specified child activity and repeats until the condition becomes false. Note that a single activity is permitted in the body of the While. For this reason, you might want to use a composite activity such as Sequence or Parallel to execute multiple activities in the loop. (The term "interleaved" is probably more appropriate than "parallel" here. There is no true concurrency going on with the Parallel activity, only interleavings within the same thread.)

Similar to a Foreach statement, the Replicator activity creates and executes a given number of instances of the specified child activity. You can specify only one child activity, but using composite or custom activities is allowed. You can't control the number of iterations through a declarative property. Instead, you write a handler for the Initialized event and populate the CurrentChildData collection with initialization data for each of the desired instances:

Sub replicator1_Initialized(ByVal sender As Object, _
                            ByVal e As EventArgs)
   Console.WriteLine("Initializing ...")
   replicator1.CurrentChildData.Add("You are #1")
   replicator1.CurrentChildData.Add("You are #2")
   replicator1.CurrentChildData.Add("You are #3")
End Sub

The preceding code snippet orders three instances of the child activity of the Replicator, each initialized with a given string. Note that if you leave the CurrentChildData collection empty, the Replicator won't run any child activity and is limited to firing top-level events such as Initialized and Completed. You can initialize child activities using an object (not necessarily a string), including instances of custom classes. The Replicator also includes events to signal initialization and completion of each child activity. By default, child instances run in sequence; by setting the ExecutionType property, though, you can opt for a parallel execution. When a parallel execution is requested, all child activity instances get created before the replicator starts and executes on parallel threads. When working in sequential mode, the next activity is instantiated only when the previous has finished.

If no global condition is set through the UntilCondition property, the Replicator ends when all replicates have completed; otherwise, the activity terminates when the UntilCondition is true. Note, though, that the activity will hang if all the child activities have completed and the UntilCondition property evaluates to false; the Replicator never loops through the child activities like in a While loop. The UntilCondition is evaluated after the global initialization of the activity, after each child activity has completed, and after all included activities have completed. If, at any time, the condition evaluates to true, the Replicator will exit immediately.

You can mix together conditional execution and loops in the ConditionedActivityGroup (CAG) activity. The CAG contains a number of child activities (possibly composite activities) and runs them until a global condition is met. Basically, a CAG combines the behavior of While and IfElse activities. Its internal logic is expressed by the following pseudocode:

While condition
   If child1.WhenCondition Then child1.Execute
   If child2.WhenCondition Then child2.Execute
    :
   If childN.WhenCondition Then childN.Execute
End While

Each child activity has a WhenCondition property. Based on the evaluation of the specified condition, the activity is run or skipped in the current iteration. Note, though, that if the child activity has no WhenCondition set, it is executed only the first time and skipped for all subsequent iterations. All conditions are evaluated whenever required based state change dependencies.

The CAG activity terminates after the UntilCondition returns true and immediately cancels any currently executing activities. If no condition is specified, CAG completes when no child activities are running either because they have no condition set or the condition evaluates to false.

Synchronization Blocks

A sequential workflow that has no interleaved activities doesn't bother about serializing access to shared members. However, operation is different when a Parallel activity is used with two or more sequences running in an interleaved manner. Take a look at Figure 5. The While activity contains a Sequence block which, in turn, chains Parallel and Code activities. Internally, the Parallel activity has four different blocks, each performing a mathematical operation on a number defined as a member of the workflow class. The four branches in the Parallel activity execute in an interleaved manner. (There is only one thread in a workflow instance at any one time. That thread switches back and forth between the branches of the Parallel activity.) What about cross-task synchronization? By default, each piece of code running inside any of the child activities will perform a direct access to the shared member.

Figure 5 Parallel Tasks
Figure 5 Parallel Tasks(Click the image for a smaller view)
Figure 5 Parallel Tasks
Figure 5 Parallel Tasks (Click the image for a larger view)

With the simple schema of Figure 5, it may happen that each parallel branch runs as an atomic operation. This would be purely coincidental, though, and strictly based on the complexity and the duration of each operation. What if each branch is made of multiple activities? SynchronizationScope provides a declarative and explicit way of modeling synchronized access of shared state within a given workflow instance across a set of activities.

The section of the workflow running inside a SynchronizationScope activity is a sort of atomic operation that can't be interrupted. There's no transactional semantics here and the operation wrapped up in the SynchronizationScope activity cannot be rolled back. The SynchronizationScope activity plays a key role in all solutions based on activities that have parallel execution paths, such as Parallel, Replicator, and CAG.

To quickly figure out the role of a SynchronizationScope activity, take a look at Figure 6, which represents a modified version of the workflow in Figure 5. The blockMultiply activity counts two code blocks interspersed with a delay. The blockDivide activity contains one code block with no synchronization. If you place the contents of the blockMultiply outside of a synchronization scope, the blockDivide will likely execute during the delay phase of blockMultiply. Depending on what kind of updates you're making, this might be a serious issue. The SynchronizationScope activity safeguards the continuity of a section of the workflow.

Figure 6 Parallel Tasks and Synchonization Scopes
Figure 6 Parallel Tasks and Synchonization Scopes(Click the image for a smaller view)
Figure 6 Parallel Tasks and Synchonization Scopes
Figure 6 Parallel Tasks and Synchonization Scopes (Click the image for a larger view)

Exception Handling

Just as any other .NET-based application, a workflow can throw and catch exceptions. In particular, you use the Throw activity to throw a particular exception. The Throw activity requires a couple of settings—the type of the exception to throw and a workflow class member to store an instance of the exception object for further customization. You can use custom exception types as long as the type is referenced in the workflow project.

To catch exceptions thrown by workflow activities, you add a FaultHandler activity. The workflow designers provide a container for all fault handlers you add. Each handler is bound to a workflow section that executes as soon as the exception is caught. After adding the FaultHandler activity, you define the type of the exception to catch. You can access the exception object through the Fault property. Note that to visualize fault handlers you have to switch the view in the Visual Studio 2005 designer and select the View Faults tab by clicking on the tab list at the bottom of the designer's view.

Transactions

The .NET Framework 2.0 includes an object to handle transactions easily and effectively—no matter the number and type of participant objects and the scope, local or distributed, of the transaction. The object is named TransactionScope and you typically use it according to this pattern:

Using ts As New TransactionScope
   ...
   ts.Complete
End Using

The TransactionScope object guarantees that the transaction is either committed or rolled back in case of failure and, more importantly, it determines whether you need a local or distributed transaction and enlists any necessary resources. As the code reaches a point where it won't be running locally, TransactionScope escalates to the Distributed Transaction Coordinator (DTC) as appropriate. Any objects that implement the ITransaction interface can be enlisted with a transaction. The list includes all standard ADO.NET 2.0 data providers, while Microsoft Message Queue works in compatibility mode.

It is worth noting that there are a number of differences between TransactionScope and Enterprise Services as far as distributed transactions are concerned. TransactionScope belongs to a transaction framework designed specifically to fit .NET-based applications—System.Transactions. Internally, though, the classes of the System.Transactions namespace sometimes end up delegating some work to DTC and COM+. Why is TransactionScope important here? The TransactionScope Windows Workflow Foundation activity is just a workflow wrapper for an instance of the TransactionScope .NET class.

When you add a TransactionScope activity to a workflow, you can set the timeout for the transaction to complete and the desired level of isolation. All activities composed in the scope of the transaction form a unit of work that fulfills the classic ACID schema. When all child activities have successfully completed, the transaction commits and the workflow proceeds with the next step. The TransactionScope activity automatically rolls back if an exception is thrown from within the scope. Writing transactional workflows doesn't require you to explicitly deal with commit and rollback semantics—you just let things go if you want to commit and throw an exception to abort. The activity will comfortably manage everything else.

Note that you can't nest two or more TransactionScope activities. Likewise, you can't suspend the workflow from within a transaction using the Suspend activity. Instead, you can incorporate event handlers in the transaction through either the Listen or the EventHandlingScope activities. In this case, though, the workflow host should include a persistence service, otherwise an exception would be thrown if the workflow attempts to save its state on idle.

A related activity, CompensatableTransactionScope, also supports compensation. Compensation is the process of logically undoing the completed transactions in case of any subsequent business exceptions. Compensation differs from rollback in that it is a way to cancel the effects of a successfully completed and committed transaction when a rule is violated later in the workflow. The typical example of a rollback is when you have a transaction that transfers money across two accounts. The first call withdraws money from one account and the second call adds the same amount to another account. As long as the database supports the two-phase commit model, the rollback restores a consistent state in case of exceptions thrown within the transaction.

Imagine an order processing workflow where a transaction is used to process payment through, say, a credit card service. As the first step, money is withdrawn from the credit card to pay for some goods. Next, after the transaction completed successfully, it turns out that the goods are no longer available for sale. Such a business rule violation requires that the results of the transaction are compensated with proper code—typically crediting money to the charge card.

By right-clicking on the CompensatableTransactionScope activity, you can switch to the compensation view of the transaction and add all the activities that would compensate the effects of the transaction. In Figure 7, the OrderScope transaction is associated with a compensation activity code that will refund any money withdrawn in case of a business exception. As I mentioned earlier, exceptions in the workflow are caught by an exception handler. For a given exception (say, ProductDiscontinued) you invoke a Compensate activity bound to the transaction activity whose effects are canceled. In general, the Compensate activity triggers the compensation mechanism for any activity that is compensatable—that is, for any activity that implements the ICompensatableActivity interface. Along with CompensatableTransactionScope, CompensatableSequence also implements this interface and can be used for nontransaction-based compensation scenarios. However, you can write custom activities that support compensation as well.

Figure 7 Transaction and Compensation Activities
Figure 7 Transaction and Compensation Activities(Click the image for a smaller view)
Figure 7 Transaction and Compensation Activities
Figure 7 Transaction and Compensation Activities (Click the image for a larger view)

Calling Out Other Code

There are a few other activities to invoke and execute code. They are Code, InvokeWorkflow, InvokeWebService, and CallExternalMethod. The Code activity is the most flexible and represents a block of custom code you insert at some point in the workflow. The Code activity fires the ExecuteCode event that you handle within the workflow and makes it run the code you want. The code, though, is incorporated in the workflow and compiled with it. The Code activity can call into an external assembly as long as you put into the ExecuteCode event handler any necessary code that loads the assembly and selects the method.

The InvokeWorkflow activity takes a reference to a workflow and invokes it asynchronously. You can pass input parameters but be aware that the InvokeWorkflow activity completes before the launched workflow starts its execution. In no way can you synchronize the current execution with that of an external workflow or process the output parameters in the current workflow.

The InvokeWebService activity synchronously invokes a Web service method through a proxy class. The behavior of the activity can be simulated using a Code activity and calling a Web service from there.

Finally, the CallExternalMethod activity is used to call a method on a local service. A local service is any class available to the workflow that implements an interface decorated with the ExternalDataExchange attribute. In addition, the class that represents the local service must be added to the workflow runtime as an external data exchange service:

Dim service As New ExternalDataExchangeService
runtime.AddService(service)
Dim localService As New YourService()
service.AddService(localService)

You configure a CallExternalMethod activity to call into any of the methods of the interface implemented by the local service. Why is CallExternalMethod limited to calling into the methods of a well-known interface? The reason lies in the fact that the activity is not a mere code executor but is designed to work in conjunction with the HandleExternalEvent activity to implement a two-way communication with a local service. With the CallExternalMethod activity you invoke a method on the service; you handle any event raised by the service by placing a HandleExternalEvent activity in the workflow.

Summary

Activities are the building blocks of workflows. You create workflow solutions by composing activities in the Visual Studio 2005 designer. Like server controls in ASP.NET and controls in Windows Forms, workflow activities are the essence of the solution and form the primary tool chest for developers. The Windows Workflow Foundation provides a number of in-box activities to cover the most common needs. Here I reviewed the native activities that cover transactions, parallel and conditional execution, loops, and exception handling. A number of free additional activities are available at wf.netfx3.com. Check them out!

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Bugslayer: Minidumps for Specific Exceptions

Bugslayer
Minidumps for Specific Exceptions
John Robbins

Code download available at: Bugslayer2006_11.exe (164KB)
Contents
Getting Started
Minidumps on All .NET Exceptions
Minidumps on a Specific .NET Exception
What About Vectored Exception Handling?
Wrap Up
Tips

Necessity is the mother of invention, and it's usually when you're faced with a nasty bug that most of the invention happens in software development. Recently I was working on a super nasty bug in an ASP.NET 2.0 application and needed to see the state of the worker process exactly at the point where it threw a specific exception. By the time I got the exception in the catch block, it was too late. Because the bug only showed up in the production environment, because it was a mission-critical application (as always!), and because I needed to fix the bug yesterday, there was no way I would have the luxury of poking at the application with the Visual Studio® 2005 debugger. Thus began the quest to find a way to get a minidump of the ASP.NET worker process at the instant an exception is thrown.

In this installment of Bugslayer, I'll cover the use of ADPlus to create a minidump of your Microsoft® .NET Framework 2.0 processes on specific exceptions. Because the configuration files for ADPlus are now mostly documented, we can jump right in. When I first started thinking about the problem, I wasn't sure it would be possible to get a minidump for just a single .NET exception type and not others. However, I was surprised at just how simple it turned out to be. I'll end the column with my attempt to fake out .NET and get the same information written from my code without using any external tools. As you can guess from my usage of the word "attempt," I failed, but it was a worthy exercise.

If you're not familiar with ADPlus, it's probably the world's largest VBScript program, and it drives the console debugger (CDB) to snap minidumps, set breakpoints, and otherwise help you automate the debugging of server-based applications in particular. ADPlus is part of the Debugging Tools for Windows® package, which you can download for free. Documentation for ADPlus is in the Debugger.chm in the same directory where ADPlus.vbs resides. You can read more about ADPlus in the March 2005 Bugslayer column. For this month's column, I'm assuming that you've read the ADPlus documentation.

Before I do anything else, I have to remind you that CDB is a native program debugger and doesn't know anything about .NET. Only if you load the Son of Strike (SOS) debugging extension, located in the .NET Framework installation directory, can you get any .NET-specific information from inside CDB. With that in mind, all the ADPlus documentation speaks in terms of native debugging. The key point to remember when reading the documentation is that a first-chance exception, in .NET terms, is when you throw the exception. A second-chance exception is when the application is crashing with an unhandled exception.

Getting Started

The source code download for this month's column contains all the configuration files I'm going to talk about, and you may want to consider running them as you read through the descriptions. As far as ADPlus and CDB are concerned, there's no difference between W3WP.EXE, the ASP.NET worker process for IIS 6.0, and a Windows Forms application. To make it easier to play with the different configurations, I included a simple application, ExceptionMaker, which throws three different exceptions so you can see how the different ADPlus configuration files perform.

One thing that's tricky about ADPlus is that all command-line options that take files or paths need to have the complete drive and path included. In each of the configuration file directories are some batch files that make it easier to test the configurations, but you'll need to edit them to include the complete path to where you installed the code. All the batch files assume you want to connect to a single running instance of ExceptionMaker.exe. Also, all the batch files assume ADPlus.vbs is in the path.

CDB, because it's a native debugger, only knows about native structured exception handling (SEH) errors such as access violations (code 0xC00000005) and the like. Fortunately, under the covers, .NET exceptions use SEH when they occur. If it wasn't for that fact, you couldn't use ADPlus to run CDB to catch .NET exceptions. If you're curious, the native exception code for a .NET exception is 0xE0434F4D.

Because I'm talking about exceptions, I'm going to have ADPlus run in crash mode. That's not as scary as it sounds. I want the application to continue running so when an exception is thrown CDB will gain control to do your bidding. When running in crash mode, CDB attaches to the process until the process ends or you press Ctrl+C in the CDB window. If you do abort the execution with Ctrl+C, CDB will terminate the debuggee.

With CDB attached in crash mode, your application is basically running at full speed. That means it's perfect for running on your production systems. However, if you're not careful it can have a huge impact on performance. If you configure ADPlus to have CDB do a minidump on every first-chance exception on your large ASP.NET application, you could be writing out many minidumps, and each could take many minutes to write to disk. As with any debugging job, you'll want to configure ADPlus to do the minimum work necessary.

Minidumps on All .NET Exceptions

When I first looked at the (spartan) documentation for the <Exceptions> element of a configuration file, I thought it was going to be easy to get a minidump on all first-chance exceptions. I took the description of FullDumpOnFirstChance at face value and created the following configuration file:

<ADPlus>
  <Settings>
    <RunMode>CRASH</RunMode>
    <Option>Quiet</Option>
  </Settings>
  <Exceptions>
    <Option>FullDumpOnFirstChance</Option>
  </Exceptions>
</ADPlus>

While FullDumpOnFirstChance does create a minidump on each exception, it does not give the file a different name for each of the exceptions that occur. Thus, all .NET exceptions write to the same file name, so you only have the last of any exception type.

Fortunately, the <Config> element under the <Exceptions> element provides an alternative means of having all exceptions go to a unique minidump file. The configuration file looks like Figure 1.

Figure 1 Using AllExceptions (Close Figure 1)
<ADPlus>
  <Settings>
    <RunMode>CRASH</RunMode>
    <Option>Quiet</Option>
  </Settings>
  <Exceptions>
    <Config>
      <Code>AllExceptions</Code>
      <Actions1>FullDump</Actions1>
    </Config>
  </Exceptions>
</ADPlus>

The AllExceptions keyword tells ADPlus that you want the configuration to apply to all exceptions and events that CDB processes. The <Actions1> element contains the commands you want to run on the first-chance exception. In this case, I'm telling ADPlus to configure CDB to write a full memory minidump with a unique name. You can specify a few other options in the <Actions1> element that are discussed in the documentation. If you wanted to have an SOS command run when exceptions are thrown, you could specify them in the <CustomActions1> element. In most cases, you won't need to configure any second-chance exception processing because the default settings will write a full memory minidump automatically.

If you read the ADPlus configuration documentation, you'll note that the discussion of the <Config> element shows that you need to include a <ReturnAction1> element to tell ADPlus which of the Go or Quit commands you want to run when processing the exception. In the configuration file in Figure 1, I'm using the AllExceptions keyword and it turns out that ADPlus doesn't like to see the <ReturnAction1> element. The error reported, "Invalid return action1 in exception command: <cmd>," is wrong because the command itself is invalid. After reading through the ADPlus source code and running a few tests, I learned that the <ReturnAction1> element is only used if you're specifying a specific exception in the <Code> element.

The main drawback to using the AllExceptions approach is that in addition to getting minidumps to all native exceptions, you'll also get minidumps for module load, module unload, and process end. While the minidump that's generated when the process ends can be useful if you're looking for COM interop problems, having the debuggee hang every time a module loads into the address space is a large waste of disk space and time. However, if you're dealing with an application that has a lot of native code in it, you may want to consider using the configuration in Figure 1 in your test environment. No matter why an exception is thrown, you'll have a great record of exactly where it occurred.

In my original problem there was no native code other than what the .NET Framework brought into the address space. Setting up an ADPlus configuration that only created minidumps for .NET exceptions required tweaking the <Code> element and turning off all other minidump creation (see Figure 2).

Figure 2 .NET-Specific Dumps (Close Figure 2)
<ADPlus>
  <Settings>
    <RunMode>CRASH</RunMode>
    <Option>Quiet</Option>
  </Settings>
  <Exceptions>
    <Option>NoDumpOnFirstChance</Option>
    <Config>
      <Code>clr</Code>
      <Actions1>FullDump</Actions1>
      <ReturnAction1>gn</ReturnAction1>
    </Config>
  </Exceptions>
</ADPlus>

First I set the <Option> element to NoDumpOnFirstChange to turn off all minidump creation for all exceptions and events. Then in the <Code> element, where you're allowed to configure a single exception or a list of exceptions, I set all .NET exceptions to create a full memory minidump. If I had wanted to have a full memory minidump on an access violation, I would have specified clr;av in the <Code> element. One limitation of ADPlus is that you can only have one <Code> element so you can't have different processing for different exceptions.

Minidumps on a Specific .NET Exception

While getting the minidumps from all .NET exceptions is nice, if you have an application that had a few too many gratuitous .NET exceptions, the blocking that occurs every time you write a dump would be a big problem. In the bug I was working on that precipitated all this poking, I needed to know the instance on which an OutOfMemoryException was being thrown.

If you've looked at the .NET Framework 2.0 version of SOS, you've seen the !stoponexception command (abbreviated !soe) that will break on the .NET exception when it's thrown during live debugging. When you issue the following command:

!soe -create System.ArgumentNullException 3
The help says that it will break when the ArgumentNullException is thrown. The "3" is the debugger pseudo-register in which the command will store a one if the exception is an ArgumentNullException. If the exception is anything else, it will store a zero in the third pseudo-register. By the way, the help for !stoponexception has two mistakes in it. The first is that it says it will automatically use pseudo-register number one if you don't specify the register. However, if you leave the pseudo register off, the command won't work. Secondly, the help text showing the use of the sxe command is incorrect. The command shown needs to have "clr" on the end.

When I first used the !stoponexception command, I was wondering how it worked. Since the only way the native debugger will know that a .NET exception occurred is if the SEH exception number 0xE0434F4D shows up, I thought that the !stoponexception command was doing a bit more work. Running the debugger sx command to show the exception settings and commands associated returns the following for .NET exceptions.

clr - CLR exception - break - not handled
       Command: "!soe  System.ArgumentNullException 3;
                 .if(@$t3==0) {g}
                 .else {.echo 'System.ArgumentNullException hit'}"

Note that the CDB/WinDBG code for .NET exceptions is clr. (You'll notice that I wrapped the line beginning with "Command:" to fit in the column.)

When you use the !stoponexception command it sets the command to run on .NET exceptions to itself. When a .NET exception is thrown, the special execution of !stoponexception looks to see if the exception is the specific type. If the type matches, the command will put 1 in the pseudo-register. The interesting part is the debugger command that does the conditional execution.

The configuration file in Figure 3 shows how you can get the minidump on a specific exception. Note that I spaced out and wrapped the lines in the <CustomActions1> element so they would fit on the page. In the real configuration files, you can do the same, but make sure to end all commands with a semicolon like I have.

Figure 3 Get Minidump on a Specific Exception (Close Figure 3)
<ADPlus>
  <Settings>
    <RunMode>CRASH</RunMode>
    <Option>Quiet</Option>
  </Settings>
  <Exceptions>
    <Option>NoDumpOnFirstChance</Option>
    <Config>
      <Code>clr</Code>
      <Actions1>Void</Actions1>
      <CustomActions1>
       .loadby sos mscorwks;
       !stoponexception System.ArgumentNullException 3;
       .if(@$t3==1)
       {
        .dump /ma /u c:\\dumps\\ANE.dmp
       }
    </CustomActions1>
    <ReturnAction1>gn</ReturnAction1>
    </Config>
  </Exceptions>
</ADPlus>

The difference between this configuration and the previous one that wrote a minidump on all .NET exceptions is that I'm taking advantage of the <CustomActions1> element, which specifies the custom commands you want to run on first-chance exceptions. I'm also setting the <Actions1> element to Void so that none of the default commands run and I'm only getting the dump on the single .NET exception I want to track.

When a .NET exception occurs, the first command that runs loads the SOS extension from the directory where MSCORWKS.DLL was loaded. You could also put the .loadby command in the configuration file <PreCommands> section, which contains commands that are executed as soon as the debugger attaches or loads the process. The second command is where I ask the !stoponexception command to check the exception type, and if it matches, set the pseudo-register specified to one. The .if checks the value, and if it's one will cause the .dump command to execute.

Unfortunately, there's no way in the configuration file to specify that the dump should be written to the directory where all other output files are written. But at least you can get the date and time on the minidump by specifying the /u option. It would be nice if you could insert a special code in the appropriate elements to indicate the output directory.

One important point about using the debugger command programs is that there's a small bug in their parsing. Notice that I used double slashes in the path portion of the file name passed to the .dump command. If you forget the double slashes all your dumps will be written to the root directory, if the debugger user account has write access.

On some bugs you may need to create minidumps on multiple dumps. Fortunately, it's quite easy to do because the debugger command program extension supports else operations. The configuration file in Figure 4 writes different dumps on different .NET exceptions.

Figure 4 Different Dumps for Each .NET Exception (Close Figure 4)
<ADPlus>
  <Settings>
    <RunMode>CRASH</RunMode>
    <Option>Quiet</Option>
  </Settings>
  <Exceptions>
    <Option>NoDumpOnFirstChance</Option>
    <Config>
      <Code>clr</Code>
      <Actions1>Void</Actions1>
      <CustomActions1>
    .loadby sos mscorwks;
    !stoponexception System.ArgumentNullException 3;
    !stoponexception System.IO.FileNotFoundException 4;
    .if(@$t3==1)
    {
        .dump /ma /u c:\\Dumps\\ANE.dmp
    }
    .elsif(@$t4==1)
    {
        .dump /ma /u c:\\Dumps\FNFE.dmp
    }
    </CustomActions1>
      <ReturnAction1>gn</ReturnAction1>
    </Config>
  </Exceptions>
</ADPlus>

If you need to handle multiple conditionals, be careful about a second parsing problem for debugger command programs—if you're going to use the .elsif statement, make sure you spell the statement without the "e." If you use the incorrect ".elseif" the debugger will terminate as soon as it evaluates the invalid statement.

What About Vectored Exception Handling?

Once I had ADPlus doing all the heavy lifting to snap minidumps, I had to wonder if there was a way I could integrate similar techniques into my applications without relying on external tools. Having done Windows development for a little too long, I remembered that in native code you could get all exceptions reported in your application through vectored exception handling.

Back in the September 2001 MSDN®Magazine, my good friend Matt Pietrek wrote an excellent introduction to vectored exception handling. What makes it so interesting is that from native code, you can jam in a callback on all exceptions so you are notified before any exception chain unwinding occurs. Under the covers, .NET exception handling uses structured exception handling. My plan was to see if there was a way to write some sneaky .NET code that would allow me to fake out .NET, get notification for all exceptions, and do it all without disturbing the .NET infrastructure.

Whenever you even consider doing anything a little out of the ordinary, spending some quality time searching the Web is well worth the effort. A bit of searching turned up Mike Stall's excellent blog entry on vectored exception handling and .NET. As Mike is a developer on the Common Language Runtime (CLR) Debugging Team, I pay attention to anything he says, and he's quite clear that using vectored exception handling from .NET is definitely not a good idea. However, if you read his blog entry closely, you'll see he's mainly talking about attempting to use .NET code to implement your vectored exception handling.

Mike doesn't say anything specifically about implementing your vectored exception handling from pure native code, so that got me thinking. I wanted to see if I could do all the vectored exception handling through native C++ code so that I could snap dumps of specific exceptions without having the Debugging Tools for Windows installed on the machine. If I could do this, I could offer a great diagnostic tool. If your end users were reporting even intermittently reproducible problems in your application, you could have them toggle a setting, and on each exception of a specific type, you could write an SOS-compatible minidump whenever that exception bubbled through your system. If you've been reading the Bugslayer column for a while, you know that this is the type of functionality that interests me quite a bit.

Keeping Mike Stall's blog in mind, my plan was to come up with a solution that would avoid messing up .NET but allow you to get the minidumps when you needed them. Mike says very explicitly that you can't call any .NET code from the vectored exception handler and that you only want to consider touching .NET exceptions. That seemed fair enough, so I started on my quest to build the library.

Obviously, my first task was to write the native C++ DLL that I could call from managed code to deal with all the vectored exception handling needs. There wasn't anything exciting about the core code and I was able to get everything hooked up so no matter how I created an exception, or what exception I tried, my native vectored exception handler was called.

My next step was to try to figure out the type of the .NET exception being thrown. If you look at the native prototype for the vectored exception handling code, your native callback function receives the PEXCEPTION_POINTERS, which completely describes the exception at the instant it's thrown. Since all .NET exceptions are 0xE0434F4D, it's easy to see a .NET exception, but I wanted to zoom in a little closer on the particular exception.

After a good amount of experimentation, I noticed that the pExceptionInfo->ExceptionRecord->NumberParameters value was always one on any type of .NET exception. The NumberParameters field shows how many items in the pExceptionInfo->ExceptionRecord->ExceptionInformation array describe additional data about the exception. Where an access violation would put the address you were trying to read from or write to in the ExceptionInformation array, a .NET Framework exception has a different value.

My original hope was that the extra information about a .NET exception would be the metadata ID of the actual exception type. You can see any metadata ID values by running ILDASM with the /OUT and /TOKENS switches so you can see all the metadata information for your assemblies. After dumping numerous .NET runtime and custom assemblies, I didn't see any of the values in the ExceptionInformation array that matched metadata IDs.

I started poking around the CLR headers in the <VS Install Directory>\SDK\v2.0\include, which contains the complete native declarations for everything .NET. Looking at the specific types I was throwing, I eventually stumbled on CORERROR.H, which contains all the HRESULT values for errors that occur in the native .NET interfaces. A little study of the values I was seeing in the ExceptionInformation array lead me to the "Classlib errors" section, which contains the HRESULT values of key exceptions in .NET. While extremely important values like 0x80020012 indicate a divide by zero, all programmatic exceptions are lumped under the same HRESULT so I wasn't going to be able to easily differentiate a RankException from TimeoutException.

Since Mike's blog is very specific about the fact that vectored exception handling can completely mess up managed code, I was worried that even calling into the native interfaces could cause issues. Additionally, with the native interfaces, there is no clean way to jump into .NET from a native application to grab the particular exception. I found the best I could do was look for specific values in the ExceptionInformation array. The CORERROR.H file lists the particular HRESULT codes for all the super-critical .NET exceptions, such as OutOfMemoryExceptions.

While not perfect, at least you could specify the super-critical exceptions you wanted to see, which looked promising enough. I turned my attention to figuring out how I was going to write the minidump inside my vectored exception handler native function. The documentation for MiniDumpWriteDump is quite clear that calling the function just by itself won't work when called inside the process.

After hours of poking around, I thought the best way to get the minidump was to do it from another process. That meant I was going to spawn off that process from the native vectored exception handling function. It turned out that the minidump writing process was fairly trivial to build in .NET.

With all the pieces in place, I was finally able to start doing some serious testing to see if my scheme was going to work. I tried every type of .NET application from Windows Forms, to console applications, to a slew of ASP.NET examples. Interestingly I found that my big scheme would generally work, sort of. In many runs, everything worked as expected, but some runs caused an endless loop of 0x80010108 errors when I was debugging with both managed and native debugging turned on in the process. No matter what I tried on different machines, I couldn't get everything working consistently. While I hate to admit failure, this is one case where I'm quite positive I've gotten to the bottom of the well.

Wrap Up

If you've read the Bugslayer column over the years, you know I won't hesitate to do some serious hacking to get a trick working to provide a cool tool. However, in the case of vectored exception handling, I had to throw in the towel and declare the path a dead one. Mike Stall wasn't kidding when he said vectored exception handling wasn't a great idea in .NET. As similar things had been said about other sneaky tricks in the past, I don't consider my time wasted, because I hope to save you the time of going down the same path I did.

Even without the vectored exception handling tricks, we still have the outstanding ADPlus and configuration files to get those dumps right when we need them. While it does require the Debugging Tools for Windows on the production machine, at least it's an XCOPY installation. I hope I was able to show you the power of the tools and how to think a bit outside the box on using them to get exactly the minidump you need to solve the hardest of problems.

Tips

Tip 75 Are you still compiling a bunch of .NET Framework 1.1 code that you can't move over to the .NET Framework 2.0 just yet? You probably feel like a kid with your face pressed up against the pet store window wishing you could use that cool MSBUILD system. The great news is that Microsoft has seen your smudges and has released MSBee, an extension to MSBUILD to allow compiling of .NET Framework 1.1 applications. Grab the code and the documentation.

Tip 76 Are you hot for Visual Studio 2005 Debugger Visualizers like I am? If so, make sure to check out a great list of tips by Frans Bourma on tricks learned the hard way. Great stuff!

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in the .NET and Windows platforms. His latest book is Debugging Microsoft .NET 2.0 Applications (Microsoft Press, 2006). You can contact John at www.wintellect.com.


Security Briefs: Limited User Problems and Split Knowledge

Security Briefs
Limited User Problems and Split Knowledge
Keith Brown

Code download available at: SecurityBriefs2006_11.exe (165KB)

This month I'll answer more reader questions. I'll discuss Windows Communication Foundation Web services running under normal user accounts, and the use of split knowledge and dual control of keys for protecting credit card data.

Q Why won't my simple Windows® Communication Foundation service start when I run it as a non-administrator?

A First, let me say that it's good to hear that you're testing your code under a normal user account! This is an important aspect of testing that developers should not overlook.

Now, the first approach I recommend for people trying to track down this type of problem is to fire up Filemon and Regmon, available from Sysinternals (which was recently acquired by Microsoft) at www.sysinternals.com, and look for failed attempts to open files or registry keys. Unfortunately, these tools can't diagnose every issue that could cause this problem. Even the simplest of Web services implemented in Windows Communication Foundation that listen on an HTTP channel will not run as a normal user by default, unless it's hosted in IIS. This is because the Windows Communication Foundation HTTP channel uses the HTTP.SYS driver to set up its listener, and HTTP.SYS does not allow non-administrators to register listeners without an administrator explicitly granting permission.

To demonstrate the problem, I've built a simple Web service that consists of two files: the source for the service and an application configuration file. Figure 1 shows the code for the service. When you launch this service while running as an administrator, it works just fine. But try running it as a normal user, and bang! Figure 2 shows the exception that occurs when using Beta 2 of the Microsoft® .NET Framework 3.0.

Figure 2 Exception When Run as Normal User (Close Figure 2)
System.ServiceModel.Diagnostics.CallbackException:
A user callback threw an exception.
Check the exception stack and inner exception to determine the callback that failed.
---> System.NullReferenceException: Object reference not set to
     an instance of an object.
   at System.ServiceModel.Channels.DatagramChannelDemuxer`2.
      OnListenerClosed(Object source, EventArgs args)
   at System.ServiceModel.Channels.CommunicationObject.OnClosed()
   --- End of inner exception stack trace ---
   at System.ServiceModel.Channels.CommunicationObject.OnClosed()
   at System.ServiceModel.Channels.CommunicationObject.Abort()
   at System.ServiceModel.Channels.ChannelListenerBase.OnAbort()
   at System.ServiceModel.Channels.SecurityChannelListener`1.OnAbort()
   at System.ServiceModel.Channels.CommunicationObject.Abort()
   at System.ServiceModel.Channels.ChannelListenerBase.OnAbort()
   at System.ServiceModel.Channels.CommunicationObject.Abort()
   at System.ServiceModel.Dispatcher.ChannelDispatcher.OnAbort()
   at System.ServiceModel.Channels.CommunicationObject.Abort()
   at System.ServiceModel.ServiceHostBase.OnAbort()
   at System.ServiceModel.Channels.CommunicationObject.Abort()
   at System.ServiceModel.Channels.CommunicationObject.Close(
      TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close()
   at System.ServiceModel.Channels.CommunicationObject.Dispose()
   at Program.Main(String[] args) in
      C:\work\HelloService\Program.cs:line 24
Figure 1 Simple Web Service Source Code (Close Figure 1)
//hello.cs - simple WCF Web service that listens on HTTP
using System;
using System.ServiceModel;

[ServiceContract(Namespace = "http://example.org")]
public interface IHello {
    [OperationContract]
    void SayHello();
}

public class Hello : IHello {
    public void SayHello() {
        Console.WriteLine("Hello");
    }
}

class ConsoleHost {
    static void Main(string[] args) {
        try {
            using (ServiceHost host = new ServiceHost(typeof(Hello),
                    new Uri("http://localhost:8080/MyServices/"))) {
                host.Open();

                Console.WriteLine(
                    "Service is listening, press any key to quit.");
                Console.ReadKey();
            }
        }
        catch (Exception x) {
            Console.WriteLine(x);
        }
    }
}

<!--app.config-->
<configuration>
  <system.serviceModel>
    <services>
      <service name="Hello">
        <endpoint address="Hello"
                  binding="wsHttpBinding"
                  contract="IHello"/>
      </service>
    </services>
  </system.serviceModel>
</configuration>

As I mentioned, one way to solve this problem is to host your service in IIS. But for some people, this isn't an option. If, for example, you're using Windows Communication Foundation in a Windows Forms-based application to receive notifications via an HTTP Web service, you'll need to deal with this problem in a different way, following the guidance in this column. If you're hosting in a Windows NT® service, you've got the same issue.

The problem is not difficult to solve—you just need an administrator to grant your application what is called a namespace reservation with HTTP.SYS. In doing so, the administrator is essentially saying, "It's OK for this user to listen over HTTP on a URL prefix that I specify." An administrator can grant these listening permissions to either an individual user or to a group.

Your setup program can grant a namespace reservation programmatically using the HTTP API, which unfortunately doesn't yet have a public .NET wrapper. Figure 3 shows some sample code that you can use to get started if you're working in C#. This particular sample expects you to pass a user or group account by name, and it must be executed by a member of the local Administrators group. This code also uses P/Invoke to call the HTTP API, which means it needs to run fully trusted as far as evidence-based security in the common language runtime (CLR) is concerned. If you're packaging this inside a Microsoft Installer, it should be fine. But if you're trying to do this from a downloaded application that runs with partial trust under a normal user account, it will not work.

Figure 3 Programatically Granting a Namespace Reservation (Close Figure 3)
using System;
using System.Xml;
using System.Security.Principal;
using System.Runtime.InteropServices;

public class ReserveHttpNamespace {
    static void Main(string[] args) {
        if (args.Length != 2) {
            Console.WriteLine(
                "Usage: reserveHttpNamespace " +
                "prefix account");
            return;
        }
        try {
            ModifyReservation(args[0], args[1], false);
            Console.WriteLine("Success!");
        }
        catch (Exception x) {
            Console.WriteLine(x.Message);
        }
    }

    static void ModifyReservation(
            string urlPrefix, string accountName,
            bool removeReservation) {
        string sddl = createSddl(accountName);
        HTTP_SERVICE_CONFIG_URLACL_SET configInfo;
        configInfo.Key.UrlPrefix = urlPrefix;
        configInfo.Param.Sddl = sddl;
        HTTPAPI_VERSION httpApiVersion =
            new HTTPAPI_VERSION(1, 0);
        int errorCode = HttpInitialize(httpApiVersion,
            HTTP_INITIALIZE_CONFIG, IntPtr.Zero);
        if (0 != errorCode)
            throw getException("HttpInitialize", errorCode);
        try {
            // do our best to delete any existing ACL
            errorCode = HttpDeleteServiceConfigurationAcl(
                IntPtr.Zero, HttpServiceConfigUrlAclInfo,
                ref configInfo, Marshal.SizeOf(
                    typeof(HTTP_SERVICE_CONFIG_URLACL_SET)),
                IntPtr.Zero);
            if (removeReservation) {
                if (0 != errorCode) throw getException(
                    "HttpDeleteServiceConfigurationAcl",
                    errorCode);
                return;
            }
            errorCode = HttpSetServiceConfigurationAcl(
                IntPtr.Zero, HttpServiceConfigUrlAclInfo,
                ref configInfo, Marshal.SizeOf(
                    typeof(HTTP_SERVICE_CONFIG_URLACL_SET)),
                IntPtr.Zero);
            if (0 != errorCode) throw getException(
                "HttpSetServiceConfigurationAcl", errorCode);
        }
        finally {
            errorCode = HttpTerminate(
                HTTP_INITIALIZE_CONFIG, IntPtr.Zero);
            if (0 != errorCode) throw getException(
                "HttpTerminate",errorCode);
        }
    }

    static Exception getException(string fcn, int errorCode) {
        return new Exception(
            string.Format("{0} failed: {1}",
            fcn, getWin32ErrorMessage(errorCode)));
    }

    static string createSddl(string account) {
        string sid = new NTAccount(account).Translate(
            typeof(SecurityIdentifier)).ToString();
        // DACL that allows generic execute for the user
        // specified by account.
        // See help for HTTP_SERVICE_CONFIG_URLACL_PARAM
        // for details on what this means.
        return string.Format("D:(A;;GX;;;{0})", sid);
    }

    static string getWin32ErrorMessage(int errorCode) {
        return Marshal.GetExceptionForHR(HRESULT_FROM_WIN32(errorCode));
    }
    static int HRESULT_FROM_WIN32(int errorCode) {
        if (errorCode <= 0) return errorCode;
        return (int)((0x0000FFFFU & ((uint)errorCode)) | (7U << 16) |
            0x80000000U);
    }

    // P/Invoke stubs from http.h
    const int HttpServiceConfigUrlAclInfo = 2;
    const int HTTP_INITIALIZE_CONFIG = 2;
    [StructLayout(LayoutKind.Sequential)]
    struct HTTPAPI_VERSION {
        public HTTPAPI_VERSION(short maj, short min) {
            Major = maj; Minor = min;
        }
        short Major;
        short Minor;
    }
    [StructLayout(LayoutKind.Sequential)]
    struct HTTP_SERVICE_CONFIG_URLACL_KEY {
        [MarshalAs(UnmanagedType.LPWStr)]
        public string UrlPrefix;
    }
    [StructLayout(LayoutKind.Sequential)]
    struct HTTP_SERVICE_CONFIG_URLACL_PARAM {
        [MarshalAs(UnmanagedType.LPWStr)]
        public string Sddl;
    }
    [StructLayout(LayoutKind.Sequential)]
    struct HTTP_SERVICE_CONFIG_URLACL_SET {
        public HTTP_SERVICE_CONFIG_URLACL_KEY Key;
        public HTTP_SERVICE_CONFIG_URLACL_PARAM Param;
    }
    [DllImport("httpapi.dll", ExactSpelling = true,
            EntryPoint = "HttpSetServiceConfiguration")]
    static extern int HttpSetServiceConfigurationAcl(
        IntPtr mustBeZero, int configID,
        [In] ref HTTP_SERVICE_CONFIG_URLACL_SET configInfo,
        int configInfoLength, IntPtr mustBeZero2);
    [DllImport("httpapi.dll", ExactSpelling = true,
            EntryPoint = "HttpDeleteServiceConfiguration")]
    static extern int HttpDeleteServiceConfigurationAcl(
        IntPtr mustBeZero, int configID,
        [In] ref HTTP_SERVICE_CONFIG_URLACL_SET configInfo,
        int configInfoLength, IntPtr mustBeZero2);
    [DllImport("httpapi.dll")]
    static extern int HttpInitialize(
        HTTPAPI_VERSION version,
        int flags, IntPtr mustBeZero);
    [DllImport("httpapi.dll")]
    static extern int HttpTerminate(int flags,
        IntPtr mustBeZero);
}

You can also use a tool called HTTPCFG.EXE, which is part of the support tools found in the SUPPORT subdirectory of your operating system installation disk. You can use this tool to list existing namespace reservations, like so:

httpcfg query urlacl

Here's how to create a namespace reservation for a user account using HTTPCFG:

httpcfg set urlacl –u http://+:8080/MyServices/
-a D:(A;;GX;;;S-1-5-21-1681502023-2202157333-1552196959-1028)
The -a argument to HTTPCFG stands for ACL, or Access Control List, which you must specify in a rather arcane language called Security Description Definition Language (SDDL). There isn't enough space here to explain SDDL, so I'll leave that as an exercise you can do on your own. (For more information on SDDL, check out "Security Descriptor Definition Language".) The GX you see refers to GENERIC_EXECUTE permission. HTTP.SYS expects you to grant this permission if your goal is to grant permission to listen on the prefix.

The -u argument is the URL prefix that tells HTTP.SYS the shape of the URLs that you're referring to when you grant permission. The form of the URL is as follows. The scheme must be http or https, in lowercase. The host is case-insensitive and may use either the + or * wildcards (I'll talk more about this shortly). The port is an integer value and is required, even if you're talking about the default port for the scheme. Following this is an optional case-insensitive relative URI (in the previous example, this is /MyServices). And finally, regardless of whether you supply a relative URI, you need to terminate the string with a trailing slash.

Under the covers, the Windows Communication Foundation HTTP channel registers namespaces like the URL shown earlier. It does this using the HTTP API, which has a few rules about routing requests to HTTP listeners. The way you specify the host in the URL determines the priority in which your listener will be considered when a request comes in that matches more than one listener's prefix. For example, one app might register foo.com:8080/ while another app might register foo.com:8080/MyServices/. Generally the more explicit registration wins. However, there are also wildcards that can be used to control how this prioritization works. An application might register http://*:8080/. Since this listener uses the * wildcard, it picks up the dregs of whatever the other listeners don't want. In other words, * is a low priority or weak wildcard. On the other hand, if the application registered http://+:8080/, it's going to be given top priority and any HTTP request on port 8080 will go to this application without checking for other, more specific registrations. You can read more about how these URL prefixes are formed at "UrlPrefix Strings", but suffice it to say that if you want to successfully reserve a namespace prefix for the Windows Communication Foundation HTTP channel, you'll have to use a form that is general enough to cover the actual prefix that Windows Communication Foundation is going to register.

If you refer back to Figure 1, you can see that the base address I registered with ServiceHost is localhost:8080/MyServices/. Unfortunately, if you simply do the obvious thing and try to register this URL prefix explicitly with HTTPCFG, it will not work. This is because under the covers, Windows Communication Foundation appears to be registering a strong wildcard, which casts a wider net and won't be covered by an explicit host name registration. Instead, you need to use the strong wildcard syntax and register a URL prefix of http://+:8080/MyServices/. This is true regardless of whether you do this programmatically or via the HTTPCFG tool.

You might be asking yourself, "Why do I need to go through all of this hassle? Why can't any old user just register an HTTP listener like they can open a socket?" I believe the answer has to do with the way HTTP.SYS does port sharing. With a socket, once you bind to a port and start listening, that port is in use and no other application can listen on it. But with HTTP.SYS, two applications can listen on the same port because it's actually HTTP.SYS that opens the port and routes each request based on the URL prefix. Say your company allows its users to run a legitimate application that happens to listen on port 8080 by registering an HTTP listener, and the administrator has opened his firewall to allow HTTP requests through to that port. That port is a potential target for malware. Now imagine that one of the users accidentally opened an executable attachment from a bad guy, and he ended up with malware on his machine that listens for instructions from its creator. This malware might hide on port 8080 by registering a listener through the HTTP API, sharing the port with the legitimate application! But since HTTP.SYS requires you to have permission from the administrator before installing a listener, as long as you're running as a normal user, the malware won't be able to register its listener due to your security context.

There are a lot of people out there who write code while running as an administrator. Sadly they aren't going to notice this problem until they deploy their code in a non-privileged environment, which is one reason you should always test your code as a non-admin. Better yet, you can try writing code while running as a non-admin! I won't jump on that soapbox here, but if you're interested in learning more about developing as a non-admin, check out Aaron Margosis's blog, where he's dedicated many posts to the topic.

Q What is the best way to implement split knowledge and dual control of keys?

A Many Web sites accept credit card holder data, including credit card numbers, billing addresses, and so on. In late 2004, a standard called the Payment Card Industry Data Security Standards (PCI-DSS) was published to specify security requirements for companies that store, process, or transmit this type of data. One of the newer requirements is Section 3.6.6, which requires, "Split knowledge and dual control of keys (so that it requires two or three people, each knowing only their part of the key, to reconstruct the whole key)."

In an online transaction system, it's pretty hard to imagine having two or three people standing around typing in a secret key each time it's needed to process a credit card request. But if you have long-term storage of sensitive data like this, the concept starts to seem a lot more reasonable. So how would you go about building such a system if you wanted to satisfy this requirement?

The simplest way to split a message up among N different parties is to generate N-1 random secrets (each must be exactly as long as the message to be protected) and XOR each byte of those random secrets into each corresponding byte of the message. The blob of data you end up with after all of these operations is treated as the Nth secret. You can then give out each of these secrets to different people and destroy the original message. If one person looks at his own secret, it's going to look like random garbage (after all, it is just random data). The only way to reconstruct the original message is to bring all of the people back together, gather their secrets, and XOR them together. Because of the commutative property of XOR, the order in which this happens doesn't matter. As long as every one of the split secrets is mixed back in, you'll ultimately get back the original message.

I'm using the term "message" to refer to the set of bytes that needs to be protected. This could be a string, like "Attack at dawn!" or binary data, such as a cryptographic key used to protect persistent cardholder data.

The security of this technique is perfect in theory. (I say in theory because implementations are usually less than perfect!) If one person refuses to give up her secret, the other people cannot put their secrets together to get any closer to the original message than they were on their own. But this also leads to the drawback: if anyone loses his secret, the original message can never be recovered. If this is a concern, there are other, considerably more complicated approaches that involve solving equations in a finite field (see Bruce Schneier's book Applied Cryptography, Second Edition). I'll leave implementing these more complicated schemes for a future column. For now, let me present a simple solution to the secret-splitting problem using System.Security.Cryptography.

My proof-of-concept is a layered solution that starts with two core functions in a class called SecretSplitter. These functions split or join a message using the technique I described earlier. The code for these functions is shown in Figure 4. The first function, SplitSecret, takes a byte array that represents the message to be split and an integer that indicates how many secrets you want it to be split into. JoinSecret takes an array of split secrets and joins them together to form the original message. You'll get something different if you don't provide the exact keys that were originally split.

Figure 4 Splitting and Joining a Message (Close Figure 4)
public static List<byte[]> SplitSecret(byte[] secret, int count) {
    if (null == secret || 0 == secret.Length)
        throw new ArgumentException("Non-empty value required",
        "secret");
    if (count < 2)
        throw new ArgumentException("Must be greater than one", "count");

    RNGCryptoServiceProvider random = new RNGCryptoServiceProvider();

    // Get N-1 new secrets.
    List<byte[]> newSecrets = new List<byte[]>(count);
    for (int i = 0; i < count - 1; ++i) {
        byte[] newSecret = new byte[secret.Length];
        random.GetBytes(newSecret);
        newSecrets.Add(newSecret);
    }

    // XOR all secrets into the existing one to get the final secret.
    byte[] finalSecret = (byte[])secret.Clone();
    foreach (byte[] newSecret in newSecrets) {
        for (int i = 0; i < finalSecret.Length; ++i) {
            finalSecret[i] ^= newSecret[i];
        }
    }
    newSecrets.Add(finalSecret);

    return newSecrets;
}

public static byte[] JoinSecret(List<byte[]> splitSecrets) {
    if (null == splitSecrets) throw new ArgumentNullException();

    byte[] secret = null;
    foreach (byte[] splitSecret in splitSecrets) {
        if (null == splitSecret) throw new ArgumentNullException();
        if (null == secret) {
            secret = (byte[])splitSecret.Clone();
            continue;
        }
        if (splitSecret.Length != secret.Length)
            throw new ArgumentException(
                "All secrets must be of the same length");

        // XOR all the split secrets together to get the original secret.
        for (int i = 0; i < secret.Length; ++i) {
            secret[i] ^= splitSecret[i];
        }
    }
    return secret;
}

Note the use of the RNGCryptoServiceProvider class to generate the secret keys. This is a much better technique than using something like System.Random. RNGCryptoServiceProvider was specifically designed to generate random data for cryptographic operations and thus isn't nearly as predictable as the stream of data you'll get from something like System.Random. It also seeds itself from many different sources of entropy on your computer.

Layered on top of these two functions are helpers that allow you to work purely with strings by Base64 encoding the data. This code is less relevant, so I won't show it here, but it's useful because we often deal with string-based messages and text files. All of the functionality is packaged in a library assembly, which I called SecretSplittingLibrary.dll.

On top of this library, I built a console application to demonstrate how key splitting might work in a specific application. This application is called ThumbDriveSecretSplitter and has two commands: one to split a message and one to join a message.

To split a message, you can run the following command:

ThumbDriveSecretSplitter split "Attack at dawn!"
This should split the message "Attack at dawn!" onto all the thumb drives that are currently plugged into the machine. Well, that was my original goal at least. But when I first tried this out, I discovered a problem: the app had some difficulty determining which volumes represented removable media. I have a thumb drive that shows up as a fixed drive in Disk Manager and I suspect that it's not the only one with this issue.

So I cheated and gave the program a little hint. In the application's configuration file, I included a list of volume names that the app should look for to determine which drives should be used:

<appSettings>
  <add key="volumeNames"
       value="FatDrive;PSThumbDrv"/>
</appSettings>
Now when I run the command, the program looks for drives that are included in the configured list and splits the message among those drives by dropping a Base64-encoded split secret into a file called "mysecret.txt" in the root of each drive. This means I can now split the message among all the thumb drives, and then hand out the drives to the individuals responsible for guarding the secret.

When you want to restore the original message, you just need to get those folks to plug their thumb drives into your machine, and then you run the other command:

ThumbDriveSecretSplitter join
This combines the secrets and prints out the protected message. Here's the output of a session using ThumbDriveSecretSplitter:
C:>ThumbDriveSecretSplitter.exe split "Attack at dawn!"
Writing E:\mysecret.txt
Writing F:\mysecret.txt

C:>ThumbDriveSecretSplitter.exe join
Reading E:\mysecret.txt
Reading F:\mysecret.txt

Joining secrets...

Attack at dawn!

When using this technique in a real system, you probably don't want to display the secret for all parties to see—that would defeat the purpose of splitting it up in the first place. Instead, you'd have the program collect the keys, compute the secret, and operate on it (most likely using it as a secret key to decrypt a sensitive backup of cardholder data).

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is a cofounder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight's Applied .NET Security course, as well as several books, including the .NET Developer's Guide to Windows Security, available in print and on the Web. Learn more at pluralsight.com/keith.


Concurrent Affairs: The ReaderWriterGate Lock

Concurrent Affairs
The ReaderWriterGate Lock
Jeffrey Richter

Contents
The ReaderWriterGate Object Model
Using the ReaderWriterGate
How ReaderWriterGate is Implemented
Why the ReaderWriterGate is Cool
Knowing When the Callback Completes
Conclusion

One day I was working on a consulting project and came across a thread synchronization problem that I hadn't run into before. The company was creating a Web service and almost all the client requests that came into their server needed to access some shared data in a read-only fashion. Occasionally, a request that needed to modify the shared data would come in, and we needed to keep the data in sync. It sounded like the perfect scenario for using reader-writer locks (as I discussed in my June 2006 Concurrent Affairs column).

However, after I got more involved with the project, I realized that this company's requirements had an unusual twist: when a Web service request came into the server and the lock was taken for writing, the lock was held for an extremely long time. As the writer thread was modifying the shared data, other Web service requests to read the data were coming into the server. The problem here was that threads in the pool were waking up to process the read requests, but all of these threads were immediately put to sleep waiting for the writer thread to complete its task.

It didn't take long before hundreds of pooled threads were created and all of these threads were in a wait state inside the operating system kernel. We now had lots of threads that couldn't run just sitting around wasting resources (such as each thread's stack and the associated thread kernel object). This was bad. And then, it got worse! When the writer thread completed writing to the shared data, it released the lock. The lock then saw that hundreds of reader threads were waiting for it and it released all of the waiting reader threads. The server computer had four CPUs in it and these four little CPUs had to process the work of all these runnable threads. Windows® would schedule all these threads using its normal round-robin scheduling technique, but the overhead of constantly switching the CPUs from one thread to another to another was simply killing performance, scalability, and throughput.

For months, this problem gnawed at me until I finally devised my own thread synchronization lock, which I call the ReaderWriterGate. This lock is fast, and it fits really well in a wide variety of applications. The interesting thing about the ReaderWriterGate is that calling threads almost never block, and if they do block, they are guaranteed to block for only an incredibly short period of time. And, because internally the ReaderWriterGate uses one of my ResourceLock-derived classes (discussed in the June column I mentioned earlier), I could change just one line of code to have ReaderWriterGate use a spin lock internally so that threads never block inside the Windows kernel (as I described in my October 2005 Concurrent Affairs column). This would give even better performance but, for fairness, this would require that all threads using the ReaderWriterGate be of the same thread priority and that priority boosting be disabled. This would improve performance even more. Furthermore, the ReaderWriterGate can perform an extraordinary amount of work using just a few threads. So, very few resources are required regardless of the number of requests being made against the gate.

I have my own implementation of this primitive which I make available free of charge as part of the Power Threading library, available for download from www.wintellect.com. If you want to use my implementation, you can, pursuant to the End-User License Agreement (EULA) to that accompanies the library. See the EULA for more details.

What I'd like to do now is explain the idea behind the ReaderWriterGate, discuss a possible implementation, and offer a slightly new way of thinking about threading and thread synchronization in general. It is my hope that you will see places in your existing code where this kind of thinking can be applied so that with minimal re-architecting, you could incorporate some of these ideas to give your applications better performance and scalability.

The ReaderWriterGate Object Model

To use the ReaderWriterGate class, there are only a few methods you need to be aware of:

public sealed class ReaderWriterGate
{
   public ReaderWriterGate();

   public void QueueWrite(
      ReaderWriterGateCallback callback, Object state);
   public void QueueRead(
      ReaderWriterGateCallback callback, Object state);

   ... // Some methods not shown to simplify the discussion
}
As you can see, this is an incredibly simple object model (simpler than most locks). When you want to execute a task that requires read-only/shared access to a resource, you simply call the QueueRead method and when you want to execute a task that requires write/exclusive access to a resource, you simply call the QueueWrite method. Both of these methods take two arguments. The first argument, an instance of ReaderWriterGateCallback, identifies a method that you write; this method does the actual reading or writing. ReaderWriterGateCallback is a delegate type defined as follows:
delegate void ReaderWriterGateCallback(
    ReaderWriterGateReleaser releaser);

The second argument passed to QueueRead and QueueWrite is a reference to any object. This reference will be passed to your callback method that does the actual reading or writing. This simply gives you a way to get some data from the method that wants the task done and provide it to the thread or method that performs the actual task. For those of you familiar with the .NET ThreadPool and its QueueUserWorkItem method, this model should feel a bit familiar.

Notice that the ReaderWriterGateCallback delegate requires that your callback method take just one parameter that is of the ReaderWriterGateReleaser type. The following shows you what this type looks like:

public sealed class ReaderWriterGateReleaser : IDisposable
{
   // Returns second argument passed to QueueXxx
   public object State { get; }

   // Returns reference to Gate
   public ReaderWriterGate Gate { get; }

   public void Release();    // Calls Dispose internally
   public void Dispose();    // Useful with C#'s using statement
}
Within your callback method, you can query the State property to retrieve the Object reference that was passed as the second argument to the QueueRead/QueueWrite methods. Your callback method can also query the Gate property to retrieve a reference to the ReaderWriterGate that was used to orchestrate the calling of the method. It is unlikely that your method would ever query this property unless it was going to initiate some other task on the same resource (guarded by the same ReaderWriterGate object).

The main purpose of the ReaderWriterGateReleaser object is to give your callback method a way to indicate that it is finished holding the resource gate while continuing to execute. Next I will present an example that shows why you might want to use this and how to use it properly. In fact, I suspect that most callback methods will completely ignore the ReaderWriterGateReleaser argument entirely. If the argument is ignored, then the gate will automatically be released when your callback method returns.

Using the ReaderWriterGate

Let's imagine that you're implementing a Web service that accepts catalog orders from customers. In addition, the service accepts requests to update the catalog from company employees. When the service initializes, it constructs an instance of a hypothetical CatalogOrderSystem class (shown in Figure 1). When a CatalogOrderSystem object is constructed, it initializes a private field to refer to a ReaderWriterGate object. This object will be used to orchestrate threads reading from and writing to the CatalogOrderSystem object.

Figure 1 CatalogOrderSystem Class (Close Figure 1)
internal sealed class CatalogOrderSystem
{
   // The gate used to protect access to the Catalog Order System’s data
   private ReaderWriterGate m_gate = new ReaderWriterGate();

   public void UpdateCatalogWS(CatalogEntry[] catalogEntries)
   {
      ... // Perform any validation/pre-processing on catalogEntries...

      // Updating the catalog requires exclusive access to it.
      m_gate.QueueWrite(UpdateCatalog, catalogEntries);
   }

   // The code in this method has exclusive access to the catalog.
   private void UpdateCatalog(ReaderWriterGateReleaser r)
   {
      CatalogEntry[] catalogEntries = (CatalogEntry[]) r.State;

      ... // Update the catalog with the new entries...
    } // When this method returns, exclusive access is relinquished.

   public void BuyCatalogProductsWS(CatIDAndQuantity[] items)
   {
      // Buying products requires read access to the catalog.
      m_gate.QueueRead(BuyCatalogProducts, items);
   }

   // The code in this method has shared read access to the catalog.
   private void BuyCatalogProducts(ReaderWriterGateReleaser r)
   {
      using (r)
      {
         CatIDAndQuantity[] items = (CatIDAndQuantity[])r.State;
         foreach (CatIDAndQuantity item in items)
         {
            ... // Process each catalog item to build customer’s order...
         }
      } // When r is Disposed, read access is relinquished.

      ... // Save customer’s order to database
      ... // Send customer e-mail confirming order
   }
}

Let's say that a request comes in to update the order catalog; when it does, the UpdateCatalogWS method is called and is passed a reference to an array of CatalogEntry objects. Inside UpdateCatalogWS, it first performs any validation or preprocessing required on the array's objects. This work is performed while the ReaderWriterGate is not acquired, allowing this thread to operate concurrently with other threads that may currently be accessing the CatalogOrderSystem object. If all goes well, then to actually modify the order catalog with the new entries, exclusive access is required to the catalog order data. So, QueueWrite is called and is passed UpdateCatalog (the method that actually updates the catalog) and the array of new catalogEntries.

Internally, the ReaderWriterGate will have a thread pool thread call UpdateCatalog when and only when exclusive access can be given to code executing within the method. When UpdateCatalog is called, it can assume that no other threads are reading or writing to the OrderCatalogSystem object and it can update the catalog entries or do whatever it needs to do. When the UpdateCatalog method returns, the gate will automatically be released and the gate will allow any queued method to be called, allowing the method's code to manipulate the OrderCatalogSystem object.

Now, let's say that a request comes in to place an order; this means that the BuyCatalogProductsWS method is called and this method is passed a reference to an array of CatIDAndQuantity objects. Inside BuyCatalogProductsWS, it needs to read the catalog information in order to complete the customer's order request. To make sure that the catalog can be read safely, read access is required. So, QueueRead is called passing BuyCatalogProducts (the method that actually completes the customer's order) and the array of items.

The ReaderWriterGate will call BuyCatalogProducts when and only when read access can be given to code executing within the method. When BuyCatalogProducts is called, it can assume that no other threads are writing to the OrderCatalogSystem object (although other threads may be reading from it) and it can do whatever it needs to do. When the BuyCatalogProducts method executes, it first looks up the catalog entries in the catalog; this requires read access to the catalog. Then, after all the entries have been looked up, the catalog no longer needs to be accessed but there is more work required in order to complete the processing of the customer's order. Normally, the ReaderWriterGate is released when the callback method returns. But, in order to improve performance and scalability, the BuyCatalogProducts method wants to release the gate early; as soon as it no longer executes code that reads from the catalog.

To release the gate before the method returns, I used a C# using statement. At this statement's closing brace, the C# compiler emits a finally block that contains code to call the ReaderWriterGateReleaser's Dispose method. This notifies the ReaderWriterGate that the thread has finished reading the resource and the lock may allow other queued methods to be called. At this point, the BuyCatalogProducts method does not return; instead, it continues processing the customer's request. When the BuyCatalogProducts method does return, the ReaderWriterGate is normally released. However, in this case, the code inside the ReaderWriterGate that called BuyCatalogProducts knows that the ReaderWriterGate has already been released and it will not release it again. In fact, in compliance with the .NET Framework guidelines for implementing IDisposable, if you called Dispose (or Release) on a single ReaderWriterGateReleaser object multiple times, only the first invocation actually releases the gate; additional invocations do nothing and just return.

How ReaderWriterGate is Implemented

Now that you see how to use the ReaderWriterGate class, let's look at how I implemented it. A ReaderWriterGate object contains the private instance fields shown in Figure 2.

Figure 2 ReaderWriterGate Private Instance Fields (Close Figure 2)
public sealed class ReaderWriterGate
{
   // Used for thread-safe access to other fields
   private ResourceLock m_syncLock = new MonitorResourceLock();

   // The current state of the gate; see the ReaderWriteGateStates enum
   private ReaderWriterGateStates m_state = ReaderWriterGateStates.Free;

   // The number of methods desiring shared access currently executing
   private Int32 m_numReaders = 0;

   // A FIFO queue of methods desiring exclusive access
   private Queue<ReaderWriterGateReleaser> m_qWriteRequests =
      new Queue<ReaderWriterGateReleaser>();

   // A FIFO queue of methods desiring shared access
   private Queue<ReaderWriterGateReleaser> m_qReadRequests =
      new Queue<ReaderWriterGateReleaser>();
}

private enum ReaderWriterGateStates
{
   // No methods are desiring access
   Free,

   // One or more methods desiring shared access are executing
   // and no writer methods are queued up desiring access
   OwnedByReaders,

   // One or more methods desiring shared access are executing
   // and one or more writer methods are queued up desiring access
   OwnedByReadersAndWriterPending,

   // One method desiring exclusive access is executing
   OwnedByWriter
}

When a thread calls QueueWrite, this method immediately constructs a ReaderWriterGateReleaser object, passing to the constructor the delegate to the method that will do the actual writing to the resource, a reference to the ReaderWriterGate object, a Boolean value of true indicating that the callback method will be writing to the resource, and the state object passed as the second argument to QueueWrite. The thread calling QueueWrite checks the m_state field to determine what to do. Figure 3 shows the basic logic implemented within QueueWrite.

Figure 3 Basic Logic of QueueWrite (Close Figure 3)
1. If m_state is Free
    1.1 Set m_state to OwnedByWriter
    1.2 Invoke callback method via ThreadPool’s QueueUserWorkItem
    1.3 Return to caller
2. If m_state is OwnedByReaders or OwnedByReadersAndWriterPending
    2.1 Set m_state to OwnedByReadersAndWriterPending
    2.2 Add ReaderWriterGateReleaser object to m_qWriteRequests
    2.3 Return to caller
3. If m_state is OwnedByWriter
    3.1 Do not change m_state
    3.2 Add ReaderWriterGateReleaser object to m_qWriteRequests
    3.3 Return to caller

Note that QueueWrite always returns immediately; it never waits for the callback method to complete its execution. This behavior is different from normal thread synchronization locks such as Monitor or ReaderWriterLock where the calling thread is blocked until the thread is allowed access. For the ReaderWriterGate, the calling thread either has a thread pool thread execute the callback method or it adds an entry to a queue; the calling thread never waits to get access to the resource being protected by the gate. This is an extremely important distinction. The thread calling QueueWrite cannot be sure that the callback method has executed by the time QueueWrite returns.

When a thread calls QueueRead, this method immediately constructs a ReaderWriterGateReleaser object, passing to the constructor the delegate to the method that will do the actual reading from the resource, a reference to the ReaderWriterGate object, a Boolean value of false indicating that the callback method will be reading from the resource and the state object passed as the second argument to QueueRead. Figure 4 shows the basic logic implemented within QueueRead.

Figure 4 Pseudocode for QueueRead (Close Figure 4)
1. If m_state is Free or OwnedByReaders
    1.1 Set m_state to OwnedByReaders
    1.2 Add 1 to m_numReaders
    1.3 Invoke callback method via ThreadPool’s QueueUserWorkItem
    1.4 Return to caller
2. If m_state is OwnedByWriter or OwnedByReadersAndWriterPending
    2.1 Do not change m_state
    2.2 Add ReaderWriterGateReleaser object to m_qReadRequests
    2.3 Return to caller

Like QueueWrite, QueueRead always returns immediately and never waits for the callback method to complete its execution. When QueueRead is called, it either queues the callback method to the thread pool or it adds the ReaderWriterGateReleaser object onto the m_qReadRequests queue and then the method returns. The thread calling QueueRead cannot be sure that the callback method has executed by the time QueueRead returns.

Why the ReaderWriterGate is Cool

Let's go back to the Web service example. Let's say a thread makes a write request against the server. Initially, a thread pool thread wakes up and calls QueueWrite. If the gate is Free, then the callback method is queued to the thread pool and the QueueWrite method immediately returns and this thread pool thread will likely return to the thread pool. When it returns to the thread pool, it will see the queued callback method and will call the method to do the work. In this case, one thread does everything and no context switch occurs—the performance is awesome!

Now, let's say that the callback method takes a long time to execute. When it is executing, additional requests could be coming in to the Web server. Let's say 100 requests come in to read data. All of these requests get queued to the common language runtime's (CLR's) thread pool. Since one thread pool thread is busy executing the previously queued write method, another thread pool thread will wake up and call into the Web service code. The Web service code will call QueueRead, see that the gate is not Free and will just construct a ReaderWriterGateReleaser object identifying the callback and add this object to the m_qReadRequests queue. This second thread pool thread will then return from QueueRead and will ultimately return all the way back to the thread pool.

Note that this thread is not blocked. In fact, if 99 more read requests come into the Web service, this thread will extract those 99 read requests (one at a time) off the thread pool and simply add a bunch of ReaderWriterGateReleaser objects to the m_qReadRequests queue. At this point, the thread pool has needed just two threads and now one thread is back in the pool ready to handle more incoming requests and the other thread is still executing the write method. If a write request comes in, a ReaderWriterGateReleaser object is constructed and then added to the m_qWriteRequests queue, and once again the thread returns immediately to the thread pool. This is incredibly scalable and uses an extremely small number of system resources!

Now, when the first thread pool thread finishes processing the write method, the gate is automatically released, and the ReaderWriterGate must then examine its queues to determine what write or read methods to invoke next. The internal logic looks something like Figure 5.

Figure 5 Pseudocode for Examining Queues (Close Figure 5)
1. If a reader thread is releasing the gate, do steps 1.1 & 1.2;
    else, go  to step #2
    1.1. Subtract 1 from m_numReaders
    1.2. If m_numReaders is > 0, return to caller; else, goto step #2
2. If m_qWriteRequests.Count > 0
    2.1. Set m_state to OwnedByWriter
    2.2. Invoke 1 writer callback method via ThreadPool’s
         QueueUserWorkItem
    2.3. Return to caller
3. If m_qReadRequests.Count > 0
    3.1. Set m_state to OwnedByReaders
    3.2. Set m_numReaders = m_qReadRequests.Count
    3.3. Invoke all reader callback methods via ThreadPool’s
         QueueUserWorkItem
    3.4. Return to caller
4. Set m_state to Free
5. Return to caller

From the logic, you can see that when a reader callback method returns, the gate checks to see if it is the last reader and, if not, then no additional methods will get invoked and the thread pool thread executing the callback method returns to the thread pool so that it can be used to do something else.

If there are no more readers reading, then the gate checks to see if there are any writer methods queued up in the m_qWriteRequests queue. If there are, the next writer's callback method is posted to the CLR's thread pool, causing a thread pool thread to wake up and execute the code that writes to the resource. Note that only one writer method is invoked at a time, ensuring exclusive access to the data being protected by the gate. Also note that my implementation always prefers to wake writers, which can starve readers, but this is the usually case with reader/writer locks. You could always change the policy to prefer readers if you desire.

If there are writers queued up, then the gate checks to see if there are any reader methods queued up in the m_qReadRequests queue. If there are, all of the reader callback methods are posted to the CLR's thread pool since, theoretically, they can all execute at once. However, you may remember that earlier I mentioned that there was a significant performance problem when a normal reader/writer lock releases all its waiting reader threads: all threads are runnable and the operating system must perform a lot of context switching, which adversely affects performance.

In the case of the ReaderWriterGate, all of the reader callback methods are queued up to the thread pool which knows to invoke only a few of the methods at a time using just a few threads. Ideally, the number of threads used is equal to the number of CPUs in the computer itself and, therefore, no context switching takes place. These few threads will execute the read callback methods and then go back to the thread pool and execute a few more. The end result is that very few threads are required to perform lots of work and the work executes more quickly because fewer context switches are necessary. Better performance with fewer resources—you gotta love it!

Knowing When the Callback Completes

Shortly after I released the first version of my Power Threading Library with my ReaderWriterGate, someone tried to use it in an ASP.NET application. His scenario was similar to what I have already described: clients were making requests for Web pages, and to gain scalability, the Web pages needed to access shared data using reader/writer semantics. Fortunately, ASP.NET 2.0 has support for implementing asynchronous Web pages. For more information about this, I encourage you to read Jeff Prosise's Wicked Code column about asynchronous pages in ASP.NET 2.0, which appeared in the October 2005 issue of MSDN®Magazine.

To take advantage of asynchronous Web pages, you need to issue an asynchronous request and return to the ASP.NET infrastructure an IAsyncResult object that it can use to know when the asynchronous operation has completed. You use the ReaderWriterGate class's QueueWrite and QueueRead methods to start the asynchronous operation, but in my original implementation I offered no way for you to know when the callback method had actually completed its task. This meant that the ReaderWriterGate could not easily be used in an ASP.NET Web Forms application.

After I got this user's feedback, I realized how useful this would be and I added a few additional methods to the ReaderWriterGate class. Here are the new methods:

public sealed class ReaderWriterGate
{
   public IAsyncResult BeginRead(ReaderWriterGateCallback callback,
      Object state, AsyncCallback asyncCallback, Object asyncState);
   public void EndRead(IAsyncResult ar);

   public IAsyncResult BeginWrite(ReaderWriterGateCallback callback,
      Object state, AsyncCallback asyncCallback, Object asyncState);
   public void EndWrite(IAsyncResult ar);

   ... // Other methods not shown
}

You would use the BeginWrite/BeginRead and EndWrite/EndRead methods instead of the QueueWrite/QueueRead method when writing an asynchronous ASP.NET Web page or any kind of application where you need to know when the callback method has completed execution. These BeginXxx methods always return immediately but now, when they return, I give you back an IAsyncResult object that identifies the queued callback method. The IAsyncResult object I return fits in with the rest of the CLR's asynchronous programming model and you can use it just like you would any other IAsyncResult object. In the case of an ASP.NET Web page, you can return my object to the ASP.NET infrastructure directly and it will know to complete processing the page when the callback method completes. Also, just like the CLR's asynchronous programming model, when you call EndWrite/EndRead, I throw any exception that was unhandled by the callback method.

Conclusion

I am particularly proud of the ReaderWriterGate idea and of my implementation. I think it solves a real-world need for producing scalable applications and services using a minimum of resources. Even though threading has been around for many years, I still believe that there are new ways to think about it that can lead to new ideas and ways of architecting applications and services to start taking advantage of the multiple-CPU machines that are now becoming commonplace.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), an architecture review, consulting, and training firm. He is the author of several books, including CLR via C# (Microsoft Press, 2006). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


.NET Matters: Event Accessors

.NET Matters
Event Accessors
Stephen Toub

Q C# makes it very easy to create events on classes, simply by adding the keyword "event" to a delegate member variable declaration. However, it also allows a property-like syntax where add and remove accessors for the event can be implemented explicitly. Why would I ever want to do that? Wouldn't I always just be recreating the same code the C# compiler generates for me?

A There are several reasons you might want or need to implement your own add and remove accessors for an event in C#. I'll take a look at several of these (this isn't an exhaustive list) and show how custom accessors can enable new functionality or even improve performance.

To begin, consider a simple class, MyClass, with a typical instance event, MyEvent:

class MyClass
{
    public event EventHandler MyEvent;
    ...
}

When the C# compiler generates code for MyClass, the output Microsoft® Intermediate Language (MSIL) is identical in behavior to what would have been produced using code like that in Figure 1.

Figure 1 Expanded Event Implementation (Close Figure 1)
class MyClass
{
    private EventHandler _myEvent;

    public event EventHandler MyEvent
    {
        [MethodImpl(MethodImplOptions.Synchronized)]
        add
        {
            _myEvent = (EventHandler)Delegate.Combine(_myEvent, value);
        }
        [MethodImpl(MethodImplOptions.Synchronized)]
        remove
        {
            _myEvent = (EventHandler)Delegate.Remove(_myEvent, value);
        }
    }
    ...
}

With the simple event syntax, when you reference MyEvent in your code in order to invoke it, the C# compiler translates that into a call to the underlying delegate, such that a call like the following

MyEvent(this, EventArgs.Empty);
actually compiles down to code like:
_myEvent(this, EventArgs.Empty);

Using the explicit implementation, where you write your own custom add and remove accessors, the compiler doesn't know about the underlying data store for the event, so you can no longer invoke the event using the name of the event itself, but rather you must reference the delegate directly.

This hints at one of the primary reasons for writing your own add and remove accessors: to provide your own underlying data store. One reason you might want to do this is if you have lots of exposed events on your class, but in such a way that only a few are typically in use on an instance at any point in time. In such a scenario, there can be significant memory overhead associated with maintaining a delegate field for each event. Take the Windows® Forms Control class as an example.

In the Microsoft .NET Framework 2.0, Control exposes 69 public events. If each of these events had an underlying delegate field, on a 32-bit system these events would add an overhead of 276 bytes per instance. Instead, Control (or, more specifically, its base class System.ComponentModel.Component) maintains a list of key/value pairs, where the value is a delegate. Every event on Control then has custom add and remove accessors that store the registered delegates for all events into this list, an instance of the System.ComponentModel.EventHandlersList class. Assuming that only a few events on a particular Control are handled, the performance impact of having to search through the EventHandlersList for a particular delegate is minimal, and the memory overhead is limited to what's necessary for the EventHandlersList implementation. You can implement this same pattern in your own classes, as shown in Figure 2.

Figure 2 Custom Store for Event Delegates (Close Figure 2)
class MyClass
{
    private static readonly _myEvent = new object();

    private EventHandlerList _handlers = new EventHandlerList();

    public event EventHandler MyEvent
    {
        add { _handlers.AddHandler(_myEvent, value); }
        remove { _handlers.RemoveHandler(_myEvent, value); }
    }

    private void OnMyEvent()
    {
        EventHandler myEvent = _handlers[_myEvent] as EventHandler;
        if (myEvent != null) myEvent(this, EventArgs.Empty);
    }

    ...
}

I've followed the convention used in Windows Forms, which is to have one static object for each event to use as the key into each EventHandlerList. Since this is a static field, there's only one for the whole AppDomain, rather than one for each instance. You could opt to use a string for the key, but then you run the risk of typos that won't be caught until the relevant lookups fail at runtime; the static object approach moves most of these kinds of failures to compile time.

A related use for explicit event implementation, even if you want to use a single delegate as the backing store for the event, is to augment the delegate with attributes. The most common example of why this may be important is serialization. The .NET serialization mechanism, by default, serializes out all fields of a serializable class, regardless of their accessibility (this is in contrast to XML serialization, where by default only public fields and public properties are serialized). All fields of a serializable class must either themselves be serializable or must be exempted from the serialization process using the NonSerializable attribute (alternatively, the class can implement the ISerializable interface to completely control what data is serialized and how), since they will be traversed by a formatter saving out the object graph.

The trouble with events when it comes to .NET serialization is that they're backed by a private delegate field, and delegates are serializable. When you create a delegate for an instance method, the delegate maintains a reference to the instance on which to call that method, so when you serialize an object that contains an event, the formatter, while walking the object graph, will continue on down through the delegate attempting to serialize any instances registered with that delegate. If you're not aware of this and don't plan accordingly, you could experience some unwelcome side effects. If it's not the behavior you want, a solution is to mark the backing store for the event with the NonSerializableAttribute, as shown in Figure 3. This will cause the formatter to skip that field when serializing out the object graph.

Figure 3 Expanded Event Implementation (Close Figure 3)
[Serializable]
class MyClass
{
    [NonSerializable]
    private EventHandler _myEvent;

    public event EventHandler MyEvent
    {
        [MethodImpl(MethodImplOptions.Synchronized)]
        add
        {
            _myEvent = (EventHandler)Delegate.Combine(_myEvent, value);
        }
        [MethodImpl(MethodImplOptions.Synchronized)]
        remove
        {
            _myEvent = (EventHandler)Delegate.Remove(_myEvent, value);
        }
    }
    ...
}

Another use for explicit event implementation is to provide a custom synchronization mechanism (or to remove one). You'll notice in Figure 1 that both the add and remove accessors are adorned with a MethodImplAttribute that specifies that the accessors should be synchronized. For instance events, this attribute is equivalent to wrapping the contents of each accessor with a lock on the current instance:

add { lock(this) _myEvent += value; }
remove { lock(this) _myEvent -= value; }

If you're writing a class that will not be used from multiple threads, there's no point in the overhead of taking this lock, so you could provide a custom implementation that lacks the MethodImplAttribute. However, the overhead of taking such a lock in typical event-usage scenarios is negligible, so it's rare that this would justify a custom implementation. More importantly, however, it's considered poor form to use lock(this), as in effect you're exposing what should be an implementation detail (the lock) to all consumers of your class (they, too, can lock on your instance). If locking is necessary, it's considered a better practice to lock on a private object. And if you're not synchronizing around the "this" reference, then the default event implementation that does synchronize around that reference will most likely be incorrect, since it won't be synchronized in the same manner as the rest of your class.

Another situation exists in which you may need to explicitly implement an event. Consider two interfaces that both have an event with the same name:

interface I1
{
    event EventHandler MyEvent;
}

interface I2
{
    event EventHandler MyEvent;
}
If I want a class to implement both of these interfaces, I can do so as follows:
class MyClass : I1, I2
{
    public event EventHandler MyEvent;
}
Here, MyClass.MyEvent implements both I1.MyEvent and I2.MyEvent. But what if I want MyClass to provide separate implementations for I1.MyEvent and I2.MyEvent? The only way to do so is to use an explicit interface implementation for at least one of the interfaces (though I could do so for both):
class MyClass : I1, I2
{
    public event EventHandler MyEvent;

    private EventHandler _i2MyEvent;
    event EventHandler I2.MyEvent
    {
        add { _i2MyEvent += value; }
        remove { _i2MyEvent -= value; }
    }
}

A fourth reason you might implement custom add or remove accessors is to execute custom logic any time a delegate is registered or unregistered from the event. In what scenarios would that be useful? Several.

Consider the Microsoft.Win32.SystemEvents class. This class exposes over a dozen public static events that are raised in response to various happenings on the system: DisplaySettingsChanged, InstalledFontsChanged, UserPreferenceChanging, and so forth. An application is informed of most of these system notifications through Windows messages. For SystemEvents to raise .NET events for the notifications, it needs a window to listen for them and a thread running a message loop to respond to them. There's overhead involved in such functionality, so SystemEvents only wants to initialize (creating the broadcast window, running the message loop, and so forth) if a delegate is registered with one of the events. And the perfect place to kick off the logic to ensure the class has been initialized is in the add accessor for the delegate, as shown in Figure 4.

Figure 4 Initialization Logic in Add Accessor (Close Figure 4)
class MyClass
{
    private static object _lock = new object();
    private static bool _initialized;
    private static EventHandler _myEvent;

    public static event EventHandler MyEvent
    {
        add
        {
            lock(_lock)
            {
                EnsureInitialized();
                _myEvent += value;
            }
        }
        remove { lock(_lock) _myEvent -= value; }
    }

    private static void EnsureInitialized()
    {
        if (!_initialized)
        {
            ... // do initialization here
            _initialized = true;
        }
    }

    ...
}

Another kind of custom logic you might want executed when a delegate is registered is permission demands. Take the System.Diagnostics.EventLog class as an example. Many of the operations on EventLog are gated by the EventLogPermission, which allows for permission states as specified through an EventLogPermissionAccess enumeration, with members like Administer, Write, and None. The EventLog class exposes a public EntryWritten event that is raised whenever an entry is written to the log, but that act of responding to new entries in an event log requires the Administer permission. Where should that permission be demanded? Code that doesn't have the permission shouldn't be allowed to register a delegate with the event, so a demand can be done in the add accessor, as shown in Figure 5.

Figure 5 Permission Demand in Add Accessor (Close Figure 5)
class MyClass
{
    private EventHandler _myEvent;

    public event EventHandler MyEvent
    {
        add
        {
            new EventLogPermission(
                EventLogPermissionAccess.Administer, ".").Demand();
            _myEvent += value;
        }
        remove { _myEvent -= value; }
    }
}

The System.Console class makes use of custom logic for both custom initialization and permission demands. Console exposes the static CancelKeyPress event, which is raised when the Ctrl+C or Ctrl+Break key combinations are used in the console. Typically these will terminate the process, but if a ConsoleCancelEventArgs delegate is registered with the event, that delegate can override the termination through the ConsoleCancelEventArgs.Cancel property. In order to perform this interception, Console must register a handler to hook the signals using SetConsoleCtrlHandler from Kernel32.dll, and an appropriate place to do this is in the add accessor for the event (the remove accessor contains the corresponding logic to unregister the handler). Additionally, CancelKeyPress requires UIPermissionWindow.SafeTopLevelWindows permission, and thus it's demanded in both accessors for the event.

A much more obscure use of custom logic in an add accessor (for the very advanced developers out there) is useful for events to be raised within constrained execution regions, or CERs. (For a detailed look at CERs, see my article in the October 2005 issue of MSDN®Magazine. The basic idea here is that asynchronous exceptions resulting from situations such as out-of-memory can destabilize an application by interrupting normal code flow in a way that prevents standard backout code from executing correctly. In order to help prevent these types of conditions from occurring, code can be JIT-compiled and prepared ahead of time such that these sorts of conditions are hopefully moved to either before or after the code runs.

Now consider an event that is to be raised from within a CER, where you want to avoid all of these dangerous conditions. You want to prepare ahead of time all of the code to be executed in the CER, but if you're raising an event to which any delegate can be registered, unprepared code could in fact run within the CER. This is exactly the situation that several events on the AppDomain class run into. For example, AppDomain.ProcessExit is raised from within a CER, so all delegates registered with it should be prepared ahead of time. How can this be accomplished? By adding code to the event's add accessor that prepares the supplied delegate before it's registered with the event, as shown in Figure 6.

Figure 6 Delegate Preparation in Add Accessor (Close Figure 6)
class MyClass
{
    private EventHandler _myEvent;

    public event EventHandler MyEvent
    {
        add
        {
            System.Runtime.CompilerServices.
                RuntimeHelpers.PrepareDelegate(value);
            _myEvent += value;
        }
        remove { _myEvent -= value; }
    }
}

This is just a subset of reasons why you might want to provide your own add and remove accessors for an event. Most of the time, the stock accessors generated by the compiler are fine. But there may be times when you need more control, making this a very handy feature to have in your back pocket.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


Netting C++: Introducing Regular Expressions

Netting C++
Introducing Regular Expressions
Stanley B. Lippman

Code download available at: NettingC++2006_11.exe (159KB)

One of the most significant benefits of moving from ISO-C++ to C++/CLI is the enormous Microsoft® .NET Framework Class Library (FCL) that immediately becomes available. Most readers are aware of the more glamorous domains supported by the FCL, such as Web services, networking, threads, ASP.NET, and so on. In this column, I introduce the support for regular expressions in the .NET Framework.

Because ISO-C++ is without standard support for regular expressions, my TQL application (introduced in previous Netting C++ columns) uses the standard string class for all pattern matching. This occurs, for example, with the need to normalize word occurrences within a text, such as recognizing "tried", "tries", and "trying" as instances of "try". With the move to C++/CLI, I'd like to replace the string handling with the more concise and powerful .NET regular expression support. That's what I hope to accomplish in the next column, after making my way through a brief regular expression tutorial here.

Before I look at how the FCL supports regular expressions, it would be a good idea to get some feel for what they are and how you can use them.

A regular expression is a pattern of characters and symbols representing a character sequence of some arbitrary length. For example, let's say you need to find all lines of text that begin with a certain sequence of characters. The line must begin with the numeral 5. The number can be of any length, but must be followed by a dash (5- would be the minimal matching string, but 51- and 510- and so on would also match). Following the dash must be one of the letters a, b, or c—for example, 51-a, 511100101-b, and so on. This must be followed by one or more letters or characters and must end with the sequence 2001. (I know, it seems bizarre, but these are just the kinds of sequences that a regular expression denotes.)

In order to express the rules of this sequence using a regular expression, you need a set of symbols to do the following:

First, indicate that you want to begin the search at the beginning of the line. You do this with the caret (^). So, for example, ^5 means that you want the line to begin with a literal value of 5.

Second, indicate that you want to match on a particular flavor of character. So, for example, \d means that you want to match a single digit between 0-9, and \D means that you want to match against a single non-digit character. (Technically, \d and \D also work with international digits, as pointed out in Raymond Chen's blog post at blogs.msdn.com/86555.aspx.) \s means that you want to match a single white space character, while \S means that you want to match a single character that is not white space. \w matches any alpha-numeric character [a-z, A-Z, 0-9], and \W matches any character that is not alphanumeric.

Third, indicate that you want to match on any character, regardless of its type. For example, the period (.) matches any non-newline character.

Fourth, indicate that you want to match on multiple (or no) instances of a character type. The plus operator (+) means that you want to match one or more characters of the same type. \d+, for example, matches "2", "22", "1217", and so on. The following regular expression matches any line that begins with 5 followed by one or more additional digits, followed by one or more non-digit characters followed by the literal 2001:

^5\d+\D+2001

What if you were not sure if the 5 were going to be followed by one or more digits? You'd like the flexibility to indicate that you will accept any number of digits, or no digits at all. You can do that by using the asterisk (*). For example, let's turn the previous example into one in which the 5 can be followed by zero or more digits followed by zero or more non-digit characters, but ending with the literal 2001:

^5\d*\D*2001

Fifth, indicate that you want to match on a fixed number of characters. Therefore the following regular expression requires three digits followed by a hyphen followed by four digits (like a telephone number), such as 375-4128:

\d{3}-\d{4}

Finally, indicate that you want to match against one of a set of different characters. You do this by placing a set of alternative characters within parentheses, separated by an OR-bar (|). For example, the expression (a|e|i|o|u) means that you want to match against one of the five English vowels. Adding the plus operator means that you want to match against one or more consecutive occurrences of the five English vowels. Following it with the asterisk means you want to allow for no matches as well.

Regular expressions take some getting used to. In the beginning, they seem quite complicated because they offer such compact notation. Figure 1 shows the run of a small regular expression tester-outer program (note that my Console input is highlighted in red).

Figure 1 Regular Expression Tester (Close Figure 1)
Would you like to enter a string to match against? (Y/N/?) y
Please enter a string, or ‘quit’ to exit.
==> 5abc2001

Would you like to change regular expressions? (Y/N/?) y
Please enter regular expression:
      **> ^5\d*(a|d|e)\w+2001

original string:  5abc2001
attempt to match: ^5\d*(a|d|e)\w+2001

The characters 5abc2001 match beginning at position 0

Would you like to enter a string to match against? (Y/N/?) y
Please enter a string, or ‘quit’ to exit.
==> 527ar2001

Would you like to change regular expressions? (Y/N/?) n

original string:  527ar2001
attempt to match: ^5\d*(a|d|e)\w+2001
The characters 527ar2001 match beginning at position 0

Of course, there can be multiple matches as well. For example:

original string:  r24d2
attempt to match: \d+
The characters 24 match beginning at position 1
The characters 2 match beginning at position 4

So let's try our hand at programming regular expressions. First, let me show you the code that does the actual matching, and then I'll explain what's going on. In Figure 2, System::Text::RegularExpressions is the namespace within which the regular expression support is contained.

Figure 2 Building Regular Expressions (Close Figure 2)
using namespace System::Text::RegularExpressions;
void doTheMatch( String^ inputString, String^ filter )
{
    Console::WriteLine( "original string:  {0}", inputString );
    Console::WriteLine( "attempt to match: {0}", filter );

    Regex^ regex = gcnew Regex( filter );
    Match^ match = regex->Match( inputString );

    if ( ! match->Success )
    {
        Console::WriteLine(
            "Sorry, no match of {0} in {1}", filter, inputString );
        return;
    }

    for ( ; match->Success; match = match->NextMatch() )
    {
        Console.WriteLine(
            "The characters {0} match beginning at position {1}",
            match->ToString(), match->Index );
    }
}

The Regex class represents the regular expression. You pass its constructor the string representation of the expression. Once the Regex object is constructed, its associated regular expression is immutable—that is, you cannot change it. So each regular expression within your program requires its own Regex object.

The Match method performs the actual matching algorithm of the regular expression against its string argument. It returns a Match class object that holds the results of the pattern matching. The Match object is also immutable.

To discover whether the match succeeded, query the Success property of the Match class. Each match is spoken of as a capture. The Index property returns the position in the original string where the first character of the captured substring was found. Length returns the length of the captured substring. The ToString method returns the captured substring.

Here is a typical for-loop to collect and process the collection of matching patterns:

for ( Match^ match = regex.Match( inputString );
      match->Success; match = match->NextMatch() )
{ ... }
The Match object holds the results of the first capture. If the regular expression captures multiple substrings, use NextMatch to access the second and each subsequent capture. Before you actually manipulate the next object, you need to test that it represents a success. A sentinel Match object for which Success evaluates to false marks the end of the captured substrings. Consider the following three lines:
5040      bez( 99,   -3.194, 43.8,  85    )
4930.7823 bez( 10.7, 19.59,  -20,  -20.48 )
-5123      bez( -3.5,  2.46,  89,     0.02 )
These represent samples of lines that you need to match. First, you have to come up with a regular expression that can match each of these lines.

As you can see, each line begins with a number. The number can be either positive or negative and can represent either a scalar or floating point value. The number is followed by a space, then the literal substring "bez". Four comma-separated numbers follow that enclosed within parentheses. The numbers can be negative or positive and they can be either integers or floating point values. Before you look at my solution, try your hand at coming up with a regular expression that captures each of these lines in its entirety.

Once you have the regular expression, you're still not finished. The next problem is, how do you gain access to the individual parts of the line? That is, the regular expression captures the entire string and you now need to pick it apart in order to access the five numeric fields.

The regular expression syntax supports a grouping mechanism in which you assign index numbers to particular subfields of the match.You can subsequently use these numbers to access the subfields. For example, the following identifies a group associated with the index 1 using the special ?<1> syntax:

(?<1>(-?\d+\.\d+)|(-?\d+))

Can you read this? It represents an alternate pair of regular expressions. The first one matches a floating point number that may or may not be negative (the ? character is used to specify zero or one matches):

-?\d+\.\d+
The second matches an integer value that also may or may not be negative:
-?\d+
The entire regular expression, with five identified groups is shown in Figure 3. For clarity, I've broken it up and identified each subfield.

Figure 3 The Assembled Regular Expression (Close Figure 3)
String^ filter =
    // the digit before the bez literal
    "(?<1>(-?\\d+\\.\\d+)|(-?\\d+))"

    // arbitrary white space, bez literal and open paren
    "\\s*bez\\("

    // the four internal numeric values and literal comma
    "(?<2>(-?\\d+\\.\\d+)|(-?\\d+)),"
    "(?<3>(-?\\d+\\.\\d+)|(-?\\d+)),"
    "(?<4>(-?\\d+\\.\\d+)|(-?\\d+)),"
    "(?<5>(-?\\d+\\.\\d+)|(-?\\d+))"
;

Now attempt the match on the line of text:

Regex^ regex = gcnew Regex( filter );
Match^ match = regex->Match( line );
If the match is successful, you then need to grab each of the five numeric subfields and translate them into values of type float:
float loc = match->Group(1)->ToString()->ToSingle();
float m_xoffset1 = match->Group(2)->ToString()->ToSingle();
float m_yoffset1 = match->Group(3)->ToString()->ToSingle();
float m_xoffset2 = match->Group(4)->ToString()->ToSingle();
float m_yoffset2 = match->Group(5)->ToString()->ToSingle();
The Group class represents a capturing group within the returned Match class object. You can access each Group object through its associated index. The ToString method returns the captured substring. In this case, you invoke the ToSingle conversion method on each string to transform the value into type float.

A useful Regex class method is Split. Just like the String class Split method, it returns a string array. Unlike the String method, it separates the input string based on a regular expression rather than a set of characters:

String^ textLine =
    "Danny%Lippman%%Point Guard%Shooting Guard%%floater";

String^ splitMe = "%+";
Regex^  regex = gcnew Regex( splitMe );

for each ( String^ capture in regex->Split( textLine ))
    Console::WriteLine( "capture: {0}", capture );
In this example, the textLine is being split at each point where one or more percent characters (%) appear. When executed, this generates the following output:
capture: Danny
capture: Lippman
capture: Point Guard
capture: Shooting Guard
capture: floater

Another useful Regex class method is Replace, which allows you to replace captured substrings with alternative text. Figure 4 shows a simple example.

When compiled and executed, this generates the following output—I've reformatted it slightly to display better:

original text: XP.109 is currently in alpha.
XP.109 represents a staggering leap forward

regular expression : XP.\d+

replacement text: ToonShooter is currently in alpha.
ToonShooter represents a staggering leap forward
The programs used to generate this output are available for download. The only thing left to accomplish is to replace all the uses of the String class within TQL with the .NET Framework regular expression support. In the next column, I'll do just that. Until then, may all your programs run optimally. Cheers.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. Later, Stan worked in feature animation both at Disney and DreamWorks and served as a Software Technical Director on Fantasia 2000. He has since served as Distinguished Consultant with JPL, and an Architect with the Visual C++ team at Microsoft.


{ End Bracket }: Peripheral and Foveal Vision.

{ End Bracket }
Peripheral and Foveal Vision.
Bill Hill

To write software that’s friendly and easy to use, first you have to really understand who’s using it.

We think we’re so civilized, but we’ve only been doing this civilization stuff for about 13,000 years, not nearly enough time for our perception system—the way we interpret input from the world outside ourselves—to have evolved far from where it started. We’re still Homo Sapiens version 1.0. Our "operating system" first shipped about 140,000 years ago, and there’s still no upgrade in sight. HS 1.0 is the most important OS for which you’ll ever write code.

You wouldn’t dream of writing code for Windows® without understanding the Windows event-handling model or APIs, would you? If you want to write code that’s usable by people, you have to understand their event-handling model.

When we feel information overload as we sit in front of our computer display, it’s because the flow of information is not well matched to our perception system. Our primary input comes through our vision. We have about 207 degrees of peripheral vision. We use our ears to monitor sounds, which tells us when to turn our heads to give us 360 degree visual coverage. There’s a high resolution area of our vision, slightly smaller than a sheet of paper, held at normal reading distance. Our peripheral vision uses the whole retina, but this high-resolution area uses only the foveal area—and that’s only about 0.2mm in diameter.

As hunter-gatherers, our brains developed to prioritize inputs. An unexpected sound from behind, for instance, often signals danger. We are incredibly sensitive to sudden movement on the periphery of our vision; it’s a "Priority 0 Interrupt" because to a hunter-gatherer it means either threat—or lunch.

We use our peripheral vision to triage information, and our brain can pattern-match and sort huge amounts of data to give attention only to what’s important. Once we decide to focus in on something, we pick it up or get close enough to use our high-resolution foveal vision.

That’s the key to information overload. Our computer display generally sits right in our foveal vision. There’s some periphery there, but not much. It’s like we took this 64-bit channel, then put an 8-bit nozzle on it—and wonder why we feel increased pressure. As we stream more and more information through that nozzle, the channel gets overloaded. As the computer becomes our window on the world, this gets to be a problem. If the world is so big, why is my window so small?

The lessons are pretty straightforward. Use bigger displays or multiple displays. But whatever the display size, keep what’s going on in the foveal area as stable as possible—unless you REALLY need to get the user’s attention. It’s the Priority 0 Interrupt nature of movement that makes pop-ups or flashing adverts on a Web page so irritating—they’re bound to distract you from reading or any other task. Advertisers know this—that’s exactly what they want.

Our perception system is very sophisticated and highly developed—for basic survival. But we’re really just beginning to use computers as information interfaces. We still have a lot to learn and a long way to go before this way of acquiring information will feel natural. We can help by trying to better match information flow to human perception.

There’s a practical exercise you can do to help understand your own natural perception system. Go out to a park or, even better, a natural area. Relax and try to just look without focusing on anything. After a few minutes, you’ll be amazed at how much visual input you can easily handle. This is a technique my animal tracker friend Jon Young calls "owl vision."

If you want to practice this while moving, you have to combine it with a second technique Jon calls "fox walking" which helps remove unwanted noise from the visual signal. Most of us bounce up and down a little as we walk, which introduces a lot of signal noise and confuses the input. You need to glide along, keeping your eyes on the same plane. Imagine you’re carrying a tray of very full glasses of water which you don’t want to spill. When you’ve practiced enough to get it right, you can leave the tray behind. You’ll get some pretty strange looks at first, until it all becomes natural. You’ll be stunned at how much you’ve never noticed before.

This is your "free laboratory." Once you’ve reset your perception and understand its baseline or natural state, you can use that as a benchmark. And if you think this has nothing to do with software or computers, just remember: you can take the Man out of Nature—but you never take Nature out of Man.

Figure

Bill Hill joined Microsoft in 1995 because he believed the company would lead the transition from reading on paper to reading on the screen. He is one of the inventors of the ClearType technology which dramatically improved the screen readability of text.