 MSDN Magazine November 2003

MSDN Magazine November 2003

Protect It: Safeguard Database Connection Strings and Other Sensitive Settings in Your Code

Protecting application secrets, such as database connection strings and passwords, requires careful consideration of a number of pertinent factors such as how sensitive the data is, who could gain access to it, how to balance security, performance, and maintainability, and so forth. This article explains the fundamentals of data protection and compares a variety of techniques that can be used to protect application settings. The author discusses what to avoid, such as hiding keys in source code and the use of Local Security Authority. In addition, he presents some effective solutions such as the Data Protection API.

Alek Davis Encrypt It: Keep Your Data Secure with the New Advanced Encryption Standard

The Advanced Encryption Standard (AES) is a National Institute of Standards and Technology specification for the encryption of electronic data. It is expected to become the accepted means of encrypting digital information, including financial, telecommunications, and government data. This article presents an overview of AES and explains the algorithms it uses. Included is a complete C# implementation and examples of encrypting .NET data. After reading this article you will be able to encrypt data using AES, test AES-based software, and use AES encryption in your systems.

James McCaffrey Authorize It: Use Role-Based Security in Your Middle Tier .NET Apps with Authorization Manager

Authorization Manager in Windows Server 2003 represents a significant improvement in the administration of role-based security, making it more scalable, flexible, and easier to implement. Using Authorization Manager, you can define roles and the tasks those roles can perform. You can nest roles to inherit characteristics from other roles, and you can define application groups. In addition, Authorization Manager lets you use scripts to modify permissions dynamically, and it allows you to wrap your security logic in a security policy that can be stored in Active Directory. Authorization Manager also includes an easy-to-use API for running access checks. The author discusses all of these topics and demonstrates them with a working sample.

Keith Brown Review It: Expert Tips for Finding Security Defects in Your Code

Reviewing code for security defects is a key ingredient in the software creation process, ranking alongside planning, design, and testing. Here the author reflects over his years of code security reviews to identify patterns and best practices that all developers can follow when tracking down potential security loopholes. The process begins by examining the environment the code runs in, considering the roles of the users who will run it, and studying the history of any security issues the code may have had. After gaining an understanding of these background issues, specific vulnerabilities can be hunted down, including SQL injection attacks, cross-site scripting, and buffer overruns. In addition, certain red flags, such as variable names like "password", "secret," and other obvious but common security blunders, can be searched for and remedied.

Michael Howard Secure It: WS-Security and Remoting Channel Sinks Give Message-Level Security to Your SOAP Packets

As more organizations adopt XML-based Web Services, the need for message-level security has become evident. WS-Security, now supported in the Microsoft .NET Framework, addresses this need. Using the WS-Security framework, developers can implement channel sinks to intercept Remoting messages as they pass through the .NET Remoting infrastructure. The sink can read the message, change it, and pass it along. During this process, the message can be signed for added security. This article explains how to implement a Remoting channel sink that will modify the Remoting message by including a UserName token in the header, then sign the body using the token.

Neeraj Srivastava Obfuscate It: Thwart Reverse Engineering of Your Visual Basic .NET or C# Code

One of the advantages of the .NET architecture is that assemblies built with it contain lots of useful information that can be recovered using ILDASM, the intermediate language disassembler. A side effect, though, is that someone with access to your binaries can recover a good approximation of the original source code. Here the authors present program obfuscation as a way to deter reverse engineering. In addition, they discuss the different types of obfuscation technologies available and demonstrate the new obfuscation tool that is included in Visual Studio .NET 2003.

Gabriel Torok and Bill Leach Editor's Note: Security and Whips

Last September, MSDN Magazine published its first-ever issue devoted entirely to the topic of security. Feedback was so positive that we thought "Hey! We can do this again next year!" It's this kind of forward thinking that's behind this year's security issue.

New Stuff: Resources for Your Developer Toolbox

Desaware is shipping CAS/Tester (automated code access security testing) for Visual Studio® . NET. CAS/Tester was developed to address the issue of security constraints as software is deployed through the Internet, intranets, and other distribution channels.

Theresa W. Carey Web Q&A: Secure Passwords, Nested XML, and More

Edited by Nancy Michell Data Points: SQL Server User-defined Functions

John Papa Cutting Edge: Custom Provider Controls

Dino Esposito The XML Files: XML in Microsoft Office Word 2003

Aaron Skonnard Advanced Basics: SQL Server Metadata

Ken Spencer The ASP Column: ATL Server Versus ASP.NET

George Shepherd Bugslayer: Google from Visual Studio .NET

John Robbins C++ Q&A: Menu Tips in an MFC App

Paul DiLascia Resource File: Threat Model Your Security Risks


Protect It: Safeguard Database Connection Strings and Other Sensitive Settings in Your Code

Protect It

Safeguard Database Connection Strings and Other Sensitive Settings in Your Code

Alek Davis
This article assumes you're familiar with C# and Visual Basic .NET
Level of Difficulty 1 2 3
SUMMARY

Protecting application secrets, such as database connection strings and passwords, requires careful consideration of a number of pertinent factors such as how sensitive the data is, who could gain access to it, how to balance security, performance, and maintainability, and so forth. This article explains the fundamentals of data protection and compares a variety of techniques that can be used to protect application settings. The author discusses what to avoid, such as hiding keys in source code and the use of Local Security Authority. In addition, he presents some effective solutions such as the Data Protection API.

Contents
Data Hiding
Restricting Access to Data
Encrypting Data
Weighing Your Options
Storing Encrypted Data
One-way and Two-way Encryption
Hashing Algorithms
Encryption Keys
Protecting Cryptographic Keys
Beware of Local Security Authority
Using DPAPI
Hiding Keys in the Application Source Code
Isolated Storage
Conclusion

T
he problem of protecting database connection strings, passwords, and other sensitive application settings is frequently discussed in developer newsgroups and online forums. While most security professionals agree that it is impossible to hide secrets using software, there are certain known techniques that can provide sufficient protection for most types of applications. All existing methods of data protection are based on three fundamental techniques: hiding, access control, and encryption. These techniques can be used individually or in combination. Whether you decide to protect application secrets by choosing an existing technology or by building your own data protection mechanism, your basic options will be limited to these three methods.

In this article I will discuss the fundamentals of data protection and compare several techniques that can help you better manage confidential data. I will identify the scenarios in which these techniques are appropriate and the times when they should be avoided. I will also provide references to code samples, tools, and APIs that can help you implement a data protection mechanism.

Data Hiding

Data hiding is sometimes referred to as security through obscurity. If you rely on this technique, you assume that only you know where sensitive information is stored and hope that nobody else will be able to figure it out. The caveat here is that the application must also know how to access the data, so being able to keep the secrets safe will largely depend on the ability of the application to protect this knowledge.

The most common places for hiding sensitive application settings include the application source code, configuration files, and the Windows® registry. Data can also be stored in such exotic locations as the IIS metabase, Universal Data Link (UDL) files, custom files, and elsewhere.

With few exceptions, simply hiding data offers little, if any, security. In most cases, the amount of work it requires for a hacker to discover your secrets is minimal. For example, an intruder may be able to discover the location of data by monitoring system changes performed by the running application (see Figure 1). This can be done with the help of utilities such as regmon, filemon, and diskmon from SysInternals.

Figure 1 Regmon
Figure 1 Regmon

The ease of decompiling .NET assemblies poses an even more serious threat. Using decompilers like Anakrino or Salamander, the application source code can be reverse engineered, exposing sensitive data or the application logic (see Figure 2).

Figure 2 Decompile
Figure 2 Decompile

Restricting Access to Data

If access to data is restricted, its location does not need to be kept secret. Instead, this technique relies on the technological features built into the operating system, which can prevent unauthorized entities from accessing data. Normally, access restrictions are based on the caller's identity or knowledge of a common secret (see Figure 3). Access Control Lists (ACLs), the Microsoft® Data Protection API (DPAPI), and isolated storage use this technique to protect data.

Figure 3 Access Control
Figure 3 Access Control

When applied correctly, restricted access can be one of the most efficient security measures, but unfortunately it has limitations. For example, access control based on user identity does not work well for applications running as anonymous users or under different user contexts. Another problem with identity-based access control is that it may not prevent different applications running under the same identity from accessing each other's data. Figure 4 lists common issues associated with identity-based access controls.

Figure 4 Problems with Identity-based Access Control (Close Figure 4)

Identity Type | Problems
User | Will not work for applications running as anonymous users, such as ASP.NET applications. Will not work for applications which can run under multiple identities. Requires data to be set and retrieved by the same user. Allows unrelated applications running under the same user identity to access each other's data.
Machine | Allows all applications running on the same machine to access each other's data. May lose data if the machine is rebuilt.
Assembly | Requires data to be set and retrieved from the same application. Only supported by code access security and isolated storage.

On the other hand, when access control is based on the knowledge of a secret, protecting this secret becomes a problem. Additional difficulties associated with restricted access depend on a particular implementation. For example, ACLs can be hard to manage in a large enterprise environment and protected storage may require administrative privileges on the part of the caller, thus violating the rule of least privilege.

Encrypting Data

When encryption is used, neither hiding nor controlling access to data are the primary goals. Since the information is encrypted, knowing where to find it and being able to access it does not necessarily make the information useful to an unauthorized person—unless the intruder finds the decryption key.

Even though it is a time-tested data protection technique, encryption does not offer a perfect solution, but merely transforms the problem of protecting data into a problem of protecting cryptographic keys. As with any other types of sensitive data, cryptographic keys are protected using the same techniques: hiding and access control.

When choosing or building a data protection mechanism, you should weigh its vulnerabilities against potential threats. This assessment may be easier if you think of your solution in terms of the basic data protection techniques already discussed. For example, if you store database connection information containing a user's SQL credentials in a UDL file (see Figure 5), your data security essentially relies on file permissions (ACLs), which are a form of restricted access. (To a certain degree, it also relies on data hiding, but in this particular case, data hiding is a very weak security measure because it takes little effort to find a UDL file.) While ACLs play a vital role in data protection, it is important to understand that if a hacker manages to get read access to the UDL file, your SQL credentials will be revealed.

Figure 5 Credentials in UDL File
Figure 5 Credentials in UDL File

Unless the potential threat is very low or your data is not very valuable, you should avoid solutions that use just one data protection technique. Normally, the more techniques the solution employs, the more secure it is.

Weighing Your Options

As has already been said by many security experts, the best method of protecting secrets is not keeping secrets in the first place. For example, if you want to protect database connection strings containing SQL credentials, consider not using SQL authentication. If your application can use Windows authentication to connect to a database server instead of SQL authentication, you will not need to worry about protecting SQL credentials at all.

Unfortunately, storing sensitive information cannot always be avoided. If you absolutely need to store confidential data, start your search for the right solution by analyzing your application and its security requirements. The following questions can help to determine which option will work best for you:

  • How valuable is the information you are protecting? What kind of damage can you expect to suffer if your data security is compromised?
  • What kind of users are you protecting your data from? Are you protecting data from opportunistic hackers such as employees who would love access to the information they are not supposed to see, but who will not intentionally attack the system? Or are you protecting data from motivated hackers who are willing to spend time, effort, and money to compromise the system? How far do you think a potential hacker would go to get your data?
  • Does your application absolutely need to know the unencrypted text of the sensitive data?
  • Are you protecting database connection strings, or do you also want to manage other types of data?
  • How many identities does the application accessing your data use? Does it run under a single user identity or does it use multiple user identities?
  • Which versions of the operating system is it necessary for the application to support?
  • What is more important, application performance or security? Will you be willing to sacrifice performance to achieve a more secure application?
  • What is the mechanism responsible for defining data? Is it done manually, using a text editor for example, or is it done programmatically? If the data is created programmatically, are the application settings being defined from the same application that uses them or from another application?
  • How many applications in your organization or company use data protection? If it's more than one application, do they all have the same requirements? Do you want to reuse a single solution, which satisfies the requirements of all applications, or do you want to have a different data protection mechanism for each of your applications?
  • When your application is deployed, how hard will it be for support personnel to manage its security? Will the team supporting your application accept your solution?

Depending on the answers to these questions, you may want to choose one approach over another. Later in this article, I will look at several data protection technologies which can be used for the most common types of applications, but first I will outline certain options you should avoid under any circumstances.

All methods of storing sensitive data in plain text are considered insecure because they can expose data to anyone who manages to get read access to the data source—a task that may be easier than you think. You should avoid keeping unencrypted data in the Windows registry, configuration files, the COM+ catalog, isolated storage, custom files, the IIS metabase, and UDL files. Moreover, you should never, ever store sensitive data in plain text.

You may never achieve absolute security, but sufficient security is definitely feasible. Some applications have security requirements that call for complex and unorthodox solutions, especially in areas like banking, military secrets, law enforcement records, and scientific research. Most, however, entail the following three steps: encrypting data, restricting access to encrypted data using ACLs, and hiding and restricting access to encryption keys.

When implementing data protection, you may consider other security aspects such as data tampering, but since data tampering does not affect secrecy, I will not be discussing it in this article. Instead, I will cover the more practical issues of where to store your encrypted data, which type of encryption and cryptographic algorithm to use, and how to protect your cryptographic keys.

Storing Encrypted Data

Unless you have a compelling reason to do otherwise, store encrypted data in the application configuration files or Windows registry. The main advantages of both options include the ability to protect data using ACLs and the ease of programmatic access. Any other type of data storage is likely to lack at least one of these advantages and even though it might add another benefit, it is unlikely to justify the extra implementation effort.

Although some developers insist that Microsoft .NET Framework-based applications should never use the registry, it is not written in stone. The main shortcoming of the registry is that it does not fit well in the XCOPY deployment scenario, but since many applications these days are deployed with the help of setup programs and require certain configuration steps to be performed on the systems where they are installed, XCOPY compliance may not be necessary. Another drawback of the registry is that it is specific to the Windows platform and may not work with other operating systems. This limitation will only become a major problem when platforms other than Windows start supporting the common language implementation (CLI).

If you care about XCOPY deployment or interoperability with platforms other than Windows, use configuration files; otherwise, select the option that best fits your needs. If you have to store settings that are shared by several applications, or define these settings programmatically, you may find the registry easier to manage. Application-specific settings, which are not changed programmatically, can be stored in configuration files. In either case, do not forget to apply appropriate ACLs to the data store. For example, it would be a good idea to assign the read-access rights on a file or registry key holding sensitive data to the user account under which the application runs. Similarly, write-access should be explicitly assigned to the user—or a group of users (such as Administrators)—allowed to modify data. Everybody else should be denied all access rights.

One-way and Two-way Encryption

There are two methods of encrypting data: one-way (commonly called hashing) and two-way. Technically, hashing is not encryption, but because both techniques can address data protection in a similar manner—that is, by transforming plain text data into ciphertext—I will treat them as logical counterparts.

While hashing can serve other purposes, in the area of data protection it offers certain advantages over two-way encryption. First, hashing is a bit easier to use. Second, because hashing does not use encryption keys, it eliminates the key management problem, which will be discussed later in this article. The main disadvantage of hashing is that it does not support decryption, but in a few cases this problem can be a blessing in disguise.

Password-based authentication is a typical case in which hashing is more appropriate than two-way encryption. If your application keeps passwords for authentication purposes only, do not encrypt them with a symmetric or public key, but store their hashed values instead. When a user logs in, instead of decrypting and comparing plain text passwords, the application can compare password hashes. To reduce the risk of dictionary attacks, always use salt values with hashes. A salt value is random data that is appended to the plain text before it is hashed and stored along with the hash so it can be used later when another plain text value is compared to the hash. To see an example of using salted hashes for password-based authentication, check out the Security Briefs column in the August 2003 issue of MSDN® Magazine.

Hashing Algorithms

MD5 and SHA-1 are the most popular hashing algorithms. SHA-1 hashes are 160-bits long, while MD5 hashes are 128-bits long. The SHA-1 algorithm is a little slower, but more secure than MD5. In addition to MD5 and SHA-1, the .NET Framework provides support for 256, 384, and 512-bit versions of the SHA algorithm, which should be even more secure, though probably slower.

The easiest way to hash data is by calling the HashPasswordForStoringInConfigFile method of the FormsAuthentication class, as in the following example:

using System.Web.Security;
•••
string base64HashValue =
    FormsAuthentication.HashPasswordForStoringInConfigFile
        ("mypassword", "sha1");
The Loaded User Profile

Even though Microsoft documentation lists a loaded user profile as one of the requirements of an application that wants to use DPAPI with a user store, it does not define exactly what a loaded user profile is. If you are puzzled by this term, you can think of it as the profile of an interactive user, which is created when the user first logs onto the system. All applications launched by users interactively run with the loaded profile of the interactive user. Noninteractive applications, such as Windows services configured to run as LocalSystem, cannot load a user profile, and therefore cannot use DPAPI with a user store. The same is true for system processes, such as the ASP.NET process, which run as built-in system accounts. To allow Windows services to use DPAPI with a user store, they must be configured to run as a local or domain user with an already created profile. In practical terms, this means that the user must log onto the system interactively at least once before the application can make DPAPI calls on behalf of the user.

Unfortunately, this method only supports MD5 and SHA-1 hashing algorithms, so if you want to use SHA-256, SHA-384, or SHA-512 hashes, you will need to write a few more lines of code. An example at How To: Hash Data with Salt explains how to generate and compare hashes using different hashing algorithms.

Encryption Keys

If your application needs to know the text values of sensitive data, you cannot use hashing. In this case, use two-way encryption with either symmetric or public-private keys. If you are not sure which type of encryption keys to choose, use symmetric keys. The main disadvantage of public-key encryption is slow performance, which can be in the range of 1,000 times worse than symmetric-key encryption. Public-key encryption also imposes certain limitations on the size of the text-based data that can be encrypted. While public-key encryption can be used for data protection, practically anything you want to do using public keys can be accomplished with the help of other technologies. Because public keys are best suited for secure key exchange and digital data signing, I will omit them from this discussion and concentrate instead on symmetric-key encryption.

When choosing an encryption algorithm, it makes sense to select the most secure algorithm with the longest possible key. Out of all symmetric-key algorithms supported by the .NET Framework, the U.S. government-approved Rijndael algorithm (also referred to as the Advanced Encryption Standard, or AES, algorithm) is considered the most secure. This algorithm supports 128, 192, and 256-bit keys. See the article "Encrypt It: Keep Your Data Secure With the New Advanced Encryption Standard" by James McCaffrey in this issue for more information about Rijndael.

The Rijndael algorithm has another advantage over other symmetric-key algorithms supported by the .NET Framework. While other algorithms are offered in the form of thin .NET Framework wrapper classes over existing CryptoAPI modules, Rijndael (implemented as the RijndaelManaged class) is written completely in managed code. Note that some developers consider this a disadvantage and prefer to use an unmanaged implementation of the Rijndael algorithm to achieve better performance.

Unfortunately, this implementation of the Rijndael algorithm is only supported on Windows XP or later and on the systems with the .NET Framework installed. If your application needs to be compatible with unmanaged applications running on Windows 2000 or earlier, use Triple DES, which is a better version of the less secure DES algorithm. The Triple DES algorithm supports 112 and 168-bit keys, though 168-bit keys are recommended. Due to inconsistencies in the way the parity bits of the key are treated, 168-bit Triple DES keys are sometimes referred to as 192-bit keys. If you want to allow a managed and an unmanaged application to encrypt or decrypt data with the same Triple DES key, be aware that the pass phrase (password) from which the key is derived should only contain printable ASCII characters; otherwise, the generated keys will not match (this is due to a limitation of the .NET implementation of the Triple DES algorithm).

Original DES, RC2, and RC4 algorithms are generally considered less secure than Rijndael and Triple DES and therefore should be avoided. Under no circumstances should homegrown encryption algorithms be used. Never assume that if you are the original author of code, nobody will be able to crack it.

Figure 6 Data Encryption Schemes
Figure 6 Data Encryption Schemes

Figure 6 describes the various data encryption schemes that you can use.

Protecting Cryptographic Keys

When using symmetric keys to encrypt and decrypt application data, it is important to be able to generate the same key over time. This poses the problem of protecting keys or key generation logic (not to be confused with encryption algorithms). While several techniques can be used to accomplish this task, they all have shortcomings. Let's look at the available options and the scenarios for which they are most appropriate.

Persistent symmetric keys can be generated in two ways: by defining the key yourself or by letting the operating system do it for you. When choosing the first option, you can hardcode the actual key bytes in the application source code or implement a consistent logic to derive the key from certain unchangeable characteristics. These characteristics normally include a pass phrase, from which the key is derived, and may also include other values such as an initialization vector. Whether hardcoding the key bytes or implementing the key generation logic, you are essentially hiding the key in the application source code. This option gives you more control and flexibility, but it also puts your data at greater risk if your application source code is reverse engineered. Alternatively, you can store the key in a persistent location, such as a file or the registry, and protect this location by ACL. However, because this approach is prone to maintenance errors, I don't recommend it.

A more secure, but also more restrictive way of generating the key is to let the operating system do it for you. This can be done using such operating system features as Local Security Authority (LSA) or DPAPI.

Beware of Local Security Authority

During the heyday of Windows NT® 4.0, the LSA Policy functions LsaStorePrivateData and LsaRetrievePrivateData provided a reasonably secure way of protecting application secrets. Although the LSA Policy functions are available on Windows 2000 and later (and still used for protecting such settings as passwords defined for Windows services), Microsoft does not recommend them, so I am only mentioning these functions for completeness and to explain why they should be avoided.

One problem with the LSA Policy functions is that along with key management and encryption they handle data storage using the protected area of the Windows registry. This may seem like a good thing, but it is not because the amount of storage available to the LSA Policy functions is limited to 4096 slots, half of which are already taken by the system. If applications keep using the LSA Policy for storing sensitive data, they will be at risk of running out of space. Next, because only highly privileged users can call the LSA Policy functions, they will not work for applications running under unprivileged accounts, such as ASP.NET. What's worse, there are tools such as LSADUMP2, which can reveal the LSA secrets. The bottom line: do not use the LSA Policy functions for data protection.

Using DPAPI

As an alternative to the LSA Policy functions, Microsoft recommends using a subset of CryptoAPI called DPAPI. DPAPI includes two functions which can be used for data protection: CryptProtectData and CryptUnprotectData. These functions are implemented in crypt32.dll and can also be called from .NET Framework-based applications via P/Invoke. DPAPI is part of the operating system and is available on Windows 2000 and later.

Unlike the LSA functions, DPAPI does not handle data storage, but it can generate machine- or user-specific keys to encrypt and decrypt data. To differentiate between the two types of keys, DPAPI documentation refers to them as machine store and user store.

Machine-specific keys are generally not safe because they can be used by anyone who gains access to the system. It is possible to generate a more secure version of a machine-specific key by passing a password (also known as secondary entropy) to a DPAPI function. This option is better because it requires a caller to know the value of this password, but it also creates the challenge of storing the password, which brings us back to the original problem. Another issue with machine-specific keys is that they are not guaranteed to be the same if a change in the environment occurs. Such changes as moving the application to another computer or rebuilding the system can risk breaking the application using DPAPI with machine-specific keys.

While user-specific keys are considered more secure, they have the same limitations as any other type of identity-based access control. First, because user keys can only be generated by programs running with loaded user profiles, they will not work with ASP.NET applications and applications running as certain built-in accounts. While there is a way to overcome this limitation by using serviced components, it comes at a cost of greater complexity and degraded application performance. It also requires a serviced component to run under a privileged account, which violates the principle of least privilege. If you decide to use DPAPI with user-specific keys, you must also be aware that only the user who encrypts data can decrypt the results. Obviously, this will not work for applications that can run under different user accounts or whose settings can be defined by different users. Another problem with user-specific keys is that they allow all applications running under the same user profile to access each other's data, which can be a potential security breach. As with DPAPI using a machine store, this problem can be also addressed by requiring the caller to provide secondary entropy, but as in the previous example it will pose a problem of storing this entropy. Unfortunately, DPAPI does not allow you to use machine and user-specific keys at the same time (on the same CryptProtectData call).

DPAPI is recommended for applications that run under the single user account with a loaded user profile and whose settings are defined by the same user. (For more information, see the sidebar "The Loaded User Profile.") A typical example would be a Windows service application running under an account of a local or domain user. If your application satisfies these requirements, it should use DPAPI with user-specific keys. For other applications, you can use DPAPI with machine-specific keys protected by secondary entropy (see "Use DPAPI (Machine Store) from ASP.NET" or the sample at Use DPAPI to Encrypt and Decrypt Data). You can define this entropy in the application source code and obfuscate the application binary so the entropy cannot be easily found. If you choose this approach, however, be aware that even obfuscated code can be reverse engineered. To find out what obfuscation can and cannot do for you, read the article "The Premier Obfuscator for Microsoft .NET Applications!" by Brent Rector at and "Obfuscate It: Thwart Reverse Engineering Attempts on Your Visual Basic .NET or C# Code" by Gabriel Torok and Bill Leach in this issue.

If your application cannot use DPAPI with user-specific keys directly and cannot tolerate the risk of reverse engineering, consider implementing DPAPI with a serviced component, but be aware that this option will affect your application's performance. (see "Use DPAPI (User Store) from ASP.NET with Enterprise Services"). If you choose this option, be prepared to implement an authorization scheme to prevent malicious applications and users from calling the serviced component, which may not be a simple task. To see how this issue can be addressed, take a look at the CipherSafe.NET tool and the accompanying documentation; this application illustrates the use of DPAPI for sensitive application management and offers an interesting approach to solving the authorization problem.

Hiding Keys in the Application Source Code

While security professionals keep urging application developers to stop "hiding" cryptographic keys (and other sensitive data) in their application source code, application developers continue ignoring this advice. The explanation of this resentment can be twofold. One obviously inexcusable reason why developers choose this approach is that it is very easy to implement. The other explanation is that the alternative options may not work in some situations. For example, consider a scenario in which several applications (say Windows services) running on different computers have to use the same key for encrypting and decrypting data stored in a database. In this situation, using DPAPI for encryption is not practical, so developers may choose to embed the key in the application source code. This would be easy, but not secure. A better approach would be to require an administrator to define the encryption key at application installation, encrypt its value via DPAPI with a user-specific key of the account under which the application runs, and save the encrypted value in the Windows registry or the application configuration file. Be aware, though, that while this method is a more secure technical implementation, it poses procedural challenges.

Another situation in which embedding the key in the application source code may be the only option is when the application data owners have limited or no access to the machines hosting their applications. A Web hosting environment is one of the obvious examples. If embedding the key in the application source code is your only option, you must realize the associated risks and address them accordingly.

The major risk in this case comes from reverse engineering and at this time it can only be addressed by obfuscation. Obfuscation does not make reverse engineering impossible, but it can make the process expensive and time consuming. Compare Figure 7, which shows a decompiled assembly obfuscated by a Demeanor, to Figure 3. Because all nonpublic symbols in the obfuscated assembly are renamed using unprintable characters and the strings are encrypted, reconstructing application logic from the assembly may be next to impossible. Commercial decompilers, such as Salamander, can make the job of reverse engineering easier by converting nonprintable characters to their printable equivalents and converting application classes into source files, but they cost money and still require a hacker to make sense of the unreadable symbols (this is where spaghetti code could actually help you!).

Figure 7 Decompile of Obfuscated Assemblies
Figure 7 Decompile of Obfuscated Assemblies

If you define the key in the application, in addition to obfuscating the assembly, try not to store the actual key bytes in the source code. Instead, implement key-generation logic using persistent characteristics, such as the encryption algorithm, key size, pass phrase, initialization vector, and salt (see an example at Encrypt and Decrypt Data Using a Symmetric (Rijndael) Key). This will introduce an extra layer of indirection, so the key will not be accessible by simply dumping the symbols from the application binary. As long as you do not change key-generation logic and key characteristics, the resulting key is guaranteed to be the same. It may also be a good idea not to use static strings as key-generation characteristics, but rather build them on the fly. Another suggestion would be to treat the assembly the same way as the data store should be treated, that is, by applying the appropriate ACLs. And only use this option as a last resort, when none of the other data protection techniques work and your only alternative is leaving sensitive data unencrypted.

Isolated Storage

Isolated storage is sometimes mentioned as a method of protecting data. It lets you restrict data access to the application assembly. While this option can be helpful, isolated storage is not recommended for protecting sensitive data because it does not use encryption; it is better suited for storing user-specific application settings. To store sensitive user-specific application data, use DPAPI along with isolated storage.

Conclusion

When it comes to protecting sensitive application data, there are no perfect solutions. None of the existing software technologies can guarantee absolute security. The goal is to pick the best, or "least worst" option that will work for your application and comply with your security requirements. Understanding how the various techniques and technologies provide data protection will help you assess vulnerabilities and prevent you from succumbing to a false sense of security.

Whether you decide to build or buy a data protection solution, make sure that it uses strong encryption, provides reasonable protection for cryptographic keys, and restricts access to encrypted data. While no software solution will make your application data absolutely safe, the right approach can help you avoid the most common types of security breaches.

For related articles see:
Data Access Security
Security: Protect Private Data with the Cryptography Namespaces of the .NET Framework
Windows Data Protection

For background information see:
Create an Encryption Library
Store an Encrypted Connection String in the Registry
Writing Secure Code

Alek Davis is a senior developer of a security engineering team at Intel Corporation in Folsom, CA. He has experience in developing apps for Windows with a focus on enterprise application security. Alek has a B.S. degree in Computer Science and an M.S. degree in Software Engineering, both from the California State University, Sacramento.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Encrypt It: Keep Your Data Secure with the New Advanced Encryption Standard

Encrypt It

Keep Your Data Secure with the New Advanced Encryption Standard

James McCaffrey
This article assumes you're familiar with C# and bit manipulation
Level of Difficulty 1 2 3
Code download available at: AES.exe (143KB)
SUMMARY

The Advanced Encryption Standard (AES) is a National Institute of Standards and Technology specification for the encryption of electronic data. It is expected to become the accepted means of encrypting digital information, including financial, telecommunications, and government data. This article presents an overview of AES and explains the algorithms it uses. Included is a complete C# implementation and examples of encrypting .NET data. After reading this article you will be able to encrypt data using AES, test AES-based software, and use AES encryption in your systems.

Contents
Overview of the AES Algorithm
Field Addition and Multiplication in GF(28)
Key Expansion
The AES Class Constructor in C#
The AES Cipher Method in C#
The AES InvCipher Method in C#
Using the AES Class
Implementation Alternatives
Conclusion

T
he National Institute of Standards and Technology (NIST) established the new Advanced Encryption Standard (AES) specification on May 26, 2002. In this article I will provide a working implementation of AES written in C#, and a complete explanation of exactly what AES is and how the code works. I'll show you how to encrypt data using AES and extend the code given here to develop a commercial-quality AES class. I'll also explain how and why to incorporate AES encryption into your software systems, and how to test AES-based software.

Note that the code presented in this article and any other implementation based on this article is subject to applicable Federal cryptographic module export controls (see Commercial Encryption Export Controls for the exact regulations).

AES is a new cryptographic algorithm that can be used to protect electronic data. Specifically, AES is an iterative, symmetric-key block cipher that can use keys of 128, 192, and 256 bits, and encrypts and decrypts data in blocks of 128 bits (16 bytes). Unlike public-key ciphers, which use a pair of keys, symmetric-key ciphers use the same key to encrypt and decrypt data. Encrypted data returned by block ciphers have the same number of bits that the input data had. Iterative ciphers use a loop structure that repeatedly performs permutations and substitutions of the input data. Figure 1 shows AES in action encrypting and then decrypting a 16-byte block of data using a 192-bit key.

Figure 1 Some Data
Figure 1 Some Data

AES is the successor to the older Data Encryption Standard (DES). DES was approved as a Federal standard in 1977 and remained viable until 1998 when a combination of advances in hardware, software, and cryptanalysis theory allowed a DES-encrypted message to be decrypted in 56 hours. Since that time numerous other successful attacks on DES-encrypted data have been made and DES is now considered past its useful lifetime.

In late 1999, the Rijndael (pronounced "rain doll") algorithm, created by researchers Joan Daemen and Vincent Rijmen, was selected by the NIST as the proposal that best met the design criteria of security, implementation efficiency, versatility, and simplicity. Although the terms AES and Rijndael are sometimes used interchangeably, they are distinct. AES is widely expected to become the de facto standard for encrypting all forms of electronic data including data used in commercial applications such as banking and financial transactions, telecommunications, and private and Federal information.

Overview of the AES Algorithm

The AES algorithm is based on permutations and substitutions. Permutations are rearrangements of data, and substitutions replace one unit of data with another. AES performs permutations and substitutions using several different techniques. To illustrate these techniques, let's walk through a concrete example of AES encryption using the data shown in Figure 1.

The following is the 128-bit value that you will encrypt with the indexes array:

00 11 22 33 44 55 66 77 88 99 aa bb cc dd ee ff
0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15
The 192-bit key value is:
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17
0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21 22 23

Figure 2 Sbox
Figure 2 Sbox

When the AES constructor is called, two tables that will be used by the encryption method are initialized. The first table is a substitution box named Sbox. It is a 16 × 16 matrix. The first five rows and columns of Sbox are shown in Figure 2. Behind the scenes, the encryption routine takes the key array and uses it to generate a "key schedule" table named w[], shown in Figure 3.

Figure 3 Key Sched.
Figure 3 Key Sched.

The first Nk (6) rows of w[] are seeded with the original key value (0x00 through 0x17) and the remaining rows are generated from the seed key. The variable Nk represents the size of the seed key in 32-bit words. You'll see exactly how w[] is generated later when I examine the AES implementation. The point is that there are now many keys to use instead of just one. These new keys are called the round keys to distinguish them from the original seed key.

Figure 4 State
Figure 4 State

The AES encryption routine begins by copying the 16-byte input array into a 4×4 byte matrix named State (see Figure 4). The AES encryption algorithm is named Cipher and operates on State[] and can be described in pseudocode (see Figure 5).

Figure 5 Cipher Algorithm Pseudocode (Close Figure 5)
Cipher(byte[] input, byte[] output)
{
  byte[4,4] State;
  copy input[] into State[]
  AddRoundKey
  for (round = 1; round < Nr-1; ++round)
  {
    SubBytes
    ShiftRows
    MixColumns
    AddRoundKey
  }
  SubBytes
  ShiftRows
  AddRoundKey
  copy State[] to output[]
}

The encryption algorithm performs a preliminary processing step that's called AddRoundKey in the specification. AddRoundKey performs a byte-by-byte XOR operation on the State matrix using the first four rows of the key schedule, and XORs input State[r,c] with round keys table w[c,r].

For example, if the first row of the State matrix holds the bytes { 00, 44, 88, cc }, and the first column of the key schedule is { 00, 04, 08, 0c }, then the new value of State[0,2] is the result of XORing State[0,2] (0x88) with w[2,0] (0x08), or 0x80:

1 0 0 0 1 0 0 0
0 0 0 0 1 0 0 0  XOR

1 0 0 0 0 0 0 0

The main loop of the AES encryption algorithm performs four different operations on the State matrix, called SubBytes, ShiftRows, MixColumns, and AddRoundKey in the specification. The AddRoundKey operation is the same as the preliminary AddRoundKey except that each time AddRoundKey is called, the next four rows of the key schedule are used. The SubBytes routine is a substitution operation that takes each byte in the State matrix and substitutes a new byte determined by the Sbox table. For example, if the value of State[0,1] is 0x40 and you want to find its substitute, you take the value at State[0,1] (0x40) and let x equal the left digit (4) and y equal the right digit (0). Then you use x and y as indexes into the Sbox table to find the substitution value, as shown in Figure 2.

ShiftRows is a permutation operation that rotates bytes in the State matrix to the left. Figure 6 shows how ShiftRows works on State[]. Row 0 of State is rotated 0 positions to the left, row 1 is rotated 1 position left, row 2 is rotated 2 positions left, and row 3 is rotated 3 positions left.

Figure 6 Running ShiftRows on State
Figure 6 Running ShiftRows on State

The MixColumns operation is a substitution operation that is the trickiest part of the AES algorithm to understand. It replaces each byte with the result of mathematical field additions and multiplications of values in the byte's column. I will explain the details of special field addition and multiplication in the next section.

Suppose the value at State[0,1] is 0x09, and the other values in column 1 are 0x60, 0xe1, and 0x04; then the new value for State[0,1] is shown in the following:

State[0,1] = (State[0,1] * 0x01) +
             (State[1,1] * 0x02) +
             (State[2,1] * 0x03) +
             (State[3,1] * 0x01)

           = (0x09 * 0x01) + (0x60 * 0x02) + (0xe1 * 0x03) +
             (0x04 * 0x01)

           = 0x57
The addition and multiplication are special mathematical field operations, not the usual addition and multiplication on integers.

The four operations SubBytes, ShiftRows, MixColumns, and AddRoundKey are called inside a loop that executes Nr times—the number of rounds for a given key size, less 1. The number of rounds that the encryption algorithm uses is either 10, 12, or 14 and depends on whether the seed key size is 128, 192, or 256 bits. In this example, because Nr equals 12, the four operations are called 11 times. After this iteration completes, the encryption algorithm finishes by calling SubBytes, ShiftRows, and AddRoundKey before copying the State matrix to the output parameter.

In summary, there are four operations that are at the heart of the AES encryption algorithm. AddRoundKey substitutes groups of 4 bytes using round keys generated from the seed key value. SubBytes substitutes individual bytes using a substitution table. ShiftRows permutes groups of 4 bytes by rotating 4-byte rows. MixColumns substitutes bytes using a combination of both field addition and multiplication.

Field Addition and Multiplication in GF(28)

As you've seen, the AES encryption algorithm uses fairly straightforward techniques for substitution and permutation, except for the MixColumns routine. The MixColumns routine uses special addition and multiplication. The addition and multiplication used by AES are based on mathematical field theory. In particular, AES is based on a field called GF(28).

The GF(28) field consists of a set of 256 values from 0x00 to 0xff, plus addition and multiplication, hence the (28). GF stands for Galois Field, named after the mathematician who founded field theory. One of the characteristics of GF(28) is that the result of an addition or multiplication operation must be in the set {0x00 ... 0xff}. Although the theory of fields is rather deep, the net result for GF(28) addition is simple: GF(28) addition is just the XOR operation.

Multiplication in GF(28) is trickier, however. As you'll see later in the C# implementation, the AES encryption and decryption routines need to know how to multiply by only the seven constants 0x01, 0x02, 0x03, 0x09, 0x0b, 0x0d, and 0x0e. So instead of explaining GF(28) multiplication theory in general, I will explain it just for these seven specific cases.

Multiplication by 0x01 in GF(28) is special; it corresponds to multiplication by 1 in normal arithmetic and works the same way—any value times 0x01 equals itself.

Now let's look at multiplication by 0x02. As in the case of addition, the theory is deep, but the net result is fairly simple. If the value being multiplied is less than 0x80, then the result of multiplication is just the value left-shifted 1 bit position. If the value being multiplied is greater than or equal to 0x80, then the result of multiplication is the value left-shifted 1 bit position XORed with the value 0x1b. This prevents "field overflow" and keeps the product of the multiplication in range.

Once you've established addition and multiplication by 0x02 in GF(28), you can use them to define multiplication by any constant. To multiply by 0x03 in GF(28), you can decompose 0x03 as powers of 2 and additions. To multiply an arbitrary byte b by 0x03, observe that 0x03 = 0x02 + 0x01. Thus:

b * 0x03 = b * (0x02 + 0x01)
         = (b * 0x02) + (b * 0x01)
This can be done because you know how to multiply by 0x02 and 0x01 and how to perform addition. Similarly, to multiply an arbitrary byte b by 0x0d, you do this:
b * 0x0d = b * (0x08 + 0x04 + 0x01)
         = (b * 0x08) + (b * 0x04) + (b * 0x01)
         = (b * 0x02 * 0x02 * 0x02) + (b * 0x02 * 0x02) + (b * 0x01)

The other multiplications needed for the AES MixColumns routine in the encryption and decryption algorithm follow the same general pattern, as shown here:

b * 0x09 = b * (0x08 + 0x01)
         = (b * 0x02 * 0x02 * 0x02) + (b * 0x01)

b * 0x0b = b * (0x08 + 0x02 + 0x01)
         = (b * 0x02 * 0x02 * 0x02) + (b * 0x02) + (b * 0x01)

b * 0x0e = b * (0x08 + 0x04 + 0x02)
         = (b * 0x02 * 0x02 * 0x02) + (b * 0x02 * 0x02) + (b * 0x02)

To summarize, addition in GF(28) is the XOR operation. Multiplication in GF(28) reduces to additions and multiplications by 0x02, where multiplication by 0x02 is a conditional 1-bit left shift. The AES specification contains a lot of additional information about operations in GF(28).

Key Expansion

The AES encryption and decryption algorithms use a key schedule generated from the seed key array of bytes. The AES specification refers to this as the KeyExpansion routine. Generating, in essence, multiple keys from an initial key instead of using a single key greatly increases the diffusion of bits. Although not overwhelmingly difficult, understanding KeyExpansion is one of the trickier parts of the AES algorithm. In high-level pseudocode, the KeyExpansion routine looks like the following:

KeyExpansion(byte[] key, byte[][4] w)
{
  copy the seed key into the first rows of w

  for each remaining row of w
  {
    use two of the previous rows to create a new row
  }
}

The "use two of the previous rows to create a new row" routine makes use of two subroutines, RotWord and SubWord, and a table of constants named Rcon (for "round constants"). Let's look at each of these three items and then come back to the KeyExpansion routine as a whole.

The RotWord routine is simple. It accepts an array of 4 bytes and rotates them 1 position left. Because the round schedule table w[] has four columns, RotWord rotates a row of w[] to the left. Notice that the RotWord function used by KeyExpansion is very similar to the ShiftRows routine used by the encryption algorithm except that it works on a single row of the key schedule w[] instead of the entire encryption state table State[].

The SubWord routine performs a byte-by-byte substitution on a given row of the key schedule table w[] using the substitution table Sbox. The substitutions in KeyExpansion operate exactly like those in the encryption algorithm. The input byte to be substituted is separated into an (x,y) pair which are used as indexes into the substitution table Sbox. For example, substitution for 0x27 results in x = 2 and y = 7, and Sbox[2,7] returns 0xcc.

The KeyExpansion routine uses an array Rcon[], called the round constant table. These constants are 4 bytes each to match with a row of the key schedule table. The AES KeyExpansion routine requires 11 round constants. You can see these constants listed in Figure 7.

Figure 7 Initializing Rcon (Close Figure 7)
private void BuildRcon()
{
  this.Rcon = new byte[11,4] { {0x00, 0x00, 0x00, 0x00},
                               {0x01, 0x00, 0x00, 0x00},
                               {0x02, 0x00, 0x00, 0x00},
                               {0x04, 0x00, 0x00, 0x00},
                               {0x08, 0x00, 0x00, 0x00},
                               {0x10, 0x00, 0x00, 0x00},
                               {0x20, 0x00, 0x00, 0x00},
                               {0x40, 0x00, 0x00, 0x00},
                               {0x80, 0x00, 0x00, 0x00},
                               {0x1b, 0x00, 0x00, 0x00},
                               {0x36, 0x00, 0x00, 0x00} };
}  // BuildRcon()

The leftmost byte of each round constant is a power of 2 in the GF(28) field. Another way of looking at it is to observe that each value is the previous value times 0x02, as described in the previous section discussing multiplication in GF(28). Notice that 0x80 × 0x02 = 0x1b is 0x80 left-shifted 1 bit followed by an XOR with 0x1b, as described earlier.

Now let's take a closer look at the loop inside KeyExpansion. In more detailed pseudocode than before, the loop is:

for (row = Nk; row < (4 * Nr+1); ++row)
{
  temp = w[row-1]

  if (row % Nk == 0)
    temp = SubWord(RotWord(temp)) xor Rcon[row/Nk]
  else if (Nk == 8 and row % Nk == 4)
    temp = SubWord(temp)

  w[row] = w[row-Nk] xor temp
}

Ignoring the if clause for a moment, you'll see that each row of the key schedule table w[] is the result of XORing the previous row with the row Nk (4, 6, or 8 depending on the key size) rows before. The first part of the if conditional modifies every fourth, sixth, or eighth row of the key schedule with SubWord, RotWord, and XORing with a round constant, depending on whether the key size is 128, 192, or 256 bits. The second part of the conditional will modify rows 12, 20, 28 and so on—every eighth row—for a 256-bit key to add additional variability to the key schedule.

Let's see how KeyExpansion gets started with the example presented at the beginning of this article. The seed key is the 192-bit / 6-word value:

00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17

The key schedule byte table w[] has the dimensions 4 columns and Nb × (Nr + 1) equals 4 × (12 + 1), or 52 rows. The KeyExpansion routine copies the values in the seed key into the first rows of the key schedule byte table w[]. Because my seed key is 192 bits (24 bytes), and the w[] table always has 4 columns, in this case KeyExapansion copies the seed key into the first 6 rows of w[]. Now let's see how the KeyExpansion routine fills the rest of the key schedule table. In my example, the first calculated row is row 6 because rows 0 to 5 were filled with the seed key values:

temp = w[row-1] = 14 15 16 17
The condition (row % Nk == 0) is true, so first the RotWord subroutine is applied:
temp = 15 16 17 14
Then SubWord is applied:
temp = 59 47 f0 fa
Then XORed with Rcon[row / Nk] = Rcon[6 / 6] = 01 00 00 00:
temp = 58 47 f0 fa
This is then XORed with w[row-Nk] = w[6-6] = 00 01 02 03, yielding the following result:
w[6] = 58 46 f2 f9
The process repeats itself for all of the remaining rows in key schedule table w[].

To summarize, an important part of AES encryption and decryption is the generation of multiple round keys from the initial seed key. This KeyExpansion algorithm generates a key schedule and uses substitution and permutation in a way that is similar in most respects to the encryption and decryption algorithms.

The AES Class Constructor in C#

Now that I've examined all the components of the AES encryption algorithm, I'll implement it in C#. The official specification of the AES algorithm is contained in Federal Information Processing Standards Publication 197. I decided to base my implementation on it as closely as possible, but I quickly discovered that the specification was more of a theory document than an implementation guide. To exploit the official specification as a resource, I have used the same variable names as those used in the standards publication (even when they are rather cryptic like "Nr" and "w").

My design uses the nine data members and one enumeration type as shown here:

public enum KeySize { Bits128, Bits192,
                      Bits256 };

private int Nb;
private int Nk;
private int Nr;

private byte[] key;
private byte[,] Sbox;
private byte[,] iSbox;
private byte[,] w;
private byte[,] Rcon;
private byte[,] State;
Because the key size can only be 128, 192, or 256 bits, it is a great candidate for an enumerated type:
public enum KeySize { Bits128, Bits192, Bits256 };

The specification document generally uses bytes as its basic storage unit but uses 4-byte words as the size for two important data members. The two members Nb and Nk represent the block size in words and key size in words, respectively. Nr represents the number of rounds. The block size is always 16 bytes (or 128 bits, which is 4 words for AES), so it could have been declared as a constant. The key size is assigned a value of 4, 6, or 8 according to the value of the enumeration parameter KeySize. The AES algorithm iterates through a number of rounds to increase the complexity of the encrypted data. The number of rounds is either 10, 12, or 14 and is based on cryptanalysis theory. It depends directly on the key size.

When designing a class interface, I like to work backwards. I imagine calling the constructor and methods from an application. Using this approach, I decided that I wanted to instantiate an AES object like the following:

Aes a = new Aes(the key size, the seed key)
I called the encryption and decryption routines as follows:
a.Cipher(plainText, cipherText);
a.InvCipher(cipherText, decipheredText);

I chose the slightly awkward method names Cipher and InvCipher because they are used in the AES specification document. Here is the code for the AES class constructor:

public Aes(KeySize keySize, byte[] keyBytes)
{
  SetNbNkNr(keySize);

  this.key = new byte[this.Nk * 4];
  keyBytes.CopyTo(this.key, 0);

  BuildSbox();
  BuildInvSbox();
  BuildRcon();
  KeyExpansion();

}

The constructor first set the values of Nb, Nk, and Nr by calling a helper method SetNbNkNr, which is shown in Figure 8. If efficiency is a concern, you could put this code directly in the constructor to avoid the overhead of a method call.

Figure 8 SetNbNkNr Method (Close Figure 8)
private void SetNbNkNr(KeySize keySize)
{
  this.Nb = 4;

  if (keySize == KeySize.Bits128)
  {
    this.Nk = 4;
    this.Nr = 10;
  }
  else if (keySize == KeySize.Bits192)
  {
    this.Nk = 6;
    this.Nr = 12;
  }
  else if (keySize == KeySize.Bits256)
  {
    this.Nk = 8;
    this.Nr = 14;
  }
}  // SetNbNkNr()

Next, you have to copy the bytes that are passed into the constructor into the class field variable. The key is declared with the other class fields and it gets its value by doing this:

this.key = new byte[this.Nk * 4];
keyBytes.CopyTo(this.key, 0);
I decided to call the initialization of the substitution tables Sbox[] and iSbox[] using the private helper methods BuildSbox and BuildInvSbox in the constructor. Now Sbox[] and iSbox[] are required by the key expansion routine and the Cipher and InvCipher methods, respectively, so I could have put initialization of Sbox[] and invocation of the KeyExpansion method in both the Cipher and the InvCipher methods, but putting them in the constructor results in a cleaner code structure. sBox[] gets populated in Figure 9. The code that populates iSbox[] is similar. The code is structured for readability. As you'll see later, there is a surprising alternative to this way of supplying values for the Sbox and iSbox tables.

Figure 9 Sbox Initialization (Close Figure 9)
private void BuildSbox()
{
  this.Sbox = new byte[16,16] {  // populate the Sbox matrix
/*        0     1     2     3     4     5     6     7     8     9     a     b     c     d     e     f */
/*0*/  {0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76},
/*1*/  {0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0},
/*2*/  {0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15},
/*3*/  {0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75},
/*4*/  {0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84},
/*5*/  {0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf},
/*6*/  {0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8},
/*7*/  {0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2},
/*8*/  {0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73},
/*9*/  {0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb},
/*a*/  {0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79},
/*b*/  {0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08},
/*c*/  {0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a},
/*d*/  {0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e},
/*e*/  {0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf},
/*f*/  {0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16} };

}  // BuildSbox()

Declaring the key schedule table w[], the round constants table Rcon[] and the state matrix State[] in the constructor, and assigning values to Rcon[] and w[] with private helper methods seems to me to be the best way to organize them, but that is mostly a matter of style. The code that populates the round constant table Rcon is shown in Figure 7.

Recall that the left byte of each row of Rcon[] is a power of 2 in GF(28) so this table could be built computationally using something like the following:

newVal = prevVal * 0x02;
The AES constructor finishes by building the key schedule table w[] which is done in the KeyExpansion method (see Figure 10). The code is fairly straightforward. The specification document uses a hypothetical 4-byte word data type. Since C# has no such type, it is simulated with an array of 4 bytes. After the key schedule w[] is allocated space using the operator new, the first Nk (4, 6, or 8) rows of w[] get values from the seed key[] array that was passed into the constructor:
this.w[row,0] = this.key[4*row];
this.w[row,1] = this.key[4*row+1];
this.w[row,2] = this.key[4*row+2];
this.w[row,3] = this.key[4*row+3];

Figure 10 KeyExpansion Method (Close Figure 10)
private void KeyExpansion()
{
  this.w = new byte[Nb * (Nr+1), 4];

  for (int row = 0; row < Nk; ++row)
  {
    this.w[row,0] = this.key[4*row];
    this.w[row,1] = this.key[4*row+1];
    this.w[row,2] = this.key[4*row+2];
    this.w[row,3] = this.key[4*row+3];
  }

  byte[] temp = new byte[4];

  for (int row = Nk; row < Nb * (Nr+1); ++row)
  {
    temp[0] = this.w[row-1,0]; temp[1] = this.w[row-1,1];
    temp[2] = this.w[row-1,2]; temp[3] = this.w[row-1,3];

    if (row % Nk == 0)
    {
      temp = SubWord(RotWord(temp));

      temp[0] = (byte)( (int)temp[0] ^ (int)this.Rcon[row/Nk,0] );
      temp[1] = (byte)( (int)temp[1] ^ (int)this.Rcon[row/Nk,1] );
      temp[2] = (byte)( (int)temp[2] ^ (int)this.Rcon[row/Nk,2] );
      temp[3] = (byte)( (int)temp[3] ^ (int)this.Rcon[row/Nk,3] );
    }
    else if ( Nk > 6 && (row % Nk == 4) )
    {
      temp = SubWord(temp);
    }

    // w[row] = w[row-Nk] xor temp
    this.w[row,0] = (byte) ( (int)this.w[row-Nk,0] ^ (int)temp[0] );
    this.w[row,1] = (byte) ( (int)this.w[row-Nk,1] ^ (int)temp[1] );
    this.w[row,2] = (byte) ( (int)this.w[row-Nk,2] ^ (int)temp[2] );
    this.w[row,3] = (byte) ( (int)this.w[row-Nk,3] ^ (int)temp[3] );

  }  // for loop
}  // KeyExpansion()

The operation of XORing 2 bytes together happens a lot in this code. It requires some casting from byte to int and back to byte because the XOR operator ^ is not defined on the C# byte type. For example

temp[0] = (byte)( (int)temp[0] ^ (int)this.Rcon[row/Nk,0] );
is used instead of:
temp[0] = temp[0] ^ this.Rcon[row/Nk,0];

The KeyExpansion method conditionally calls the private methods SubWord and RotWord to maintain naming consistency with the specification. Again, because there is no word type in C#, I implemented one using an array of 4 bytes. The code for SubWord and RotWord is fairly simple and you should be able to understand it easily by examining it in the AesLib source code that accompanies this article.

The slightly tricky part is looking up the substitution values in SubWord. Recall that to find the substitute value, you separate the input byte into its leftmost 4 bits and its rightmost 4 bits. For a given byte, right-shifting 4 bits with the >> operator will yield the x index, and logical ANDing with 0000 1111 will yield the y value. In slightly longer, but more readable form than the actual code, I could have done something like the following

int x = word[0] >> 4;
int y = word[0] & 0x0f;
byte substitute = this.Sbox[x,y];
result[0] = substitute;
instead of the code I used:
result[0] = this.Sbox[ word[0] >> 4, word[0] & 0x0f ];

To summarize, the AES constructor accepts a key size of 128, 192, or 256 bits and a byte array seed key value. The constructor assigns values for the input block size, the seed key size, and the number of rounds for the encryption algorithm and copies the seed key to a data member named key. The constructor also builds four tables: two substitution tables used by the encryption and decryption methods, a table of round constants, and a key schedule of round keys.

The AES Cipher Method in C#

The code for the Cipher method is shown in Figure 11. It is really very simple because it mostly farms out the work to the private methods AddRoundKey, SubBytes, ShiftRows, and MixColumns.

Figure 11 The Cipher Method (Close Figure 11)
public void Cipher(byte[] input, byte[] output)
{
  // state = input
  this.State = new byte[4,Nb];
  for (int i = 0; i < (4 * Nb); ++i)
  {
    this.State[i % 4, i / 4] = input[i];
  }

  AddRoundKey(0);

  for (int round = 1; round <= (Nr - 1); ++round)
  {
    SubBytes();
    ShiftRows();
    MixColumns();
    AddRoundKey(round);
  }

  SubBytes();
  ShiftRows();
  AddRoundKey(Nr);

  // output = state
  for (int i = 0; i < (4 * Nb); ++i)
  {
    output[i] = this.State[i % 4, i / 4];
  }

}  // Cipher()

The Cipher method starts by copying the plaintext input array to the state matrix State[]. After an initial call to AddRoundKey, the Cipher method iterates one time fewer than the total number of rounds. On the last round, the call to MixColumns is omitted as described in the specification.

The code for the AddRoundKey and SubBytes private methods is shown in Figure 12. The AddRoundKey method needs to know what round it is at so that it can reference the correct four rows of the key schedule array w[]. Notice that State[r,c] is XORed with w[c,r] and not w[r,c]. The SubBytes method extracts indexes from the input byte using the same right-shift-4-bits and mask-with-0x0f technique used in the KeyExpansion method.

Figure 12 AddRoundKey and SubBytes Methods (Close Figure 12)
private void AddRoundKey(int round)
{
  for (int r = 0; r < 4; ++r)
  {
    for (int c = 0; c < 4; ++c)
    {
      this.State[r,c] = (byte) ( (int)this.State[r,c] ^
                                 (int)w[(round*4)+c,r] );
    }
  }
}  // AddRoundKey()

private void SubBytes()
{
  for (int r = 0; r < 4; ++r)
  {
    for (int c = 0; c < 4; ++c)
    {
      this.State[r,c] = this.Sbox[ (this.State[r,c] >> 4),
                                   (this.State[r,c] & 0x0f) ];
    }
  }
}  // SubBytes

The code for the ShiftRows method is shown in Figure 13. Recall that ShiftRows (which might have been better named RotateRows) rotates row[0] 0 positions to the left, row[1] 1 position to the left, and so forth.

Figure 13 ShiftRows Method (Close Figure 13)
private void ShiftRows()
{
  byte[,] temp = new byte[4,4];
  for (int r = 0; r < 4; ++r)
  {
    for (int c = 0; c < 4; ++c)
    {
      temp[r,c] = this.State[r,c];
    }
  }

  for (int r = 1; r < 4; ++r)  //
  {
    for (int c = 0; c < 4; ++c)
    {
      this.State[r,c] = temp[ r, (c + r) % Nb ];
    }
  }
}  // ShiftRows()

After copying State[] into a temp[] matrix, the shifts are then performed with:

this.State[r, (c + r) % Nb ] = temp[r,c];
This takes advantage of the % operator to wrap around a row.

The MixColumns method (see Figure 14) takes every byte and substitutes it with a linear combination of all the other values in the byte's column using GF(28) addition and multiplication. The constant coefficients used for multiplication are based on field theory and are either 0x01, 0x02, or 0x03. The substitution for a given column c is:

State[0,c] = 0x02 * State[0,c] + 0x03 * State[1,c] + 0x01 * State[2,c] +
    0x01 * State[3,c]
State[1,c] = 0x01 * State[0,c] + 0x02 * State[1,c] + 0x03 * State[2,c] +
    0x01 * State[3,c]
State[2,c] = 0x01 * State[0,c] + 0x01 * State[1,c] + 0x02 * State[2,c] +
    0x03 * State[3,c]
State[3,c] = 0x03 * State[0,c] + 0x01 * State[1,c] + 0x01 * State[2,c] +
    0x02 * State[3,c]

Figure 14 MixColumns Method (Close Figure 14)
private void MixColumns()
{
  byte[,] temp = new byte[4,4];
  for (int r = 0; r < 4; ++r)
  {
    for (int c = 0; c < 4; ++c)
    {
      temp[r,c] = this.State[r,c];
    }
  }

  for (int c = 0; c < 4; ++c)
  {
    this.State[0,c] = (byte) ( (int)gfmultby02(temp[0,c]) ^
                               (int)gfmultby03(temp[1,c]) ^
                               (int)gfmultby01(temp[2,c]) ^
                               (int)gfmultby01(temp[3,c]) );

    this.State[1,c] = (byte) ( (int)gfmultby01(temp[0,c]) ^
                               (int)gfmultby02(temp[1,c]) ^
                               (int)gfmultby03(temp[2,c]) ^
                               (int)gfmultby01(temp[3,c]) );

    this.State[2,c] = (byte) ( (int)gfmultby01(temp[0,c]) ^
                               (int)gfmultby01(temp[1,c]) ^
                               (int)gfmultby02(temp[2,c]) ^
                               (int)gfmultby03(temp[3,c]) );

    this.State[3,c] = (byte) ( (int)gfmultby03(temp[0,c]) ^
                               (int)gfmultby01(temp[1,c]) ^
                               (int)gfmultby01(temp[2,c]) ^
                               (int)gfmultby02(temp[3,c]) );
    }
  }  // MixColumns

These expressions are a bit long already so I decided to write private helper functions that return the product of GF(28) multiplication by 0x01, 0x02, and 0x03. The helper functions are very short. For example, the code that's used to field-multiply a byte b by 0x03 is as follows:

return (byte) ( (int)gfmultby02(b) ^ (int)b );

As I discussed earlier, multiplication by 0x02 is the essential operation for all GF(28) multiplication. I called my gfmultby02 method, bending my convention of using the same method names that have been used in the specification; the specification calls this routine xtime.

The Cipher method iteratively applies four operations on its input to produce encrypted output. AddRoundKey substitutes bytes using multiple round keys derived from the single original seed key. SubBytes substitutes bytes using values in a substitution table. ShiftRows permutes bytes by shifting rows of bytes, and MixColumns substitutes bytes using field addition and multiplication of values in a column.

The AES InvCipher Method in C#

The basic premise behind the AES decipher algorithm is simple: to decrypt an encrypted block, just undo every operation in the reverse order. Although this is the basic concept, there are a few details to handle.

The AES specification names the decipher routine InvCipher rather than alternatives Decipher or Decrypt. This is a reflection of the mathematics behind AES, which are based in terms of inverse mathematical operations.

If you compare this code with the Cipher code, you will see that it is pretty much what you might expect, but with two exceptions. First, the order of the inverse method calls (like InvSubBytes) in the InvCipher method is not exactly the reverse of the corresponding calls (like SubBytes) in the Cipher method, and second, InvCipher calls the AddRoundKey method instead of an InvAddRoundKey method. Notice that the InvCipher algorithm uses the key schedule table but starts at the higher-numbered indexes and works its way down to row 0.

The code for the InvSubBytes, InvShiftRows, and InvMixColumns methods closely mirrors the code for the related SubBytes, ShiftRows, and MixColumns methods. The InvSubBytes method is just like the SubBytes method except that it uses the inverse substitution table iSbox[] instead of the Sbox[] table.

As you might guess, iSbox[] just undoes any mapping performed by Sbox[]. For example, if you have byte b that equals 0x20 and find its substitution value in Sbox[], you get 0xb7. If you look up the substitution value for 0xb7 in iSbox[], you get 0x20.

Similarly the InvShiftRows method undoes the ShiftRows method—row[0] is shifted 0 positions to the right, row[1] is shifted 1 position right, row[2] is shifted 2 positions right, and row[3] is shifted 3 positions right.

The InvMixColumns method undoes the work of MixColumns, but not in an obvious way. Recall that MixColumns replaces each byte in the state matrix with a linear combination of bytes in the original byte's column and that the coefficients were 0x01, 0x02, and 0x03. Once again, field theory comes into play. It turns out that the inverse operation is similar but uses multiplication by 0x09, 0x0b, 0x0d, and 0x0e like this:

State[0,c] = 0x0e * State[0,c] + 0x0b * State[1,c] + 0x0d * State[2,c] +
    0x09 * State[3,c]
State[1,c] = 0x09 * State[0,c] + 0x0e * State[1,c] + 0x0b * State[2,c] +
    0x0d * State[3,c]
State[2,c] = 0x0d * State[0,c] + 0x09 * State[1,c] + 0x0e * State[2,c] +
    0x0b * State[3,c]
State[3,c] = 0x0b * State[0,c] + 0x0d * State[1,c] + 0x09 * State[2,c] +
    0x0e * State[3,c]

As with the MixColumns method, I decided to write dedicated helper functions rather than expand the already long expressions inline or write a general multiplication helper function. Let me show you how I wrote the function that multiplies any byte, b, by the constant 0x0e (14 in base 10). The number 14, like any number, can be expressed as the sum of powers of 2. In this case, 14 equals 2 + 4 + 8. And since 4 equals 2 squared and 8 equals 2 cubed, you can express 14 as 2 + 22 + 23. Remember that addition is just XOR (^) in GF(28) and since I already have the gfmultby02 function, I can use it to get my result:

return (byte)( (int)gfmultby02(gfmultby02(gfmultby02(b))) ^  /* 23 + */
                    (int)gfmultby02(gfmultby02(b)) ^         /* 22 + */
                    (int)gfmultby02(b) );                    /* 2    */

All of the operations used by the AES encryption algorithm are invertible, so the decryption algorithm essentially reverses all the operations performed by encryption.

Using the AES Class

One of the features of AES as implemented in C# is its simplicity. Consider the code in Figure 15 that I used to generate the output shown in Figure 1. After declaring hardcoded values for the 16-byte plaintext input and the 24-byte (192-bit) seed key, an AES object is initialized, the encrypting Cipher method encrypts the plaintext to cipher text, and then the cipher text is decrypted back using InvCipher. Very clean and simple.

Figure 15 Using AES (Close Figure 15)
static void Main(string[] args)
{
  byte[] plainText = new byte[] {0x00, 0x11, 0x22, 0x33, 0x44, 0x55,
      0x66, 0x77,0x88, 0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff};

  byte[] cipherText = new byte[16];
  byte[] decipheredText = new byte[16];

  byte[] keyBytes = new byte[] {0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06,
      0x07,0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,0x10, 0x11,
      0x12, 0x13, 0x14, 0x15, 0x16, 0x17};

  Aes a = new Aes(Aes.KeySize.Bits192, keyBytes);

  Console.WriteLine("\nAdvanced Encryption System Demo in .NET");
  Console.WriteLine("\nThe plaintext is: ");
  DisplayAsBytes(plainText);

  Console.WriteLine("\nUsing a " + Aes.KeySize.Bits192.ToString() +
      "-key of: ");
  DisplayAsBytes(keyBytes);

  a.Cipher(plainText, cipherText);

  Console.WriteLine("\nThe resulting ciphertext is: ");
  DisplayAsBytes(cipherText);

  a.InvCipher(cipherText, decipheredText);

  Console.WriteLine("\nAfter deciphering the ciphertext, the result
      is: ");
  DisplayAsBytes(decipheredText);

  Console.WriteLine("\nDone");
  Console.ReadLine();
}  // Main()

static void DisplayAsBytes(byte[] bytes)
{
  for (int i = 0; i < bytes.Length; ++i)
  {
    Console.Write(bytes[i].ToString("x2") + " " );
    if (i > 0 && i % 16 == 0) Console.Write("\n");
  }
  Console.WriteLine("");
}  // DisplayAsBytes()

Because an AES object works on byte arrays, you can easily adapt it to work on other .NET data types. I constructed a little Windows®-based demo app that accepts a single 8-character (16-byte) string and then encrypts and decrypts it. A sample run is shown in Figure 16.

Figure 16 Encryption Demo
Figure 16 Encryption Demo

Because both the encryption and decryption routines need to know what key size the user has specified, I declared it as a class-scope variable, like this:

private Aes.KeySize keysize;
Notice that the seed key is not specified by the user. The demo uses a "null key" which consists of all zero bytes by supplying a dummy new byte[16] to the constructor. The size of the dummy argument is irrelevant because the seed key will also be initialized to all zeros. Null key encryption and decryption is an easy and effective way to deter casual external examination of your data. The Encoding.Unicode.GetBytes and Encoding.Unicode.GetString methods in System.Text make it very easy to convert a .NET string to a byte array, and vice versa.

Implementation Alternatives

Now let's look at some important variations of the AES implementation presented in this article, possible extensions of the code presented here, and cryptanalysis attacks against AES.

As much as any code that I've ever worked on, the AES algorithm has significant alternative approaches to implementation. Why is this important? AES is intended to be applicable to a wide range of systems, from smart cards with tiny memory capacities to large multiprocessor mainframe systems. In many scenarios, performance is critical and sometimes memory or processing resources are limited. Virtually every routine in AES can be modified to optimize performance at the expense of memory, or vice versa. For example, assigning the 256 values to the substitution table Sbox[] seems straightforward enough. However, these values are based on GF(28) theory and it turns out that these values can be generated programmatically. The same is true for the inverse substitution table and the round constants table.

Another interesting possibility for alternate implementation is the GF(28) multiplication used by the Cipher and InvCipher methods. My implementation codes a basic function that multiplies by 0x02 and then six additional functions which call gfmultby02. Another possibility would be to write a general multiplication function and use it instead of implementing seven separate functions like I did. At another extreme you could use a complete table of the product of all 256 possible byte values multiplied by 0x01, 0x02, 0x03, 0x09, 0x0b, 0x0d, and 0x0e. Yet another way to approach GF(28) multiplication is to implement it as a lookup into two 256-byte arrays, usually called alog[] and log[] because they are based on certain logarithm-like properties of GF(28).

Although the AES class presented here is fully capable of encrypting any form of .NET data, you might want to consider extending it in a number of ways. First, because the emphasis of this article is on presenting a clear explanation of AES, all error checking was stripped away. In my experience, adding a reasonable amount of error checking to a class similar to this AES class would triple the size of the source code. Because AES uses so many arrays, there is a lot of index bounds checking that should be done. For example, the constructor presented does not even check the size of the seed key parameter.

You might also consider extending this AES class by adding more features. The most obvious place to start would be to add methods that encrypt and decrypt fundamental .NET data types such as System.String and System.Int32. A more ambitious extension would be to implement an encrypted stream class.

How secure is AES? This is a hard question to answer, but the general consensus is that it is the most secure encryption algorithm available. AES has been subjected to more scrutiny than any other encryption algorithm to date. On both a theoretical and practical basis, AES is considered "secure" in the sense that the only effective way to crack it is through a brute-force generation of all possible keys. With a key size of 256 bits, no known brute-force attack can break AES in a reasonable amount of time (it would take years even on the fastest systems available).

Note that the most likely successful attack on an AES cipher results from a weak implementation that allows what is called a timing attack. The attacker uses different keys and precisely measures the time the encryption routine requires. If the encryption routine is carelessly coded so that execution time depends on the key value, it is possible to deduce information about the key. In AES, this is most likely to occur in the MixColumns routine because of field multiplication. Two safeguards against this attack are to insert dummy instructions so that all multiplications require the same number of instructions, or to implement field multiplication as a lookup table, as I've described.

There are many possible implementations of AES, especially using lookup tables rather than computation. The basic AES class presented in this article can be used to encrypt and decrypt any form of .NET data or can be extended into a class with added functionality.

Conclusion

The new AES will certainly become the de facto standard for encrypting all forms of electronic information, replacing DES. AES-encrypted data is unbreakable in the sense that no known cryptanalysis attack can decrypt the AES cipher text without using a brute-force search through all possible 256-bit keys.

The major obstacle I found when implementing an AES class in the Microsoft® .NET Framework was that the official specification document was written from a mathematician's point of view rather than from a software developer's point of view. In particular, the specification assumes that the reader is fairly familiar with the GF(28) field and it leaves out a few key facts regarding GF(28) multiplication that are necessary to correctly implement AES. I've tried here to remove the mystery from AES, especially surrounding GF(28) field multiplication.

It is only a question of time before AES encryption becomes widely available from Microsoft and third-party vendors in the form of .NET Framework libraries. However, having this code in your skill set will remain valuable for a number of reasons. This implementation is particularly simple and will have low resource overhead. In addition, access to and an understanding of the source code will enable you to customize the AES class and use any implementation of it more effectively.

Security is no longer an afterthought in anyone's software design and development process. AES is an important advance and using and understanding it will greatly increase the reliability and safety of your software systems.

For related articles see:
Security: Protect Private Data with the Cryptography Namespaces of the .NET Framework
Exploring Kerberos, the Protocol for Distributed Security in Windows 2000

For background information see:
The Design of Rijndael: AES - The Advanced Encryption Standard by Joan Daemen and Vincent Rijmen. (Springer-Verlag, 2002)
Announcing the Advanced Encryption Standard (AES): Federal Information Processing Standards Pub 197

James McCaffrey works for Volt Information Sciences Inc. where he manages technical training for software engineers at Microsoft. He has worked as a contractor on several Microsoft products including Internet Explorer and MSN Search. Reach him at jmccaffrey@volt.com or v-jammc@microsoft.com.


Authorize It: Use Role-Based Security in Your Middle Tier .NET Apps with Authorization Manager

New information has been added to this article since publication.
Refer to the Editor's Update below.

Authorize It

Use Role-Based Security in Your Middle Tier .NET Apps with Authorization Manager

Keith Brown
This article assumes you're familiar with Windows Server 2003
Level of Difficulty 1 2 3
Code download available at: AuthorizationManager.exe (128KB)
SUMMARY

Authorization Manager in Windows Server 2003 represents a significant improvement in the administration of role-based security, making it more scalable, flexible, and easier to implement. Using Authorization Manager, you can define roles and the tasks those roles can perform. You can nest roles to inherit characteristics from other roles, and you can define application groups. In addition, Authorization Manager lets you use scripts to modify permissions dynamically, and it allows you to wrap your security logic in a security policy that can be stored in Active Directory. Authorization Manager also includes an easy-to-use API for running access checks. The author discusses all of these topics and demonstrates them with a working sample.

Contents
Introducing Authorization Manager
A Sample App: the Corporate Library
Authorization Store
The AzMan Runtime Interface
Stores, Applications, and Scopes
Application Groups
Scripts
Supporting Authorization Scripts
Auditing
Conclusion

R
ole-based security has been evolving on the Windows® platform since the first release of Windows NT®. Using roles, the operating system can make determinations such as whether a process is privileged by checking the security context for a group called BUILTIN\Administrators. The operating system makes decisions based on this logical role (such as whether to let you install services or device drivers). When you install the operating system, you get to choose who will assume this role by adding them to the Administrators group.

Microsoft® Transaction Services (MTS) and COM+ tried to make role-based security palatable for application developers, providing a simple role-based authorization infrastructure for COM servers. The goal was to enable a trusted subsystem model for multitier server applications, where an application server is trusted by back-end resources to authorize requests. By performing authorization early, you avoid having to delegate client credentials to back-end servers. Delegation is rife with a whole host of problems that range from potential security vulnerabilities to scalability issues.

If you've been looking for a general-purpose authorization solution in the middle tier, your search may very well be over.

Introducing Authorization Manager

Authorization Manager (commonly known as AzMan) is a new general-purpose role-based security architecture for Windows. AzMan is not tied to COM+, so it can be used in any application that needs role-based authorization, including ASP.NET Web apps or Web Services, client-server systems based on .NET Remoting, and so on. As of this writing, Authorization Manager is available only on Windows Server™ 2003, service pack 4 for Windows 2000, and is slated to ship as a future service pack for Windows XP.

There are two parts to AzMan: a runtime and an administration UI. The runtime, provided by AZROLES.DLL, exposes a set of COM interfaces used by applications that employ role-based security. The administration UI is an MMC snap-in that you can try out by running AZMAN.MSC or by adding the Authorization Manager snap-in to your MMC console of choice. (Note that the admin UI was not backported to older platforms, so you'll need to use a box running Windows Server 2003 to administer AzMan.) [ Editor's Update - 1/9/2004: The Windows Server 2003 Administration Tools Pack lets you install the Windows Server 2003 management tools on a machine running Windows XP Professional.]

The first thing you'll notice when you run the AzMan admin tool shown in Figure 1 is that it's much more complex than what COM+ offered. You no longer have only roles and role assignments. You now have low-level operations that can be assigned to tasks, which can then be assigned to roles. Tasks can include other tasks, and roles can include other roles. This hierarchical approach helps cap the seemingly unbounded set of roles needed in today's complex applications.

Figure 1 Administration Manager
Figure 1 Administration Manager

Here's how tasks and roles are created. The application designer defines the entire set of low-level operations that are considered security sensitive. The designer then defines a set of tasks that map onto those operations. Tasks were designed to be understandable by business analysts, so a given task is always composed of one or more low-level operations. If a user is granted permission to perform a task, he or she is granted permission to all operations in the task. As an example, a task named Submit Purchase Order might consist of the following operations: Acquire Next PO Number, Enqueue PO, and Send Notification. Of course, you could always simply map each task to a single operation to keep things as simple as possible, but the flexibility of separating tasks and operations is available if you need it.

Once the tasks and operations are defined, you can start coding, and include calls into the AzMan runtime anytime a sensitive operation needs to be performed. This call is IAzClientContext.AccessCheck, and I'll show an example of its usage shortly.

At deployment time, the application setup program sets up an AzMan store either as part of Active Directory® or in a simple XML file, and installs the basic low-level operations and tasks. The administrator uses the AzMan snap-in to see definitions and descriptions of the tasks for the application. The administrator then defines roles that make sense for his or her organization. Just as a task is defined as a set of low-level operations, roles are usually defined as a set of tasks. The administrator can then assign users and groups to those roles. In fact, the administrator's main job in maintaining the application from here on out will be adding and removing users from roles as people join or leave the company or change job titles.

So far, I've focused on the application developer and the administrator, but there may actually be a third person helping with the deployment: the business logic scripter. Every task can have a script associated with it. The idea here is to find dynamic security decisions typically made through calls to IsCallerInRole, and move them out of application code and into a place where an administrator can make changes to an application's security policy without having to modify and recompile code.

A Sample App: the Corporate Library

Let's look at an example. Imagine you are building a system to manage a company library. You need to be able to manage the book inventory, check books in and out, and so on. You'll use AzMan to implement role-based security.

First, you need to make a list of the sensitive operations that appear in your design:

  • Read catalog
  • Place hold (for self or another)
  • Check out book
  • Check in book
  • Add book to inventory
  • Remove book from inventory
  • Read patron history (for self or another)

Note that a couple of operations are sensitive to information that you'll only have at run time. For instance, when attempting to read a patron's history, the application must provide contextual information indicating whether the user is trying to access his own history or that of someone else. While prototyping, you can use the AzMan snap-in to add these operations to a simple XML store. Figure 1 shows what this looks like.

If you want to try following along on your own, run AZMAN.MSC, and make sure you're in developer mode via the Action | Options menu. Create a new store in an XML file, then create a new application in that store. Next, add the operations one by one, giving them a name and a unique integer that represents the operation number. This number is how the application developer identifies operations in calls to AccessCheck. Note that in naming operations, I've encoded the names with the prefix "op." This is simply to avoid naming collisions later when creating tasks and roles since these names all come from the same pool and must be unique.

The AzMan snap-in operates in two modes: developer and administrator. In administrator mode, you don't have the option of creating stores or applications, and you aren't allowed to muck with the low-level operation definitions that the application code relies upon. Frankly, nothing stops the system administrator from entering developer mode and doing these things, but the point is that in administrator mode, the number of options in the UI is reduced to simplify the administrator's life and to help avoid mistakes.

Figure 2 Task Definitions in AzMan
Figure 2 Task Definitions in AzMan

The next step is to define a set of tasks that map to these low-level operations so the administrator will have an easy job defining roles. Because you kept your list of operations simple, you can define a single task for each operation. There's a good reason to keep things simple unless you absolutely need more complexity, and it has to do with business logic scripts—but more on that later. So for now, let's define a list of tasks that are basically the same as my operations. Figure 2 shows what this looks like in AzMan when you edit a Task Definition.

Authorization Store

It's time to switch hats and pretend that you're the administrator deploying the application. Switch to administrator mode under the Action | Options menu and note how the GUI changes: you are no longer allowed to edit the low-level operations for the application. Go ahead and add roles for the application as defined in Figure 3.

Figure 3 Roles and Tasks
Figure 3 Roles and Tasks

One way to simplify things here is by nesting roles. For example, Clerk can be defined in terms of the Patron role, with the addition of the Check out book and Check in book tasks, as I've shown in Figure 4. Try doing that in COM+!

Figure 4 Nesting Roles
Figure 4 Nesting Roles

The last thing the administrator needs to do is make these abstract roles concrete by adding real users to them. To do this, select the Role Assignments folder and choose the Assign Roles action. Note that a role doesn't actually become active in an application until it's been added to this folder. For example, the IAzApplication.Roles property only returns the collection of roles that have been added to the Role Assignments folder, as opposed to all roles that have been defined. Once a role has been assigned, right-click it to add either Windows users and groups, or application groups that you've previously defined in your AzMan store. I'll describe application groups later in this article.

The AzMan Runtime Interface

Once you've got some operations and tasks defined, you can start implementing access checks in code. The first thing you need to think about is authentication. If you can use some built-in Windows plumbing, like the Web server's support for Kerberos authentication, you can get a token for the client. This is by far the most natural way to use AzMan, since a token contains all the groups a user is in, making it fast to map that user onto a set of AzMan roles. If, on the other hand, you're using a form or X.509 certificate to authenticate the user, you won't have a token. Rather, you'll only have the user's name. This doesn't mean you can't use AzMan or even that you'll have to write more code. But it does mean that it'll be more expensive, since the AzMan runtime will have to look up the user's groups manually. This incurs round-trips to domain controllers.

The first thing the app needs to do is initialize the AzMan runtime, point it to the store it plans to use, and drill down to the application where the authorization settings live. For now, let's use a simple XML-based store:

AzAuthorizationStore store = new AzAuthorizationStoreClass();
store.Initialize(0, @"msxml://c:\MyStore.xml", null);
IAzApplication app = store.OpenApplication(
    "Corporate Library Application", null);
To build this application, the project needs to reference the AzMan interop assembly, which can be found in the %WINDIR%\Microsoft.NET\Framework\AuthMan directory.

Now that the application is bootstrapped, when a new client is authenticated you need to construct a representation of the client's security context. The context is a lot like a token in that it caches role mappings for a user:

IAzClientContext ctx =
  app.InitializeClientContextFromToken(htoken, null);

Where do you get the client's token? Well, that depends on what type of app you're writing. For example, here's some C# code from an ASP.NET page that obtains the client's token. In this case, web.config specifies authentication mode='Windows', and IIS has been configured to require integrated Windows authentication:

WindowsIdentity id = (WindowsIdentity)User.Identity;
IntPtr htoken = id.Token;

If you only know the client's name and don't have access to their token, try to figure out if there's a token present that you can get your hands on because a token is the most authoritative way to discover groups for a client. It's also the fastest way, as I mentioned earlier. If you are sure that there's no token for the client available to you, then use this alternate method to initialize a context from an account name in the form: domain\user. This call may incur round-trips to discover domain groups, so expect that it will take some time to execute:

IAzClientContext ctx =
  app.InitializeClientContextFromName(name, null);

Once you have a client context, you can run an access check. This call takes a number of arguments, but for now I'll keep things simple. Let's say you're implementing a function that adds a book to the inventory. I defined "Add book to inventory" as operation number five, so the code might look like what's shown in Figure 5.

Figure 5 Add Book to Inventory (Close Figure 5)
// always define constants or enums for your ops!
const int opAddBookToInventory = 5;
const int NO_ERROR = 0;

// later in the code...
object[] operations = { opAddBookToInventory };
object[] scopes = { "" };

object[] results = (object[])
  ctx.AccessCheck(nameOfBook,
                  scopes, operations,
                  null, null, null, null, null);
int result = (int)result[0];
if (NO_ERROR == result) {
  // access is granted
}
else {
  // access was denied, result holds error code
}

The first argument, nameOfBook, is a string that is used if you've got runtime auditing turned on. It identifies the object on which you're performing the operation, so you should always provide some meaningful information here. I've used the default value for the second argument, scopes, which I'll explain a bit later. The third argument is where you list one or more operations that you want to test. The result is an array that is always the same size as the operations array, with an integer status code for each operation that indicates whether access is granted or denied. Zero indicates that the access check succeeded, and the client identified by the context is allowed to perform the specified operation. Any other value indicates failure (generally what you'll see is the number 5, which is ERROR_ACCESS_DENIED).

The AzMan runtime interface isn't strongly typed. It uses VARIANTs for most of its arguments. This allows classic ASP programmers who use scripting languages to use AzMan, but means that programmers who use strongly typed languages like C# and Visual Basic® .NET may make some mistakes when calling AccessCheck that won't be caught until run time. For example, the operations array must be of type object[], not int[], but the compiler won't complain if you pass an int[] because the actual type of the argument is object. This bit me when I was first learning this API, and it took me a while to figure out exactly how to write the code to avoid getting run-time errors due to parameter type mismatches. I've heard rumors that eventually there will be a managed interface to AzMan, but until that happens, you might want to write a strongly typed wrapper around AccessCheck to avoid getting bitten. The following code shows an example that also simplifies the most common way you'll call the function:

public class AzManWrapper {
  public static int AccessCheck(
             IClientContext ctx,
             string objectName,
             int operation) {
    object[] results = (object[])ctx.AccessCheck(
         objectName, scopes, ops,
         null, null, null, null, null);
    return (int)results[0];
  }
}

With a wrapper, you can provide your own overloads of AccessCheck to handle more sophisticated situations where other optional arguments are necessary. Using a wrapper for this function in particular should save you a lot of grief and will reduce clutter in your application code. You could even use this wrapper to convert AccessCheck failures into exceptions instead of returning a status code, along the lines of IPermission.Demand. Don't go crazy and wrap the whole suite of interfaces though, since this function is really the only one that's tricky to call.

One thing you might be wondering at this point is whether you can use AzMan without using Windows accounts to represent users. The runtime was designed with this in mind, although you need to define custom security identifiers (SIDs) for each user, which is not terribly difficult, and you must call an alternate method to initialize a client security context, namely InitializeClientContextFromStringSid. The biggest hurdle is that you won't be able to use the AzMan snap-in to manage your stores, which are pretty tightly coupled to Windows users and groups. For more details on how to approach this, see the whitepaper by Dave McPherson referenced at the beginning of this article.

Stores, Applications, and Scopes

I want to back up for a moment and discuss the structure of the authorization store a bit. First of all, you have two choices for storing your authorization settings: you can drop the entire store into an XML file or you can host it in Active Directory. I'd strongly recommend using Active Directory for production apps, as it provides a lot more functionality and often better performance than a simple XML file.

If you have a test domain in a lab that you can play with, try bringing up the AzMan snap-in and creating a new store in Active Directory with a distinguished name like this: "CN=MyStore, CN=Program Data, DC=MyDomain, DC=com", replacing MyDomain with your own. To see what objects AzMan created in Active Directory, use a tool like ADSIEdit, an MMC snap-in that you can install from your Windows Server 2003 CD by running SUPPORT\TOOLS\SUPTOOLS.MSI. Create an application in the store and bring up the new application's property page. You'll notice that there are security and auditing tabs that aren't present when you're using a simple XML file.

In Active Directory stores, you can delegate responsibility for administering individual applications within a store, and you can audit changes to the store at a very detailed level. With an XML file, you're limited to securing the file itself with NTFS permissions and auditing. Currently, run-time auditing is only supported if the store is housed in the directory, and auditing is tremendously important to most applications. If Active Directory is available, I would strongly urge you to house your AzMan stores in it, since it's the best place for hosting security policy on Windows.

A single store can house multiple applications. Each application has its own namespace for operations, tasks, and roles. Be aware of concurrency issues if you share a store among multiple applications because stores do not yet support concurrent editing. If you think there's a chance two administrators might be editing a single store at the same time, you need to provide some external locking to serialize access to the store; otherwise, it might become corrupted. The AzMan snap-in doesn't provide this, and until it does, you will be better off limiting the contents of each store to avoid concurrent editing. The simplest solution is to limit each store to housing a single application.

Each application can also define multiple scopes, which is an advanced feature of Authorization Manager that I recommend only to people who have studied AzMan further and absolutely need this extra level of complexity. Scopes allow you to have different authorization settings for different parts of your application. For example, in a large Web application, roles might be assigned one way under a certain subdirectory; under a different subdirectory, the roles might be assigned differently or a totally different set of roles might be defined.

Scopes can be convenient in this case because they can share the low-level operation definitions and maybe even some tasks, roles, and application groups. Unfortunately, they can also be confusing and easy to misuse. For example, when you call AccessCheck, the second argument is where you specify the scope you want to use for the check. If the user is providing the scope name, perhaps via the URL in the request, you'd better be sure that scope name is canonicalized before you pass it to AccessCheck; otherwise, you may allow clever users to fool you into using a weaker scope by encoding the name in an unexpected way. If you're new to this type of attack, you should read the chapter on canonicalization in Writing Secure Code, Second Edition (Microsoft Press®, 2002). To learn more about advanced features like scopes, see the whitepaper by Dave McPherson referenced at the beginning of this article.

Application Groups

There's a nifty feature in AzMan called Application Groups. In large organizations, it can be a real pain to get new groups added to the directory for your application to use. In fact, if it's only your application that needs a particular group definition, you might be simply out of luck when an overworked domain administrator refuses to add yet another entry to his already barely manageable list of groups. In this case, Application Groups can save you. At the store, application, or scope level, you can define groups of users and assign a logical group name to them. You can then use these application groups in role assignments.

AzMan provides two types of application groups: basic and Lightweight Directory Access Protocol (LDAP) query groups. The basic groups are a lot like the groups in Active Directory, but with a twist: you can define both included members and excluded members. For example, you could define a group called EveryoneButBob, as I did in Figure 6. The benefit is increased functionality and convenience. The drawbacks are the increased number of CPU cycles required to determine membership in the application group and the memory required to store the membership list in the application group, so use this feature with care. If you like the exclusion feature shown in Figure 6, you can still get it by using domain groups as members in your application groups, thus reducing the size of the membership list AzMan needs to keep in memory.

Figure 6 Allow Everybody Except Bob
Figure 6 Allow Everybody Except Bob

LDAP query groups are an expensive but nifty feature of AzMan. Here you can use LDAP query syntax to define a group of users who are similar in some way. For example, here's how you might define the set of engineers who are at least 21 years old:

(&(age>=21)(memberOf= CN=eng,DC=foo,DC=com))

Regardless of the type of group, Basic or LDAP query, the administrator can use these application groups as alternative ways of assigning users to roles.

Scripts

For cases where static grants aren't enough, your application can provide extra context in the form of variables and object references to AccessCheck. This allows a script writer to add business logic using JScript® or VBScript without having to change and recompile the application. For example, the Read patron history task defined earlier could be supplied with extra context, perhaps the set of roles in which the user is a member, and a Boolean indicating whether the client is accessing her own history or that of someone else. This would enable you to write a script that permits Managers to review any patron's history, but restricts normal patrons to reviewing their own histories. A script such as the one shown in Figure 7 could be written for this task.

Figure 7 Review Patron's History (Close Figure 7)
' always start by being pessimistic
AzBizRuleContext.BusinessRuleResult = false
isManager = false
roles = AzBizRuleContext.GetParameter("roles")
for each role in roles
  if "Manager" = role then
    isManager = true
    exit for
  end if
next
if isManager then
  ' Managers are allowed to read any patron's history
  AzBizRuleContext.BusinessRuleResult = true
else
  ' anyone else is allowed to read their own history
  self = AzBizRuleContext.GetParameter("self")
  AzBizRuleContext.BusinessRuleResult = self
end if

To associate this script with the Read patron history task, bring up the definition page for the task, browse to a file containing this script, specify the language as VBScript, and then press the "Reload Rule into Store" button.

Supporting Authorization Scripts

To support the script shown in Figure 7, you'll need to pass a few more arguments to any access check that involves the Read patron history task. Since you've kept things simple and only defined one task per operation, this means you can provide this context whenever you ask about the corresponding operation. Here's a snippet of code that shows how you would provide these extra context variables for a script writer:

bool self = _userIsCheckingOwnHistory();
object[] operations = { opReviewPatronHistory };
object[] scopes     = { "" };
object[] varNames   = { "roles", "self" };
object[] varValues  = { ctx.GetRoles(""), self };

object[] results = (object[])
ctx.AccessCheck(nameOfPatronHistory,
                scopes, operations,
                varNames, varValues,
                null, null, null);

AzMan is a bit picky about the order of the varNames and varValues array. You must order varNames alphabetically, as I did. The varValues array must then provide the corresponding value for each named parameter in varNames, which is pretty obvious. If you want to get even fancier, you can use the last three arguments of AccessCheck to pass in named object references. This will expand the object model that the script writer sees beyond the default AzBizRuleContext object. I'll leave you to experiment on your own with that feature and address some of the challenges you'll run into if you decide to support scripting.

The first odd thing you'll probably notice about scripts is that they are defined at the task and role level, not at the operation level. But application programmers perform access checks and provide context variables for individual operations, so how does the script writer know what variables will be available for a given task? Clearly it's up to the developer to document this carefully on a per-operation basis. One strategy is to keep things really simple and always pass the same context variables no matter what operation is being performed. This would certainly simplify things for a script writer. In my library example, I was careful to define one task per operation so I could customize the context variables for each task, but remember that administrators can define new tasks when running in admin mode. What would happen if the systems administrator were to define a new task that incorporated two operations that each provided different context variables? Just try to keep things simple and document carefully how scripts should be written in order to avoid these nasty situations.

Another thing to watch out for when writing scripts is that the results of scripts are cached in the client context object for efficiency. Throw away the client context and you throw away the cache. This is good to know because some scripts might be dependent on external data that could change over time.

Be aware that scripts are designed to qualify the tasks or roles to which they are attached. Say, for example, that Alice is a member of roles R1 and R2. Role R1 is directly granted permission to perform operation X. Role R2 is granted the same permission, but this grant is qualified by a script. When Alice tries to perform operation X, AzMan won't even bother running the script in role R2 because operation X is already statically granted through role R1. Thus, scripts cannot be used to deny permissions outright. They can only be used to decide whether a particular task or role should be considered in an access check. Just because a scripted task or role is omitted because its corresponding script evaluated to false doesn't mean there isn't an entirely different task or role that will still grant the operation being tested. Dave McPherson's whitepaper provides a very detailed description of how the runtime implements an access check. You'll find this in the Performance section. For anyone serious about using AzMan, I suggest you study this section carefully.

Auditing

Runtime auditing of access checks is an important feature that's available only if you're using an Authorization Manager store within Active Directory. If you want to enable this feature, right-click the application, choose Properties, and turn it on through the Auditing tab. At run time, the account your server process runs under is important: it must be granted the Generate Audits privilege. The built-in accounts Local Service, Network Service, and SYSTEM all have this privilege by default. Finally, note that the server machine must have auditing of object access turned on for these audits to be captured in the security event log.

What you'll see after enabling auditing is that each call to AccessCheck results in an audit entry, where the object name in the entry is whatever string you passed as the first argument to AccessCheck. The operation's friendly name will be shown in the audit, along with the client's name. If the check succeeded, a successful access will be logged; otherwise you'll see a failed access in the log. Whether you see both success and failure audits depends on what level of object access auditing you enabled via the Windows security policy.

Conclusion

Authorization Manager is an important tool for building secure systems on Windows. It expands on the idea of role-based security popularized by MTS and COM+, but can be used by any server application, not just COM-based servers. Authorization Manager strives to help you centralize security logic into a concise security policy that can be stored in Active Directory, and it provides a simple API for performing access checks. Runtime auditing satisfies a longtime need as well.

Authorization Manager has loads of features, and part of your job when writing secure code is to figure out what subset of those features your application requires. And finally, remember to keep things as simple as possible to avoid opening security holes.

For related articles see:
Role-Based Access Control for Multi-tier Applications Using Authorization Manager

For background information see:
Programming Windows Security by Keith Brown (Addison-Wesley, 2000)

Keith Brown is an independent consultant who specializes in application security on the Windows platform. He is the author of Programming Windows Security (Addison-Wesley, 2000), and teaches regularly for DevelopMentor. Keith is working on a new book which you can read online at http://www.develop.com/kbrown.


Review It: Expert Tips for Finding Security Defects in Your Code

Review It

Expert Tips for Finding Security Defects in Your Code

Michael Howard
This article assumes you're familiar with Security, C#, and Visual Basic .NET
Level of Difficulty 1 2 3
SUMMARY

Reviewing code for security defects is a key ingredient in the software creation process, ranking alongside planning, design, and testing. Here the author reflects over his years of code security reviews to identify patterns and best practices that all developers can follow when tracking down potential security loopholes. The process begins by examining the environment the code runs in, considering the roles of the users who will run it, and studying the history of any security issues the code may have had. After gaining an understanding of these background issues, specific vulnerabilities can be hunted down, including SQL injection attacks, cross-site scripting, and buffer overruns. In addition, certain red flags, such as variable names like "password", "secret," and other obvious but common security blunders, can be searched for and remedied.

Contents
Allocating Time and Effort
Buffer Overruns in C and C++
Integer Overflows in C and C++
Database Access Code in Any Language
Web Page Code in Any Language
Secrets and Cryptography in Any Language
ActiveX Controls in Visual Basic and C++
Conclusion

A
large part of my job involves reviewing other people's code, looking for security errors. Admittedly, it's not my number one task—that tends to be design review and threat modeling—but I sure get to see quite a lot of code.

Hopefully, you realize that reviewing other people's code, while a good thing to do, is not how you create secure software. You produce secure software by having a process to design, write, test, and document secure systems, and by building time into the schedule to allow for security review, training, and use of tools. Simply designing, writing, testing, and documenting a project, and then looking for security bugs doesn't create secure software. Code reviewing is just one part of the process, but by itself does not create secure code.

In this article I will not discuss the nature of code vulnerabilities such as integer overflow attacks, SQL injection, and buffer overruns; you can learn about these issues in books (such as my book Writing Secure Code, Microsoft Press®, 2002). Instead, I will be taking a high-level look at issues I watch for during a code review. Before I get started, though, I want to point out that this is how I review code for security bugs; it's not necessarily how you should review code, nor can I guarantee it to be the most complete form of review for certain kinds of defects. I want to document what goes on in my head when I look at code in the hopes that it will prove helpful to you as well.

As I see it, there are three ways to review code: in-depth analysis, rapid analysis, and a hybrid approach. I tend to focus on a hybrid approach since it has the advantage of covering a lot of ground quickly; if I see something that I think requires more in-depth analysis I will flag it for a future code review, possibly involving other experts in the area of concern. But for the moment, let me discuss the initial rapid code review or, as I like to call it, the Sweep 'n' Tag method—sweep the code quickly, and tag code requiring further review. The following outlines my process.

Allocating Time and Effort

I have a ranking system I use to determine how much relative time I need to spend reviewing the code. The system is based on the damage potential if a vulnerability is exploited and the potential for attack. The quota system is based on the following traits:

  • Does the code run by default?
  • Does the code run with elevated privileges?
  • Is the code listening on a network interface?
  • Is the network interface unauthenticated?
  • Is the code written in C/C++?
  • Does the code have a prior history of vulnerability?
  • Is this component under close scrutiny by security researchers?
  • Does the code handle sensitive or private data?
  • Is the code reusable (for example, a DLL, C++ class header, library, or assembly)?
  • Based on the threat model, is this component in a high-risk environment or subject to many high-risk threats?

If I get more than three or four positive hits from this list, then I'll review the code in more depth. In fact, if the code is listening on a Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) socket and is running by default, be prepared to spend a lot of time reviewing the code.

In looking for security errors, I tend to review three major categories of code C/C++ code, web-server application code (such as ASP, ASP.NET, CGI, and Perl), and managed code (C# mainly, and some Visual Basic® .NET).

There are some nuances of each language of which you should be aware. First, the number one issue with C and C++ is buffer overruns. Admittedly, there are other issues too, but when you hear the words "buffer" and "overrun" in the same sentence you can almost always guarantee that C or C++ is involved. Higher-level languages such as C#, Visual Basic .NET, and Perl should have no buffer overruns. If there is one, the bug is probably in the runtime environment and not in the code under review. However, such languages are often used in Web server application code and are subject to other kinds of defects. Buffer overruns are nasty because the attacker can inject code into the running process and hijack it. So let's look at buffer overruns first.

Buffer Overruns in C and C++

Buffer overruns are the scourge of the software industry and you should do everything in your power to remove them from your code. Better yet, make sure that they don't get into the code in the first place.

There are two ways I review for buffer overruns. The first is to identify all the entry points into the application, especially network entry points, tracing the data as it moves through the code and questioning how that data is being manipulated. I assume that all data is malformed. When looking at any code that touches (reads from or writes to) the data, I ask, "Is there a version of the data that can cause this code to fail?" This method is thorough, but very time consuming. Another technique is to look for known and potentially dangerous constructs and trace the data back to the point of entry. For example, take a look at this code:

void fuction(char *p) {
    char buff[16];
    •••

    strcpy(buff,p);

    •••
}
If I see code like this, I'll trace the variable p back to its source, and if it came from somewhere I don't trust, or is not checked for validity close to the point at which it's copied, then I know I've found a security bug. Note that strcpy by itself is not dangerous or insecure. Rather, it's the data that makes functions like this scary. If you check that the data is well formed, then strcpy can be safe. Of course, if you are wrong, then you have a security bug. I also check for the "n" string manipulation functions, such as strncpy, because you also need to check that the buffer size calculation is correct.

I'm wary of code that handles tagged file formats. By tagged I mean files that are made of chunks, where each chunk has a header that describes the next chunk of data. A good example is the MIDI music format. A serious security bug was found and fixed in a Windows component called quartz.dll that handles MIDI files. A malformed MIDI construct caused the code that was handling the file to fail, or worse. You can read more about this bug at Unchecked Buffer in DirectX Could Enable System Compromise.

Another construct I look out for is this:

while (*s != '\\')
    *d++ = *s++;
This loop is constrained by a character in the source; it is not constrained by the destination size. Basically, I scan for *x++ = *y++ using the following regular expression:
\*\w+\+\+\s?=\s?\*\w+\+\+
Of course, people can also use *++x = *++y, so you need to scan for this, too. I want to stress once again that this construct is not dangerous unless the source isn't trusted, so you need to determine the trustworthiness of the source data.

Next is another kind of issue related to buffer overruns that you should be aware of: the integer overflow vulnerability.

Integer Overflows in C and C++

If you don't know what integer overflow attacks are and how to fix them, you should read my article at Development Impacts of Security Changes in Windows Server 2003 first. The real security breach occurs with these defects when arithmetic is performed to calculate a buffer size and the calculation causes an overflow or underflow. Look at the following example:

void func(char *b1, size_t c1, char *b2, size_t c2) {
    const size_t MAX = 48;
    if (c1 + c2 > MAX) return;
    char *pBuff = new char[MAX];
    memcpy(pBuff,b1,c1);
    memcpy(pBuff+c1,b2,c2);
}
The code looks fine until you realize that there's an issue if you add c1 and c2 together and the result wraps beyond 232-1. For example, the result of adding 0xFFFFFFF0 and 0x40 together is 0x30 (48 decimal). When these values are used for c1 and c2, the addition passes the size check, and the code then copies nearly 4GB into a 48-byte buffer. You just got a buffer overrun! Many bugs like this one are exploitable, allowing an attacker to inject code into your process.

When reviewing C and C++ code for integer overflows, I look for all instances of the operator new and dynamic memory allocation functions (alloca, malloc, calloc, HeapAlloc, and so on) and then determine how the buffer size is calculated. I then ask myself questions like the following:

  • Can the values wrap beyond some maximum value?
  • Can the values wrap below zero?
  • Is the data truncated (copying a 32-bit value into a 16-bit value, and then copying the 32-bit size)?

There's a rule of thumb employed by a colleague of mine at Microsoft: if you perform a mathematical operation in an expression that is used in a comparison, then you have a potential to overflow and underflow data. This can be doubly bad if the calculation is used to determine a buffer size, especially if one or more of the buffer-size calculation elements is cracked by an attacker.

Database Access Code in Any Language

As a general rule, developers write database applications in higher-level languages like C#, scripting languages, and the like. Relatively speaking, very little database code is written in C and C++, but some people use various C/C++ class libraries such as the CDatabase class in MFC.

There are two issues that can be detected. The first is connection strings that include hardcoded passwords or connect using administrative accounts. The second issue that can be detected is SQL injection attack vulnerabilities.

When I'm looking at managed code, the first thing I do is search all the code for the System.Data namespace, especially System.Data.SqlClient. The alarm bells ring when I see these! Next, I look for words like "connect" in the code (usually a connection string is close by). The connection string has two interesting properties to look for: the connection id (often, uid) and the password (often, pwd). Something like this is a potential security hole:

DRIVER={SQL Server};SERVER=hrserver;UID=sa;PWD=$esame

Actually, there are two bugs in this example. First, the connection is made as the sysadmin account, sa; this defeats the principle of granting the least privilege necessary. Code should never connect to a database as the sysadmin account because such an account can wreak havoc on the database when used by people with evil intentions. Second, the password is hardcoded. This is wrong for two reasons: first, it will be discovered, and second, what if the password changes? (You would have to update all the clients.)

The next topic is SQL injection attacks. The crux of SQL injection is using string concatenation to build SQL statements. When scanning the code, I look to see where SQL statements are created. Generally speaking, this involves searching for words like "update", "select", "insert", "exec", and any table or database names I know are used. To help out, I use the following ildasm.exe on the managed assembly under scrutiny:

ildasm /adv /metadata /out:file test.exe

I then look at the "User Strings" section in the resulting output. If I find any database query using string concatenation, it's a potential security defect and must be fixed by using parameterized queries instead.

Using string concatenation to build stored procedures is no cure for SQL injection either. In short, string concatenation plus SQL statements is bad, but string concatenation plus SQL statements plus sysadmin account equals disaster.

Web Page Code in Any Language

The most common errors in Web-based applications are cross-site scripting (XSS) issues. Although there are other issues I look for, such as SQL injection and poor cryptography, XSS bugs are quite common. The core XSS vulnerability is the potential to display untrusted user input in the victim's browser, so first I search for any code construct that sends data to the user. For example, in ASP I look for Response.Write and <%= %> tags. Next, I look at the data being written to see where it came from. An XSS bug exists if the data came from an HTTP entity, such as a form or a querystring, and was not checked for validity and is sent to a user's browser. Here's a grossly simple, but not uncommon XSS example:

Hello,
<% Response.Write(Request.QueryString("Name")) %>
As you can see, the "Name" parameter is sent back to the user without first checking that it is valid and well-formed.

Secrets and Cryptography in Any Language

Some developers have a penchant for storing secret data in code, such as passwords and cryptographic keys, and creating their own magic cryptographic algorithms. Don't do either!

I first look for variable names and functions with names that include "key," "password," "pwd," "secret," "cipher," and "crypt." Any hit needs analysis. You can often get false positives for "key," but the others are interesting and may yield embedded secret data, or "magic" cryptography systems. When searching for cryptographic algorithms, I also look for XOR operators, as they are commonly used in crypto. The worst code is code that uses an embedded key to XOR a data stream!

ActiveX Controls in Visual Basic and C++

I always ask one question when I review a new ActiveX® control: why isn't it written in managed code? I ask this question because managed code allows for partial-trust scenarios, which ActiveX does not.

Next, I look at all the methods and properties on the control (the .IDL file is the best source for this) and I put myself in the mindset of a bad guy. What nasty things could I do with each of these methods or properties? Generally, many methods are named with a VerbNoun format, such as ReadRegistry, WriteFile, GetUserName, and NukeKey, so I look for onerous-sounding verbs, and nouns (resources) that pertain to sensitive resources.

For example, a SendFile method is potentially dangerous if the attacker can access any file on the user's hard disk, and send it to any location, such as a Web site under an attacker's control! Anything that accesses resources on the user's machine is subject to further scrutiny.

I do extra review work if the control is marked safe for scripting (SFS) because it could then be invoked in the Web browser without warning the user. You can determine if the control is SFS if it implements the ATL IObjectSafetyImpl interface or sets the following "safe for scripting" or "safe for activation" implementation categories at install time:

[HKEY_CLASSES_ROOT\CLSID\<GUID>\Implemented
Categories\{7DD95801-9882-11CF-9FA9-00AA006C42C4}]
[HKEY_CLASSES_ROOT\CLSID\<GUID>\Implemented
Categories\{7DD95802-9882-11CF-9FA9-00AA006C42C4}]

Earlier I mentioned that accessing and sending users files through a SendFile method is bad. In fact, it's a privacy bug if I can access the SendFile method and determine that a file exists on the user's hard drive based on the error code returned by the method.

Conclusion

This is a very high-level look at the first pass I take when reviewing code. Many of these bugs are simple, and one could argue that developers should not make such errors, but they do. Knowing that your code will be reviewed for security, however, usually leads you to write more secure code in the first place.

You may also have noticed that a common theme with some of the defect types is that many are caused by untrusted input. When reviewing code, you should always ask where that data comes from, and whether you can trust it.

For background information see:
Code Secure
Security Developer Center

Michael Howard is a senior program manager, a founding member of the Secure Windows Initiative group at Microsoft, and a coauthor of Writing Secure Code (Microsoft Press, 2002). He focuses on designing, building, testing, and deploying applications designed to withstand attack, and yet still be usable by millions of non-technical users.


Secure It: WS-Security and Remoting Channel Sinks Give Message-Level Security to Your SOAP Packets

Secure It

WS-Security and Remoting Channel Sinks Give Message-Level Security to Your SOAP Packets

Neeraj Srivastava
This article assumes you're familiar with C# and SOAP
Level of Difficulty 1 2 3
Code download available at: RemotingChannelSinks.exe (119KB)
SUMMARY

As more organizations adopt XML-based Web Services, the need for message-level security has become evident. WS-Security, now supported in the Microsoft .NET Framework, addresses this need. Using the WS-Security framework, developers can implement channel sinks to intercept Remoting messages as they pass through the .NET Remoting infrastructure. The sink can read the message, change it, and pass it along. During this process, the message can be signed for added security. This article explains how to implement a Remoting channel sink that will modify the Remoting message by including a UserName token in the header, then sign the body using the token.

Contents
Channel Sink Architecture
WSE Classes
Implementing the Channel Sink
Client Channel Sink Implementation
ProcessMessage and SignMessage
Server Channel Sink Implementation
Extending the Implementation
Conclusion

O
ver the past few years there has been an explosive growth in the use of XML-based Web Services across the industry. However, some of this expansion has been constrained by a lack of message-level security as a protocol specification. The recently published WS-Security spec is an attempt to remedy this situation. WS-Security is now supported in the Microsoft® .NET Framework through the Web Services Enhancements (WSE) 1.0 for Microsoft .NET classes. WSE provides a framework to secure SOAP messages, and it supports the WS-Routing, DIME, and WS-Attachments specs. With the powerful programming model exposed by WSE, you can work directly with and modify SOAP messages.

Figure 1 Message Generation with WSE
Figure 1 Message Generation with WSE

Channel sinks are a powerful extensibility feature of .NET Remoting that let you intercept a Remoting message as it travels through the Remoting infrastructure. The sink can read the message, modify it, and pass it further down the chain. This article will show you how to implement a Remoting channel sink that will modify the Remoting message by including a UserName token in the header, then signing the body using the token. It makes use of the WSE to do this, so the message is generated according to the WS-Security specification (see Figure 1).

Channel Sink Architecture

Let's look at the architecture of a channel sink by reviewing the various elements involved in a Remoting method call, the different stages at which you can intercept messages, and the things you can do with a message once you've intercepted it.

When you construct a proxy and make a method call, you generate a Message object, which travels through a series of sinks as it makes round-trips to the server. The Message class, which is referenced through the IMessage interface within the sinks, contains a dictionary that holds the data representing the call.

The Remoting framework exposes a number of IMessage implementing classes, which represent different messages passed along either during construction (ConstructionCall, Construction Response) or during a method call (MethodCall, MethodResponse).

There are two important sinks through which the remoted message passes: FormatterSink and TransportSink. The FormatterSink does the job of serializing the message into a stream and creating transport headers. The TransportSink converts the transport headers into protocol-specific headers and transfers the stream over the wire using the chosen protocol. You can put custom sinks both before and after the FormatterSink.

Figure 2 Channel Sink Architecture
Figure 2 Channel Sink Architecture

The sinks between the FormatterSink and the TransportSink have to implement either the IClientChannelSink (if it is a client channel sink) or IServerChannelSink interface (if it is a server channel sink). Figure 2 shows the channel sink architecture and Figure 3 lists the interface definition for these interfaces. Later in the article, I will look into how to implement these interfaces.

Figure 3 Interface Definitions (Close Figure 3)
Public interface IClientChannelSink
{
    IClientChannelSink NextChannelSink {get;}

void AsyncProcessRequest(IClientChannelSinkStack sinkStack,
    IMessage msg, ITransportHeaders headers, Stream stream);

    void AsyncProcessResponse(IClientResponseChannelSinkStack
        sinkStack, object state,ITransportHeaders headers,
        Stream stream);

    Stream GetRequestStream (IMessage msg, ITransportHeaders
        headers);

    void ProcessMessage(IMessage msg, ITransportHeaders
        requestHeaders, Stream requestStream, out
        ITransportHeaders responseHeaders, out Stream
        responseStream);
}

Public interface IServerChannelSink
{
    IServerChannelSink NextChannelSink {get;}

    void AsyncProcessResponse(IServerResponseChannelSinkStack
        sinkStack,object state,IMessage msg, ITransportHeaders
        headers, Stream stream);

    Stream GetRequestStream (IServerResponseChannelSinkStack
        sinkStack,object state,IMessage msg ,ITransportHeaders
        headers);

void ProcessMessage(IServerResponseChannelSinkStack sinkStac,
    IMessage requestMsg, ITransportHeaders requestHeaders,
    Stream requestStream,out IMessage responseMsg ,out
    ITransportHeaders responseHeaders, out Stream
    responseStream);
}

Any sink that's inserted before a FormatterSink can only access the raw message object. Such sinks have to implement IMessageSink. The interface definition of IMessageSink is:

Public interface IMessageSink
{
   IMessageSink NextSink {get;}
   IMessageSink AsyncProcessMessage (IMessage msg,
                                     IMessageSink replySink);
   IMessageSink SyncProcessMessage (IMessage msg);
}

Typically, FormatterSink is the first channel sink in the chain on the client side. FormatterSink implements both IMessageSink and IClientChannelSink/IServerChannelSink by implementing IClientFormatterSink on the client side and IServerFormatterSink on the server side. The formatter sink on the client side receives the message on IMessageSink, serializes it, and forwards it on to IClientChannelSink. On the server side, it reverses this process, sending the message through IServerChannelSink.

Remoting channels are associated with a default chain of sinks. For example, the HttpChannel is associated by default with SoapFormatterSink, and the TcpChannel with BinaryFormatterSink.

Using a configuration file, you can override the sink chain by providing a reference to custom sink providers. A sink provider implements IClientChannelSinkProvider for creating client-side channel sinks and implements IServerChannelSinkProvider for creating server-side channel sinks:

public interface IClientChannelSinkProvider
{
  IClientChannelSinkProvider Next {get; set;}
  IClientChannelSink CreateSink (IChannelSender channel, string url,
      object remoteChannelData);
}

public interface IServerChannelSinkProvider
{
    IServerChannelSinkProvider Next {get; set;}
    IServerChannelSink CreateSink (IChannelReceiver channel);
    void GetChannelData(IChannelDataStore channelData);
}

The provider chain is initialized when the config file is read and the channel initialized. The order in which the providers are placed in the chain depends on the order in which they are defined in the config file. The CreateSink method of the provider is called when the proxy is created. It should forward the call to the next provider in the provider chain, and then chain the next sink (obtained after its own channel sink). This order sets up the sink chain through which the message passes.

WSE Classes

Now let's take a look at WS-Security and employing user name tokens to sign Remoting messages. The WSE classes provide a very good place to start this process. Here they are, along with a description of each.

Microsoft.Web.Services.SoapWebRequest , derived from System.Net.WebRequest, does the job of passing the SOAP message through a series of transforming filters. This class is related to the WSE infrastructure and looks at the information in SoapContext to transform the message.

Microsoft.Web.Services.SoapContext holds the information that is used to transform the message-security tokens, referral information, and attachments. The infrastructure uses this information to perform different transforms on the incoming and outgoing SOAP messages.

Microsoft.Web.Services.SoapEnvelope encapsulates a SOAP envelope class and makes it easy to work with SOAP messages.

Microsoft.Web.Services.Security.Security holds a collection of security tokens (like the user name token) and security elements (like the signature). This class provides the GetXml method, which takes an XML document as input and returns an XML signature that conforms to the WS-Security specification.

Microsoft.Web.Services.Security.Signature holds a signature based on a security token. You can sign different and multiple portions of a message using this class. In this article, I am only signing the SOAP body.

The sink implementation shows how to use these five classes to write and sign the Remoting messages. The sink uses the user name token, which consists of the user name and password. The user name is passed along with the SOAP message; the password is used to sign the message, and may or may not be passed along as well. In the current implementation of this sink, I decided not to send the password in any form.

Implementing the Channel Sink

Let's take a quick look at using the Remoting sink. In order to use the sink, you have to write two classes: one implements the IUserNameTokenHelper interface and the other implements IPasswordProvider.

IUserNameTokenHelper is declared in the SecuritySink namespace, which is available in the code download for this article. The client sink calls the GetUserName and GetPassword methods of this interface to retrieve the user name and password for signing the SOAP message. In this implementation, I provide a property of client sink provider (see TokenHelper) which refers to an object that implements this interface.

The IPasswordProvider interface is declared in the Microsoft.Web.Services.Security namespace. The WSE framework calls this interface to get the password corresponding to a user name while validating a signature. This password is used to sign the SOAP message by comparing its signature value to the one received in the SOAP message. The class that implements this interface is specified in the WSE section of the app.config file.

In addition, you need to make sure that you position the sink correctly in the provider sink chain within app.config. The sink should be positioned after the Formatter in the client app.config file and before the Formatter in the server app.config file. This is because you need to work with the message after it has been serialized on the client side but before it gets deserialized on the server side. Figure 4 and Figure 5 show the relevant portions from each config file. Figure 4 shows the client config file, in which I reference in the client channel sink provider (in the Provider section). The Provider section also refers to the TokenHelper object as one of the properties of the provider.

Figure 5 Server Configuration (Close Figure 5)
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="microsoft.web.services"
  type="Microsoft.Web.Services.Configuration.WebServicesConfigura
  tion, Microsoft.Web.Services, Version=1.0.0.0,
  Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</configSections>
<microsoft.web.services>
       <security>
            <passwordProvider
              type="SecuritySink.PasswordProviderClass,
              SecuritySink" />
        </security>
 </microsoft.web.services>
<system.Runtime.Remoting>
    <application name="AppName">
      <service>
        <!--<activated type="TestLib.remoteobj,TestLib" />-->
        <!--<activated type="TestLib.remoteobj,TestLib" />-->
        <wellknown type="TestLib.remoteobj,TestLib"
          mode="SingleCall" objectUri="Object3.soap" />
      </service>
      <channels>
        <channel ref="tcp" port="9999">
            <serverProviders>
                <provider
                type=
               "SecuritySink.SecurityServerChannelSinkProvider,
                SecuritySink" />
                <formatter ref="soap" />
            </serverProviders>
        </channel>
        <!--<channel ref="http" port="8888" />-->
      </channels>
    </application>
  </system.Runtime.Remoting>
 </configuration>
Figure 4 Configuration Files (Close Figure 4)

Client Config for Remoting
<channels>
  <channel ref="http" port="9988">
    <clientProviders>
    <formatter ref="soap" />
     <provider type="SecuritySink.SecurityChannelSinkProvider,
       SecuritySink"
       TokenHelper="SecuritySink.TokenHelperObj,SecuritySink"/>
    </clientProviders>
  </channel>
 </channels>
Server Config for Remoting
<channels>
  <channel ref="http" port="9999">
    <serverProviders>
     <provider
       type="SecuritySink.SecurityServerChannelSinkProvider,
       SecuritySink" />
     <formatter ref="soap" />
    </serverProviders>
  </channel>
<channels/>
Server app.config for WSE
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <configSections>
    <section name="microsoft.web.services"
      type="Microsoft.Web.Services.Configuration.
      WebServicesConfiguration,Microsoft.Web.Services,
      Version=1.0.0.0, Culture=neutral,
      PublicKeyToken=31bf3856ad364e35" />
    </configSections>
<microsoft.web.services>
     <security>
          <passwordProvider
            type="SecuritySink.PasswordProviderClass,
            SecuritySink" />
      </security>
 </microsoft.web.services>
 </configuration>

Note that the config files on the server side can be merged into a single file. To do this, you would need to move the Remoting configuration section in the application config file and load it on the server while configuring Remoting. Figure 5 shows the server config file. As you can see, the server channel sink provider is referenced under the provider section. The order in which you specify the formatter and sink providers is the order in which they get called in the sink chain.

Client Channel Sink Implementation

When implementing a Remoting sink, the client channel sink provider is a good place to start as it will create the client channel sink on demand. The client channel sink provider implements the IClientChannelSinkProvider interface. The provider also has to implement a constructor that takes two parameters: a dictionary object and a collection of SinkProviderData. The dictionary object holds the properties of the sink provider as specified in the config file. The collection holds further configuration information that's specified in the config file under the <provider> element. The string corresponding to the user name token is passed in through the properties collection, and the provider passes this value to the sink.

The method of most interest on the sink provider is called CreateSink, which (as its name suggests) creates the sink and returns a reference to it. Since you're creating a chain of sinks in a specific order, you must take the sink reference you obtain here and pass it to the CreateSink method of the next provider in the chain. Your sink then stores the returned reference to the next chained sink and calls upon it when it needs to pass messages downstream for processing.

Here is the code for the implementation of this method:

public IClientChannelSink CreateSink(IChannelSender channel,
                                     String url,
                                     Object remoteChannelData)
{
    IClientChannelSink nextSink = null;
        if (_next != null)
        {
            nextSink = _next.CreateSink(channel, url,
                remoteChannelData);
            if (nextSink == null)
                return null;
        }
            return new
                SecurityClientChannelSink
                (nextSink,_tokenHelperString);
}
The ClientChannelSink, which is derived from BaseChannelSinkWithProperties, is the base class for sinks that implement named properties. (This sink has to implement the named properties Keys and Indexer.) The object must also implement the IClientChannelSink interface. Implementing IClientChannelSink means that you have to implement a ProcessMessage method for supporting synchronous processing as well as AsyncProcessRequest and AsyncProcessResponse procedures for supporting asynchronous processing. The core of the processing has been delegated to a private method, SignMessage, which does the core job of signing the SOAP message.

Before delving into the implementation of ProcessMessage, let's look into the implementation of the client channel sink's GetRequestStream method. GetRequestStream is supposed to return a stream on which the previous sink in the chain can write. If you return null, it implies that the previous sink in the chain should create its own stream to start the chain and pass it to the next sink. If you are in any way going to change or seek the stream in your implementation of the channel sink, it is recommended that you return null in your GetRequestStream implementation.

ProcessMessage and SignMessage

The ProcessMessage function handles the processing for a message before passing it on to the next sink. This preprocessing is done by delegating to the private method SignMessage. No post-processing of the response is performed.

SignMessage creates a SoapEnvelope object in WSE based on the stream that is passed in. It also creates a UsernameToken helper object from the string read from the config file and passed in by the sink provider.

This method gets the user name and password from this object:

string helperObj = _tokenHelperString;
string[] strarry = helperObj.Split(new Char[]{','});
ObjectHandle obj  = Activator.CreateInstance(strarry[1],strarry[0]);
IUserNameTokenHelper tokHelper = obj.Unwrap() as IUserNameTokenHelper;
string usr = tokHelper.GetUserName();
string pass =tokHelper.GetPassword();
A UserNameToken is created based on this user name and password. You then create a security object and add the UserNameToken to its collection of security tokens. Next, create the signature object and initialize it with the UsernameToken that was just created. This signature object is added to the security elements collection of the security object.

The SOAP envelope is then signed by passing it to the GetXml method of the security object. This call returns a WS-Security Header element, which is appended to the SOAP envelope's header. Finally, the new SOAP message is saved to the outgoing stream. The code in Figure 6 shows this implementation.

Figure 6 Processing a Message (Close Figure 6)
//Stream returnStream;
//load the soap envelope

SoapEnvelope reqEnv = new SoapEnvelope();
reqEnv.Load(requestStream);

//setup the username token

UsernameToken tok = new
  UsernameToken(tokHelper.GetUserName(),tokHelper.GetPassword());
tok.PasswordOption = PasswordOption.SendNone;

//create the security object

Security sec = new Security(actor);
sec.Tokens.Add(tok);
if(reqEnv.Header == null)
reqEnv.CreateHeader();

//signature

Signature sig = new Signature(tok);
sec.Elements.Add(sig);

//sign

reqEnv.Header.AppendChild(sec.GetXml(reqEnv));

//save

if(inStream == null)
{
    inStream = new MemoryStream();
    reqEnv.Save(inStream);
    inStream.Position=0;
}
else
{
    reqEnv.Save(inStream);
}

The async implementation of this process differs little. In AsyncProcessRequest, you push the current sink on the sink stack so that it will be called during the async processing of the response. You don't have to do any post-processing on a message; instead, you simply delegate the call to the next sink.

Server Channel Sink Implementation

The implementation of the ServerChannelSinkProvider is similar to that of the client provider. You pass in the reference to the channel object, which is required by the sink implementation. Server-side sinks don't currently support asynchronous processing at the request stage; however, the response processing can be asynchronous, allowing you to implement only the ProcessMessage and AsyncProcessResponse methods. The server response is not being processed in the current implementation of the channel sink, so the response is let through the sink chain. Keeping this in mind, the implementation of the GetResponseStream method just delegates to the next sink in the sink stack for processing. The processing of the AsyncProcessResponse method is similarly trivial, as it also delegates to the next sink in the sink stack.

In the ProcessMessage method, you must first ensure that the method has not already been deserialized. You then create a SOAP envelope that is based on the stream passed in and extract the WS-Security headers from the SOAP message. When you create a security object based on the XML in each of the headers, the signature is then validated.

The WSE processing includes a call to an object that implements IPasswordProvider; this object retrieves the password corresponding to the user name that appears in the header. The particular object that implements IPasswordProvider is user specified in the WSE config file, under <microsoft.web.services> ... <passwordProvider>. For more information, take a look at the config files in the sample code for this article. You must also make sure that you delete the Security headers from the SOAP message before passing it down the sink chain, as shown in Figure 7.

Figure 7 Security Processing (Close Figure 7)
SoapEnvelope reqEnv = new SoapEnvelope();
reqEnv.Load(requestStream);

//handle multiple security elements also

XmlNodeList nodelist;
IEnumerator nodelistEnum;
    nodelist = reqEnv.Header.GetElementsByTagName(@"Security",
        @"http://schemas.xmlsoap.org/ws/2002/07/secext");
nodelistEnum = nodelist.GetEnumerator();
ArrayList nodestodelete = new ArrayList();
try
      {
    while(nodelistEnum.MoveNext())
        {
        XmlElement elem = nodelistEnum.Current as XmlElement;

//Create the security object
Security secserver = new Security(elem);
//can't delete the node as the enumeration list will change so
//store the node
                nodestodelete.Add(elem);
        }

//delete the security nodes from the header

IEnumerator nodestodeleteEnum = nodestodelete.GetEnumerator();
while(nodestodeleteEnum.MoveNext())
        {
            reqEnv.Header.RemoveChild((XmlElement)nodestodeleteEnum.Current);
        }
MemoryStream strm = new MemoryStream();
reqEnv.Save(strm);
strm.Position=0;
processing =    _nextSink.ProcessMessage(sinkStack, null,
                requestHeaders, strm, out responseMsg,
                out responseHeaders, out responseStream);
   }

Exception handling is very important for server-side channel sinks; an exception thrown here propagates back to the transport sink, which will handle it and terminate the connection. You must handle the exception and return an appropriate SOAP fault message along with appropriate HTTP error codes. The transport header keys for modifying the HTTP response codes are __HttpStatusCode and __HttpReasonPhrase.

In the exception handler, you create a SOAP fault message based on the exception and serialize it to the response stream, setting the appropriate HTTP headers to indicate a server-side error. If you're on a TCP channel, only the SOAP fault message is passed back and no headers are set.

Extending the Implementation

One of the possible extensions to this sink is to implement the equivalent of impersonation. Once the signature is validated on the server, you have a valid security object on the server side and you can get the UserName token from this security object. The sender's user name can be obtained from the UserName token with the following code snippet:

SecurityTokenCollection col = reqCtx.Security.Tokens;
IEnumerator e = col.GetEnumerator();

e.MoveNext(); // we expect only one

UsernameToken tokn = (UsernameToken) e.Current;
string username = tokn.Username;
GenericIdentity idn = new GenericIdentity (username);
GenericPrincipal p = new GenericPrincipal (idn, null);

System.Threading.Thread.CurrentPrincipal = p;
The previous code snippet can be extended to implement role-based security on the server. On the client side of the equation, you can use the thread principal to get the user's identity. Another possible implementation would be to write a Remoting sink for encrypting SOAP messages, again using WSE.

Conclusion

The implementation of client and server Remoting sinks is made very easy thanks to the new WSE classes which are compatible with the .NET Framework. The classes make signing and encrypting SOAP messages quite simple, and the architecture of the sink chain exposes an easy path to extend your message processing.

For related articles see:
Programming with Web Services Enhancements 1.0 for Microsoft .NET
Remoting Example: Channel Sink Provider
Advanced .NET Remoting by Ingo Rammer (APress, 2002)

Neeraj Srivastava works in the Solutions Integration Engineering group at Microsoft Product Support Services. He has been a developer for about eight years, working primarily with Microsoft technologies. Currently, he specializes in middleware technologies such as COM+ Remoting, Web Services, and ASP.NET.


Obfuscate It: Thwart Reverse Engineering of Your Visual Basic .NET or C# Code

Obfuscate It

Thwart Reverse Engineering of Your Visual Basic .NET or C# Code

Gabriel Torok and Bill Leach
This article assumes you're familiar with .NET and C#
Level of Difficulty 1 2 3
SUMMARY

One of the advantages of the .NET architecture is that assemblies built with it contain lots of useful information that can be recovered using ILDASM, the intermediate language disassembler. A side effect, though, is that someone with access to your binaries can recover a good approximation of the original source code. Here the authors present program obfuscation as a way to deter reverse engineering. In addition, they discuss the different types of obfuscation technologies available and demonstrate the new obfuscation tool that is included in Visual Studio .NET 2003.

Contents
Disassembly
Decompilation
Obfuscation in Depth
Renaming Metadata
Removing Nonessential Metadata
Additional Techniques
Using Dotfuscator Community Edition
Examining the Map File
Obfuscator Pitfalls
Conclusion

B
y now you are probably familiar with all of the benefits that the metadata-rich Microsoft® .NET Framework architecture brings to the table, from easing the burdens of deployment and versioning to the rich IDE functionality enabled by self-describing binaries. You may not know that the easy availability of all this metadata has introduced a problem that until now was not a concern for most developers. Programs written for the common language runtime (CLR) are easier to reverse engineer. This is not in any way a fault in the design of the .NET Framework; it is simply a reality of modern, intermediate-compiled languages (Java-language applications display the same characteristics). Both Java and the .NET Framework use rich metadata embedded inside the executable code: bytecode in the case of Java, Microsoft Intermediate Language (MSIL) in .NET. Being much higher level than binary machine code, the executable files are laden with information that can be easily deciphered.

With the help of tools like ILDASM (the MSIL disassembler that ships with the .NET Framework SDK) or decompilers such as Anakrino and Reflector for .NET, anyone can easily look at your assemblies and reverse engineer them back into readable source code. Hackers can search for security flaws to exploit, steal unique ideas, and crack programs. This should be enough to give you pause.

Don't worry, though. There's a solution—obfuscation—that will help you thwart reverse engineering. Obfuscation is a technique that provides for seamless renaming of symbols in assemblies as well as other tricks to foil decompilers. When it is properly applied, obfuscation can increase the protection against decompilation by many orders of magnitude, while leaving the application intact. Obfuscation is commonly used in Java environments and for years has been helping companies protect the intellectual property in their Java-based products.

Several third-parties have answered the call by creating obfuscators for .NET code. Microsoft includes the Dotfuscator Community Edition with Visual Studio® .NET 2003 in partnership with our company PreEmptive Solutions, which ships a number of various obfuscator packages.

Using the Dotfuscator Community Edition, this article will teach you all about obfuscation (and a little about decompilation), the types of obfuscation commonly available, and some of the issues you will need to address when working with an obfuscator.

To demonstrate decompilation and obfuscation, we are going to use an open-source implementation of the classic Vexed game. Vexed.NET was written by Roey Ben-amotz and is available at http://vexeddotnet.benamotz.com. It's a puzzle game in which your goal is to move similar blocks together, which causes them to disappear. Below is a simple method from the source code of Vexed.NET:

public void undo() {
  if (numOfMoves>0) {
    numOfMoves--;
    if (_UserMoves.Length>=2)
        _UserMoves = _UserMoves.Substring(0, _UserMoves.Length02);
    this.loadBoard(this.moveHistory[numOfMmoves -
                      (numOfMoves/50) * 50]);
    this.drawBoard(this.gr);
  }
}

Disassembly

The .NET Framework SDK ships with a disassembler utility called ILDASM, which allows you to decompile .NET Framework assemblies into IL Assembly Language statements. In order to start ILDASM, you must make sure that the .NET Framework SDK is installed and type ILDASM on the command line followed by the name of the program that you want to disassemble. In our case, we will type "ILDASM vexed.net.exe". This will launch the ILDASM UI, which can be used to browse the structure of any .NET Framework-based application. Figure 1 shows the undo method disassembled.

Figure 1 The Undo Method Disassembled (Close Figure 1)

Decompilation

If you're now thinking that only a small circle of folks who actually know IL Assembly Language will see and understand your source code, remember that the decompilation doesn't stop there. We can recreate the actual source code by using a decompiler. These utilities can decompile a .NET assembly directly back to a high-level language like C#, Visual Basic® .NET, or C++. Let's look at the undo method generated by the Anakrino decompiler:

public void undo() {
  if (this.numOfMoves > 0) {
    this.numOfMoves =
      this.numOfMoves - 1;
    if (this._UserMoves.Length >= 2)
      this._UserMoves =
           this._UserMoves.Substring(0, this._UserMoves.Length - 2);
      this.loadBoard(
           this.moveHistory[this.numOfMoves -
               this.numOfMoves / 50 * 50]);
      this.drawBoard(this.gr);
    }
}
As you can see, the results are almost identical to the original code. Later, we will revisit this to see the results after obfuscation.

Obfuscation in Depth

Obfuscation is accomplished using a set of related technologies. Its goal is to hide the intent of a program without changing its runtime behavior. It's not encryption, but in the context of .NET code, it might be better. You could encrypt .NET assemblies to make them completely unreadable. However, this methodology suffers from a classic dilemma—since the runtime must execute unencrypted code, the decryption key must be kept with the encrypted program. Therefore, an automated utility could be created to recover the key, decrypt the code, and then write out the IL to disk in its original form. Once that happens, the program is fully exposed to decompilation.

To give an analogy, encryption is like locking a six-course meal into a lockbox. Only the intended diner (in this case, the CLR) has the key and we don't want anyone else to know what he or she is going to eat. Unfortunately, at mealtime the food will be in plain view to all observers. Obfuscation works more like putting the six-course meal into a blender and sending it to the diner in a plastic bag. Sure, everyone can see the food in transit, but besides a lucky pea or some beef-colored goop, they don't know what the original meal is. The diner still gets the intended delivery and the meal still provides the same nutritional value as it did before (luckily, the CLR isn't picky about taste). The trick of an obfuscator is to confuse observers, while still delivering the same product to the CLR.

Of course, obfuscation (or encryption) is not a hundred percent foolproof. Even compiled C++ can be disassembled. If a hacker is persistent enough, she can reproduce your code.

Figure 2 Obfuscation Process
Figure 2 Obfuscation Process

Obfuscation is a process that is applied to compiled .NET assemblies, not source code. An obfuscator never reads or alters your source code. Figure 2 shows the flow of the obfuscation process. The output of the obfuscator is another set of assemblies, functionally equivalent to the input assemblies, yet transformed in ways that hinder reverse engineering. We will now consider two essential techniques that Dotfuscator Community Edition uses to achieve that goal: renaming and removing nonessential metadata.

Renaming Metadata

The first line of defense in obfuscation is to rename meaningful names with non-meaningful ones. As you know, there is a lot of value in well-chosen names. They help make your code self-documenting and serve as valuable clues that reveal the purpose of the item they represent. The CLR doesn't care how descriptive a name is, so obfuscators are free to change them, typically to one-character names like "a".

Obviously there are constraints on the amount of renaming an obfuscator will be able to perform on a particular application. Generally speaking, there are three common renaming scenarios.

If your application consists of one or more assemblies that are standalone (that is, no unobfuscated code depends on any of the assemblies), then the obfuscator is free to rename an assembly regardless of the name's visibility, so long as the names and references to them are consistent across the set of assemblies. A Windows® Forms application is a good example of this. At the opposite extreme, if your application is designed to be used by unobfuscated code, the obfuscator cannot change the names of types or members visible to those clients. Examples of this type of application are shared class libraries, reusable components, and the like. Somewhere in between are applications that are meant to plug into existing unobfuscated frameworks. In this case, the obfuscator can rename anything not accessed by the environment in which it is running, regardless of visibility. ASP.NET applications are good examples of this type of application.

Dotfuscator Community Edition uses a patented renaming technique called overload induction that adds a twist to renaming. Method identifiers are maximally overloaded after an exhaustive scope analysis. Instead of substituting one new name for each old name, the overload induction technique renames as many methods as possible to the same name, confusing anyone trying to understand the decompiled code.

In addition, as a nice side effect, the size of the application decreases due to the smaller size of the string heap contained in the assembly. For example, if you have a name that is 20 characters long, renaming it to "a" saves 19 characters. In addition, continually reusing names saves space by conserving string heap entries. Renaming everything to "a" means that "a" is stored only once, and each method or field renamed to "a" can point to it. Overload induction enhances this effect because the shortest identifiers are continually reused. Typically, an overload-induced project will have up to 35 percent of the methods renamed to "a".

To see the impact of renaming on decompiled code, take a look at the undo method after the renaming process:

public void c() {
    if (this.p > 0) {
        this.p = this.p - 1;
        if (this.r.Length >= 2)
            this.r = this.r.Substring(0, this.r.Length - 2);
        this.a(this.q[this.p - this.p / 50 * 50]);
        this.a(this.e);
    }
}
You can see that without any other kinds of obfuscation, this method is already much more difficult to understand.

Removing Nonessential Metadata

Not all of the metadata in a compiled .NET-based application is used by the runtime. Some of it is there to be consumed by other tools such as designers, IDEs, and debuggers. For example, if you define a property called "Size" on a type in C#, the compiler will emit metadata for the property name "Size" and associate that name with the methods that implement the get and set operations (which it names "get_Size" and "set_Size", respectively). When you write code that sets the Size property, the compiler will always generate a call to the method "set_Size" itself and will never reference the property by its name. In fact, the name of the property is there for the IDE and developers who are using your code; it is never accessed by the CLR.

If your application is meant to be used by just the runtime and not by other tools, it's safe for an obfuscator to remove this type of metadata. In addition to property names, event names and method parameter names fall into this category. Dotfuscator Community Edition removes all these types of metadata when it deems that it is safe to do so.

Additional Techniques

Dotfuscator Community Edition provides good obfuscation using the techniques we've just described, but you should be aware of additional obfuscation techniques that provide even stronger protection and may foil reverse engineering altogether. Dotfuscator Professional Edition implements many additional techniques, including control flow obfuscation, string encryption, incremental obfuscation, and size reduction.

Control flow is a powerful obfuscation technique, the goal of which is to hide the intent of a sequence of instructions without changing the logic. More importantly, it is used to remove the clues that decompilers look for in order to faithfully reproduce high-level source code statements, such as if-then-else statements and loops. In fact, this technique tends to break decompilers.

To see this effect in action, look at the decompiled undo method again, after applying renaming and control flow obfuscation (see Figure 3). You can see that instead of the original nested if statements, the decompiler has produced an if statement, two nested while loops, and some gotos to tie it all together. The label i1 is referenced but it is not generated by the decompiler (this is a decompiler bug, we presume).

Figure 3 Undo Method After Renaming (Close Figure 3)
public void c() {
    if (this.o > 0) {
        goto i0;
        do {
            while (true) {
                this.a(this.p[this.o - this.o / 50 * 50]);
                this.a(this.d);
                goto i1;
                i2: this.q = this.q.Substring(0, this.q.Length - 2);
            }
            i0: this.o = this.o - 1;
        } while (this.q.Length < 2);
        goto i2;
    }
}

String encryption is a technique that applies a simple encryption algorithm to string literals embedded in your application. As mentioned before, any encryption (or specifically, decryption) that's performed at run time is inherently insecure. That is, a smart hacker can eventually break it, but for strings in application code, it is worthwhile. Let's face it, if hackers want to get into your code, they don't blindly start searching renamed types. They probably do searches for "Invalid License Key" which point them right to the code where license handling is performed. Searching on strings is incredibly easy; string encryption raises the bar because only the encrypted version is present in the compiled code.

Incremental Obfuscation helps with the challenge of issuing a patch to fix a customer's problems in the face of obfuscation. Fixing bugs in code often creates or deletes classes, methods, or fields. Changing code (for example, adding or deleting a method) may cause subsequent obfuscation runs to rename things slightly differently. What was previously called "a" might now be called "b". Unfortunately, how and what was renamed differently is a mystery.

Incremental obfuscation can combat this problem. Dotfuscator creates a map file to tell you how it performed the renaming. That same map file, however, can be used as input to Dotfuscator on subsequent runs to dictate that renames used previously should be used again wherever possible. If you release your product and then patch a few classes, Dotfuscator can be run in such a way as to mimic its previous renaming scheme. That way, you can issue just the patched classes to your customers.

Size reduction does not strictly impede reverse engineering, but it is worth mentioning because obfuscators almost always have to perform a dependency analysis on the set of input assemblies. Thus the obfuscator is in a good position to do more than obfuscate, and some of the better ones will use their knowledge of your application to remove code that your program is not using. It seems odd that unused code removal can actually do anything—who writes code they don't use? Well, the answer is all of us. What's more, we all use libraries and types written by other people that were written to be reusable.

Reusable code implies there is contingent code that handles many cases; however, in any given application, you typically only use one or two of those many cases. An advanced obfuscator can determine this and strip out all the unused code (again, from the compiled assembly, not the source). The result is that the output contains precisely the types and methods your application needs—nothing more. A smaller application has the benefits of conserving computing resources and reducing load times. This can be especially important for apps running on the .NET Compact Framework or distributed applications.

Using Dotfuscator Community Edition

Now let's use Dotfuscator Community Edition to obfuscate the Vexed application. Dotfuscator Community Edition uses a configuration file that specifies the obfuscation settings for a particular application. It has a GUI to help you easily create and maintain the configuration file as well as run the obfuscator and examine the output. In addition, the Dotfuscator Community Edition's command-line interface allows you to easily integrate obfuscation into your automated build process. You can launch the GUI right from the tools menu of Visual Studio .NET 2003.

To configure Vexed for obfuscation, you need to specify three items in the Dotfuscator Community Edition GUI: the input assembly, the map file location, and the output directory. The input assemblies (Dotfuscator calls these "trigger assemblies") are specified on the Trigger tab. You can add as many here as you want, but you only need one for the Vexed application.

You specify the map file location on the Rename | Options tab (see Figure 4). The map file is an essential piece of information that contains the unambiguous name mappings between the original and unobfuscated names. It is very important to keep this file after you obfuscate your application; without it, you will not be able to easily troubleshoot the obfuscated app. Due to its importance, Dotfuscator will not overwrite an existing map file by default unless you explicitly check the "Overwrite Map file" box.

Figure 4 Rename | Options Tab (Close Figure 4)

Finally, the Build tab allows you to specify the directory where the obfuscated application will be placed. Once you have done that, you are ready to obfuscate the application. You can save your configuration file for later use, then either press the "Build" button on the Build tab or use the "Play" button on the toolbar. While building, Dotfuscator displays progress information in the GUI's output pane. You can control the amount of information that is displayed here by choosing Quiet or Verbose on the Options tab.

Once the build is complete, you can visually explore the results on the Output tab, shown in Figure 5. As you can see, Dotfuscator displays a graphical view of the application similar to an object browser. The new names are immediately below the original names in the view. In the figure, you can see that the class named "board" was renamed to "h", and two methods with different signatures (init and ToImage) were both renamed "a".

Figure 5 Output Browser (Close Figure 5)

Examining the Map File

The map file that Dotfuscator produces is an XML-formatted file, and in addition to the already mentioned name mappings, it contains some statistics that give a sense of how effective the renaming process was. Figure 6 summarizes the statistics for types and methods after obfuscating the Vexed application.

Figure 6 Renaming Statistics (Close Figure 6)

 | Total | Number Renamed | % Renamed | % Methods Renamed to 'a'
Classes | 20 | 20 | 100% | -
Methods | 164 | 144 | 88% | 39%

Map files are also used to perform incremental obfuscation. This process allows you to import names from a previous run, which tells the obfuscator to perform renaming in the same way as it was performed previously. If you are releasing a patch (or a new plug-in) for an already obfuscated application, you can obfuscate the updates using the same name set as the original version. This is of particular interest to enterprise development teams maintaining multiple interdependent applications.

Obfuscator Pitfalls

Obfuscation—especially renaming—can be tricky on complex applications and is highly sensitive to correct configuration. If you aren't careful, the obfuscator can break your application. In this section, we'll discuss some of the more common issues that can arise when using an obfuscator.

First, you need to do a little more work when your application includes a strongly named assembly. Strongly named assemblies are digitally signed, allowing the runtime to determine if an assembly has been altered after signing. The signature is an SHA1 hash signed with the private key of an RSA public/private key pair. Both the signature and the public key are embedded in the assembly's metadata. Since an obfuscator modifies the assembly, it is essential that signing occur after obfuscation. You should delay-sign the assembly during development and before obfuscation, then complete the signing process afterward. See the .NET Framework documentation for more details about delay-signed assemblies, and remember to turn off strong name validation while testing your delay-signed assemblies.

The use of the Reflection API and dynamic class loading will also complicate the obfuscation process. Since these facilities are dynamic, they tend to defeat the static analysis techniques used by most obfuscators. Consider the following C# code snippet that gets a type by name and dynamically instantiates it, returning the type cast to an interface:

public MyInterface GetNewType() {
    Type type = Type.GetType( GetUserInputString(), true );
    object newInstance = Activator.CreateInstance( type );
    return newInstance as MyInterface;
}

The name of the type is coming from another method. GetUserInputString may be asking the user to enter a string, or perhaps it retrieves it from a database. Either way, the type name is not present in the code for a static analysis to recover, so there is no way of knowing which types in the input assemblies may be instantiated in this manner. The solution in this case is to prevent renaming of all potentially loadable types that implement MyInterface (note that method and field renaming can still be performed). This is where manual configuration and some knowledge of the application being obfuscated plays an important role. Dotfuscator Community Edition gives you the tools to prevent the renaming of select types, methods, or fields. You can pick and choose individual names; alternatively, you can write exclusion rules using regular expressions and other criteria, such as visibility on scope. For example, you could exclude all public methods from renaming.

Another issue with using an obfuscator occurs after you have deployed an obfuscated application and you are trying to support it. Say your application is throwing an exception (which happens even to the best of us) and a customer sends you a stack dump that looks something like this:

System.Exception: A serious error has occurred
   at cv.a()
   at cv..ctor(Hashtable A_0)
   at ar.a(di A_0)
   at ae.a(String[] A_0)
Obviously this is a lot less informative than a stack dump from the unobfuscated program. The good news is that you can use the map file generated during obfuscation to decode the stack trace back to the original. The bad news is that there is sometimes not enough information in the stack trace to unambiguously retrieve the original symbols from the map file. For example, notice in the dump that the method return types are omitted. In applications obfuscated with an enhanced overload induction renaming algorithm, methods that differ only by return type may be renamed to the same name. So the stack trace can be ambiguous. Most of the time, you can narrow the possibilities enough to find the original names to a high degree of certainty. To help, Dotfuscator Professional provides a tool to automatically translate the stack trace back to the original offending method.

Conclusion

You don't need to let hackers use the handy ILDASM utility on your app for questionable purposes. You can protect your code with a good obfuscator. Obfuscation raises the reverse engineering bar. In the Visual Studio .NET 2003 box, Dotfuscator Community Edition makes good obfuscation just a few clicks away.

For related articles see:
Inside Microsoft .NET IL Assembler by Serge Lidin (Microsoft Press, 2002)
Dotfuscator FAQ

For background information see:
Ildasm.exe Tutorial
Anakrino
http://vexeddotnet.benamotz.com
http://www.preemptive.com

Gabriel Torok is President of PreEmptive Solutions. He is a coauthor of JavaScript Primer Plus and of Java Primer Plus, both published by Macmillan. Gabriel has given talks and tutorials at software development conferences around the world.

Bill Leach is the Chief Technical Officer at PreEmptive Solutions. He serves as architect and technical lead for the Dotfuscator product line. Bill has served as a technical reviewer for software development books and articles.


Editor's Note: Security and Whips

Editor's Note
Security and Whips

Last September, MSDN Magazine published its first-ever issue devoted entirely to the topic of security. Feedback was so positive that we thought "Hey! We can do this again next year!" It's this kind of forward thinking that's behind this year's security issue.

But the truth is, there's so much material to cover that we could do an issue like this every year. We've always been aware of the potentially destructive effects of worms and viruses on our system. However, developer awareness of security issues has really increased over the past couple of years. Since so many applications now send and receive data over the Internet, most apps are now at risk of intrusion. Buffer overruns that used to just crash desktop programs can now be exploited to execute hostile code. If you connect to a port, a script kiddie will find you. If you don't connect to anything, they might still find you through another program.

In most cases, however, security strategy is considered something for IT to deal with. That's backward thinking, and it causes an endless cycle of fire drills as various operating systems and apps fall victim to exploits. (And make no mistake here—despite what you might read, it's not just a problem with Windows, as a quick look through BUGTRAQ (http://www.securityfocus.com/archive/1) will make clear.) Relying upon IT to keep your systems secure is not always going to cut it, especially if they don't have control of your production code. Just like anti-crime programs that address the roots of violence are more successful in lowering crime rates than post-incident enforcement, the best path to eliminating security bugs is to nail them at the source—your code.

The problem with baking security into the code, of course, is that it's not the mindset of most programmers today. Sometimes the fastest and easiest implementation of an idea isn't the most secure. It's enough of a drag to use assert statements and try/catch blocks, and now you're supposed to check strcpy buffers for overflows? We didn't intern in front of a VT100 for three weeks for this kind of treatment!

No one ever said it was going to be easy. However, we've put together lots of helpful tips for you in this issue, from an overview of data-hiding techniques to a firsthand account of how one expert performs security code reviews on products. We now use many of these guidelines to check the code that goes into our magazine. So many tips for writing secure code seem obvious once you hear them, but you'd never know it from looking at some of the programs submitted to us.

And remember, even if you're writing managed code, you can run into problems. Your code won't be able to overrun a memory buffer by itself, but once you go outside your world and send data to another program (even a database) all bets are off. A false sense of security that any unreviewed code won't cause too much damage can lead to sloppiness and potential holes.

This issue of MSDN Magazine is only one of many available security resources. We've published a lot of security-related articles over the past several years; you can browse the list online with our advanced topic listings. You should also check out the MSDN Security Developer Center and the Patterns & Practices site.

And now for the fun portion of the Editor's Note. We get a lot of press releases floating across our New Stuff desk each month. To keep the volume down a bit, we will normally filter out the stuff that's not related to our major areas of focus (building programs with Microsoft tools and technologies). We get press releases from churches, sheet metal manufacturers, limo services in Hong Kong, you name it. However, last week we received one press release that, while not strictly up our alley, caught our eye.

The Guinness World Record for "World's Fastest Whip" was set by whip expert and coach Robert Dante on September 2 at the Dream Circus in Hollywood. Dante set the record by cracking a whip 203 times in one minute, using a 6-foot bullwhip made by Australian whip maker Mike Murphy. The attempt was timed to coincide with the launch of Dante's weekly bullwhip academy at the Dream Circus in Hollywood. For more information, check them out at http://www.wildwestperformers.com.

If you've admired Indiana Jones, or Zorro, or even the Catwoman, this class is your chance to feel the thrill of gracefully and safely cracking a bullwhip.

If this whole magazine thing doesn't pan out for us, at least we've finally figured out a possible backup plan! Now, to fit into this Catwoman suit...

—J.T.

Active Directory, ActiveX, JScript, Microsoft, Microsoft Press, MSDN, Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Server is a trademark of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Test Deployment Security
Defense Against the Dark Arts
Macromedia Rolls Out MX 2004
Manage Bugs and Issues
Translate from CFML to .NET
Defining Business Rules
Automate Program Documentation
Charting in .NET
The Bookshelf

Test Deployment Security

Desaware is shipping CAS/Tester (automated code access security testing) for Visual Studio® .NET. CAS/Tester was developed to address the issue of security constraints as software is deployed through the Internet, intranets, and other distribution channels. Code access security is a key feature of the Microsoft® .NET Framework that enables administrators of each machine (via configuration settings) to decide exactly which types of operations an assembly can perform. CAS/Tester automates the process of testing an assembly by executing it under multiple security configurations and producing a report of the results.

Developers can use the command-line utility included with the product, or a GUI-based test add-in that is integrated into Visual Studio .NET 2003. Users can create scripts in Visual Basic® .NET or Visual C#® .NET. CAS/Tester supports testing of class libraries, components, user controls, Windows® Forms and console apps, and can invoke outside test assemblies or test harnesses. The output contains detailed information including permissions tested, and all exceptions that occur for each test, along with a stack trace. The reports can be customized by using XSLT templates.

http://www.desaware.com

Defense Against the Dark Arts

Flicks Software offers a comprehensive suite of solutions for protecting Windows servers against brute-force attacks. One potential barrier is WebQuota, an authentication solution for Windows servers (Microsoft Internet Information Services 3.0 and above), which includes dictionary attack protection.

Dictionary attack protection stops brute-force attacks by monitoring the number of denied login attempts originating from a particular IP address. WebQuota's dictionary attack protection feature measures all denied user name/password attempts and records the originating IP address of the suspected hacker. When the brute-force threshold is met, all subsequent requests from the IP address in question are blocked for a preconfigured period of time. WebQuota also watches all http:// requests to the server for illicit activity.

Another valuable line of defense is to implement a strong password policy. Flicks Software's PasswordCheck verifies passwords at the time they are created, ensuring that they meet the administrator's preconfigured strong password criteria. Any passwords that do not meet the network administrator's criteria are rejected. PasswordCheck includes random password generation, allowing the user to choose from many different passwords.

http://www.flicks.com

Macromedia Rolls Out MX 2004

Macromedia has released the next generation of its MX product family with new versions of Macromedia Dreamweaver, Flash, and Fireworks tools. Dreamweaver MX 2004 includes new and updated support for standards and server technologies such as CSS, secure FTP, and ASP.NET. Flash MX Professional 2004 provides developers with a forms-based programming metaphor, enabling them to build and deploy Internet applications, including video-rich, interactive experiences. Fireworks MX 2004 delivers an array of design tools and effects for Web graphics, and promotes collaboration through integration with other MX products.

Each MX 2004 product shares a consistent interface, design, development elements, and functionality. Studio MX 2004 includes Dreamweaver MX 2004, Flash MX 2004 or Flash MX Professional 2004, Fireworks MX 2004, and Freehand MX.

The MX 2004 family also includes building blocks called the MX Elements for Flash and the MX Elements for HTML, along with a unique interface for Internet applications called Halo.

http://www.macromedia.com/go/mx2004

Manage Bugs and Issues

DCom Solutions has released yKAP—Your Kind Attention Please, a Web-based product for tracking bugs, managing issues, and exchanging messages. yKAP can be installed on machines running Windows NT®, Windows 2000, and Windows XP.

Users need a browser to access yKAP. yKAP is customizable, allowing users to add custom values for system parameters, such as priority, status, and complexity. The program includes reporting capabilities, enabling export into the user's preferred format (PDF, RTF, XLS, HTML, Text, and so on) from the browser.

With yKAP, you can perform trend analysis of your data and determine, for example, whether the average turnaround time for issues is improving. The yKAP system administrator can determine how much of yKAP functionality any user can have access to by specifying a role for every user. yKAP uses your browser to process and exchange XML data and messages with the server, thus avoiding reloading of "heavy" visual elements such as images, comboboxes, listboxes, and so on.

http://www.yKap.com

Translate from CFML to .NET

New Atlanta Communications, a producer of server-side technologies, recently released the production version of BlueDragon for .NET, which allows developers to deploy ColdFusion Markup Language (CFML) Web apps onto the Microsoft .NET Framework.

BlueDragon is designed for organizations with legacy ColdFusion code who don't want to spend time and money rewriting the CF applications for other deployments. BlueDragon automatically redeploys the old code. With BlueDragon, CFML can be used as a native platform Web-scripting alternative to JSPs and ASPs, enabling full native integration support with J2EE and the .NET Framework and technologies.

http://www.newatlanta.com

Defining Business Rules

ixio has recently released the ixio Smart Data Machine 2.0, which allows developers to describe business rules through a graphical tool, the ixio Studio, and store them in a database such as Microsoft SQL Server™. The ixio Smart Data Machine then implements those rules as a set of business objects, which can be extended and used by client applications that you develop as ASP.NET forms or Windows Forms using Visual Studio. The program allows you to bypass the creation of a business logic layer by developing code for data access, application and process flow, role-based security, personalization, and localization.

http://www.ixio.com

Automate Program Documentation

Universal Report, published by Omega Computer, is an application for managing and automating the documentation of program files. Its goal is to generate a structured and highly formatted overview of a given set of program codes. A session can be achieved either step-by-step or automatically.

Universal Report is not focused on a specific programming language, and allows you to generate reports in several common formats (text, LaTeX, HTML) and languages (English, French, German, Spanish, Portuguese, and Italian). This universality is due to heuristics that use programming languages' intersection and examine further specific points. The tool's parameters can be tuned both for behavior and quality of output.

Additional features, including spell checking, general pattern research, automatic information extraction, automatic code formating, glossary maker, and graphics generator are also available.

http://www.omegacomputer.com

Charting in .NET

Quinn-Curtis Inc. has introduced QCChart2D Charting Tools for .Net, an object-oriented, .NET Framework-based charting toolkit aimed at developers who want to add interactive charting graphics to their C# and Visual Basic .NET-based apps.

The toolkit supports linear, logarithmic, and polar coordinate systems used by engineering and scientific users. It also supports advanced, discontinuous time/date coordinate systems required by business users, where weekends can be removed from the time scale and a day can have a user-defined hourly range (such as the 9:30 A.M. to 4:00 P.M. range used in financial markets). Users are able to interact with charts: adding annotations, selecting, marking and moving data points, and zooming into a particular chart region.

The software is written in C# and works with all platforms and browsers compatible with the Microsoft .NET Framework. The recommended development environment is Visual Studio .NET configured for C# or Visual Basic.

http://www.quinn-curtis.com

The Bookshelf

Adam Freeman, coauthor of Programming .NET Security, published by O'Reilly, says the majority of security lapses are caused by carelessness or lack of experience on the part of application developers. Freeman and coauthor Allen Jones wrote Programming .NET Security as both a tutorial and reference to security issues for .NET Framework-based application development.

The book offers examples for writing secure applications in both C# and Visual Basic .NET. Freeman and Jones rely on their years of experience in applying security policies for Microsoft, NASDAQ, Sun Microsystems, Netscape, and others. Before detailing the .NET Framework collection of security tools and recommendations, their book explains key concepts and common design patterns that developers must understand to build applications that can survive in a networked world.

One chapter discusses typical software development phases, and the opportunities each phase provides for uncovering vulnerabilities and defending against them. A section is devoted to .NET Framework support for cryptography, and other chapters deal with features unique to ASP.NET and COM+ component services. Also included is an API Quick Reference to all the types of the principal security-related namespaces of the .NET class libraries.

http://www.oreilly.com

Code Generation in Action by Jack Herrington is a guide to building, buying, deploying, and using code generators. Published by Manning Publications, the book demonstrates the techniques of building and using programs to write other programs. It shows how to avoid repetition and error to produce consistent, high-quality code, and how to maintain it more easily.

It demonstrates code generators for user interfaces, database access, remote procedure access, and much more. Direct examples are provided on a variety of platforms, including ASP and ASP.NET, Java/J2EE, and Open Source solutions such as Perl, Python, and PHP.

http://www.manning.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: Secure Passwords, Nested XML, and More

Web Q&A
Secure Passwords, Nested XML, and More
Edited by Nancy Michell

Code download available at: WebQA0311.exe (114KB)

Q I need to create a regular expression to validate an entry in a form and make sure that a value conforms to strong password guidelines. How can I do this?

A On the whole, regular expressions are good for matching particular patterns. In contrast, a good password should be devoid of recognizable patterns.

Here is some Visual Basic® .NET code (the ideas can be easily converted to JScript®) that uses regular expressions to determine if a password contains certain characters (a through z, A through Z, 0 through 9, and so on); it then figures out if those matches mean that the password is strong (see Figure 1). The ValidatePassword function checks for strong password characteristics, such as:

  • At least minLength characters in the password
  • At least minNumUpper (uppercase) characters
  • At least minNumLower (lowercase) characters
  • At least minNumNumbers (numeric) characters
  • At least minNumSpecial (non-alphanumeric) characters
  • At least minNumCategories of the above five categories
Then it returns a string with information about which of the tests the password may have failed.

Figure 1 Searching for Characters (Close Figure 1)
Function ValidatePasswordWork(ByVal pwd As String, _
    ByVal minLength As Integer, _
    ByVal minNumCategories As Integer, _
    ByVal minNumUpper As Integer, _
    ByVal minNumLower As Integer, _
    ByVal minNumNumbers As Integer, _
    ByVal minNumSpecial As Integer) As String

    Dim ret As String = ""
    Dim CrLf As String = ControlChars.CrLf

    ' May want to replace [A-Z] with \p{Lu}, to allow for
    ' Unicode uppercase letters.
    Dim reUpper As New System.Text.RegularExpressions.Regex("[A-Z]")
    Dim reLower As New System.Text.RegularExpressions.Regex("[a-z]")
    Dim reNumber As New System.Text.RegularExpressions.Regex("[0-9]")
    ' Special is "none of the above"
    Dim reSpecial As New System.Text.RegularExpressions.Regex _
        ("[^a-zA-Z_0-9]")

    ' Check the length.
    If Len(pwd) < minLength Then _
        ret &= "Length less than " & minLength.ToString & CrLf

    ' Check for minimum number of occurrences.
    If reUpper.Matches(pwd).Count < minNumUpper Then _
        ret &= "Fewer than " & minNumUpper & " uppers" & CrLf
    If reLower.Matches(pwd).Count < minNumLower Then _
        ret &= "Fewer than " & minNumLower & " lowers" & CrLf
    If reNumber.Matches(pwd).Count < minNumNumbers Then _
        ret &= "Fewer than " & minNumNumbers & " digits" & CrLf
    If reSpecial.Matches(pwd).Count < minNumSpecial Then _
        ret &= "Fewer than " & minNumSpecial & " special" & _
            CrLf

    ' Check for the right number of categories.
    Dim numCategories As Integer = 0

    If reUpper.Match(pwd).Success Then numCategories += 1
    If reLower.Match(pwd).Success Then numCategories += 1
    If reNumber.Match(pwd).Success Then numCategories += 1
    If reSpecial.Match(pwd).Success Then numCategories += 1

    If numCategories < minNumCategories Then _
      ret &= "Fewer than " & minNumCategories & " categories" _
          & CrLf

    ' Passes all the checks, so it is OK.
    Return ret
End Function

Function ValidatePassword(ByVal pwd As String, _
    Optional ByVal minLength As Integer = 7, _
    Optional ByVal minNumCategories As Integer = 3, _
    Optional ByVal minNumUpper As Integer = 1, _
    Optional ByVal minNumLower As Integer = 1, _
    Optional ByVal minNumNumbers As Integer = 1, _
    Optional ByVal minNumSpecial As Integer = 1) _
As Boolean
    ' Returns True is the password passes all the tests.
    ' The default values may be overridden.
    Dim temp As String
    temp = ValidatePasswordWork(pwd, minLength,
        minNumCategories, _
        minNumUpper, minNumLower, minNumNumbers, minNumSpecial)
    ' Uncomment the following line to see which tests the
    ' password failed.
    '      MsgBox(temp)

    If temp = "" Then
        ' Passed all tests.
        Return True
    Else
        Return False
    End If
End Function

A regular expression that matches the kinds of patterns you'd want would be shorter than writing a human-readable, maintainable program, but that's about all. Good password characteristics tend to cover global features of the string like how many symbols there are, which can be hard to express in a regular language.

Another measure you might consider is the amount of entropy that is in a given password. Entropy is the way unpredictability is measured. For example, if you happen to know that the first six letters of a ten-letter password are "Securi" then, though it is entirely possible that my password is "Securi^*3g", it is considerably more likely that my password really is "Security!". Those first six letters are so low-entropy that you can very easily predict the rest. If you know that the first six letters of a ten-letter password are "&5&ddV" then there is very little you can do to predict the next four, short of trying all of them. That's much higher entropy.

Anyone who is going to try and brute-force crack passwords is going to start with the small number of low-entropy passwords first (combinations of English words, for example) before trying every possible combination of letters, numbers, and symbols. By measuring the entropy of a password you can come up with some idea about how strong it is. Of course, it is important to remember that most passwords are not compromised through cracking but rather through human factors—the "Yellow Sticky Attack" or social engineering attacks, for example.

Sometimes adding even a very small amount of entropy to a password can dramatically increase its resistance to attacks. For example, which password is harder to crack, ^ft3DGpT or ^ft3DGpTABCDEFGHIJKLMNOPQRSTUVWXYZ?

The second password has barely any more entropy than the first, but by simple virtue of its length, is nearly uncrackable. This is particularly true when you consider weak authentication mechanisms—high entropy and length is best, but in a pinch lots of length plus some entropy will do.

Writing Secure Code by Michael Howard and David C. LeBlanc (Microsoft Press®, 2002) has a few guidelines on computing the equivalent bit size of a given password but does not go deep into the mathematics of computing the entropy of a password. For more, see Checklist: Create Strong Passwords.

Q I wrote a tool in Visual Basic .NET that parses an XML schema and creates documentation for it. I'm using the XMLSchema objects in the System.XMLSchema namespace.

The schema is filled with complex types that have child elements. However, there are no corresponding elements to the types; they are just included in the schema. As a result, my code isn't picking them up. How do I get to the nested child elements of a complex type that doesn't correspond to an element?

The following is an excerpt from the schema:

<xsd:complexType name="TextCell_Type" mixed="true">
    <xsd:choice minOccurs="0" maxOccurs="unbounded">
        <xsd:element name="cp" type="cp_Type" minOccurs="0"
            maxOccurs="unbounded" />
        <xsd:element name="pp" type="pp_Type" minOccurs="0"
            maxOccurs="unbounded" />
        <xsd:element name="tp" type="tp_Type" minOccurs="0"
            maxOccurs="unbounded" />
        <xsd:element name="fld" type="fld_Type" minOccurs="0"
            maxOccurs="unbounded" />
    </xsd:choice>
</xsd:complexType>

A If you iterate over the SchemaTypes collection on the XmlSchema object you will get to all global complex types including those that are not used as the type of any element. To get at the nested child elements, you need to drill down to the ContentTypeParticle property (see Figure 2). This is the post-compiled property, so make sure the schema is compiled.

Figure 2 Get Nested Children (Close Figure 2)
private void PrintParticles(string url) {
    XmlSchema schema = XmlSchema.Read(new XmlTextReader(url,
    new NameTable()), new ValidationEventHandler(ValidationCallback));
    schema.Compile(new ValidationEventHandler(ValidationCallback));
    foreach(XmlSchemaType type in schema.SchemaTypes.Values) {
// This will go over all global types even those that are not
// used as an element's type.
        XmlSchemaComplexType complexType = type as
        XmlSchemaComplexType;
        if (complexType != null) {
            TraverseParticle(complexType.ContentTypeParticle, schema);
        }
    }

    foreach(XmlSchemaElement elem in schema.Elements) {
        XmlSchemaComplexType complexType = elem.ElementType as
        XmlSchemaComplexType;
    if (complexType != null && complexType.QualifiedName ==
        XmlQualifiedName.Empty) { //Only local types of an
                                  // element since already
                                  // traversed global types
        TraverseParticle(complexType.ContentTypeParticle, schema);
        }
    }
} //End of PrintParticles

private void TraverseParticle(XmlSchemaParticle particle,
    XmlSchema schema) {

    if (particle is XmlSchemaElement) {
        XmlSchemaElement elem = particle as XmlSchemaElement;
        Console.WriteLine("XmlSchemaElement: " +
        elem.QualifiedName.ToString());
        }
else if (particle is XmlSchemaAny) {
    Console.WriteLine("Any");
}
else if (particle is XmlSchemaGroupBase) {
        Console.WriteLine("Seq, All or Choice");
        XmlSchemaGroupBase baseParticle = particle as
        XmlSchemaGroupBase;
        foreach(XmlSchemaParticle subParticle in
        baseParticle.Items) {
            TraverseParticle(subParticle, schema);
        }
    }
}

Q I used the following code to query two specific pieces of information about network adapters in Windows® Management Instrumentation (WMI):

On Error Resume Next
strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & _
    "\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * from" & _
    "Win32_NetworkAdapterConfiguration",,48)
For Each objItem in colItems
    Wscript.Echo "Description: " & objItem.Description
    Wscript.Echo "SettingID: " & objItem.SettingID
Next
How could I modify this code to query remotely? I want to query info about machine A from machine B.

A Here is a little script to get all of the network interface card (NIC) information and dump it to a servername.xml file. It takes in a list of servers from a text file.

Figure 4 Enter Server Name
Figure 4 Enter Server Name

The function in Figure 3 gets the NIC details. Figure 4 shows the code in action. For the complete code, see the download at the link at the top of this article.

Figure 3 Get NIC Details (Close Figure 3)
Function GetNICDetails(ServerName)
   On Error Resume Next
   Dim WshShell
   Dim NetworkAdapterSet
   Dim NetworkAdapter
   Dim DataHolder

   Set NetworkAdapterSet = _
      GetObject("winmgmts:{impersonationLevel=impersonate}//" & _
      ServerName).ExecQuery("SELECT PNPDeviceID, Index FROM" & _
      Win32_NetworkAdapter WHERE Manufacturer != 'Microsoft'")

   Select Case Err.Number
      Case(0)
         If IsNull (NetworkAdapterSet) OR IsEmpty
            (NetworkAdapterSet) Then
            'WMI unable to query NIC
            DataErrors = DataErrors & vbCrLf & _
               "<server name=" & """" & ServerName & """" _
               & ">" & vbCrLf & vbTab & _
               "<errnumber>" & NetAdaptersNotFound & _
               "</errnumber>" & vbCrLf & _
               "<errdescription>The server " & ServerName" & _
               " does not have any Network Adapters which" & _
               " can be queried. " & _
               "Please check that the adapters are " & _
               " recognized by the OS and try" & _
               " again.</errdescription>" & vbCrLf & "</server>"
            'Log the fact we couldn't get any adapter info
            LogDetails "There are no network adapters that" & _
                       "can be queried for this Server." & _
                       vbCrLf & "Server Name: " & ServerName

         Else
            ' Undefine the connection NIC index.
            ' Loop through the list of adapters looking for the
            ' adapter details
            For Each NetworkAdapter In NetworkAdapterSet
               If (Not IsNull(NetworkAdapter) and
                   (Len(NetworkAdapter.PnPDeviceID) > 0)) Then
                  'Call the sub routine to get the registry values for us
                 DataHolder = DataHolder & vbCrLf & _
                 GetRegValues(CStr(NetworkAdapter.PnPDeviceID), _
                 ServerName)
               End If
            Next
         End If

         DataHolder = "<nics>" & vbCrLf & vbTab & _
             DataHolder & "</nics>"

      Case (ServerNotFound)
         DataErrors = DataErrors & vbCrLf & _
            "<server name=" & """" & ServerName & """" & ">" & _
            vbCrLf & vbTab & _
            "<errnumber>" & ServerNotFound & "</errnumber>" & _
            vbCrLf "<errdescription>The server " & _
            ServerName & " could not be found. Check that " & _
            " the server has " & _
            "been switched and\or connected to the" & _
            "network</errdescription>" & vbCrLf & _
            "</server>"
      Case Else
         DataErrors = DataErrors & vbCrLf & _
            "<server name=" & """" & ServerName & """" & _
            ">" & vbCrLf & vbTab & _
            "<errnumber>" & Err.Number & "</errnumber>" & _
            vbCrLf & _
            "<errdescription>An error has occurred.  The" & _
            " description for the error is " & _
            Err.Description & "</errdescription>" & vbCrLf & _
            "</server>"
   End Select

   If Err.Number <> 0 Then
        LogDetails "Error occurred in 'GetNICDetails'" & _
                   "function. Error details are:" & _
                    vbCrLf & "Error Number: " & Err.Number & _
                    vbCrLf & "Error Details: " & _
                    Err.Description & _
                    vbCrLf & "Server Name: " & ServerName
      Err.Clear
      DataHolder = ""
    End If

   Set NetworkAdapterSet = Nothing
   Set NetworkAdapter = Nothing

   GetNICDetails = DataHolder
End Function

Q Say you have a date time string of "2003-07-02 00:06:32.700" and you know it is Greenwich Mean Time. Is there an easy way of converting this date time string to Pacific Standard Time?

A You need to convert your date into a Common Information Model (CIM) datetime format. This would then convert it into whatever time the local machine is running:

set datetime =
    CreateObject("WbemScripting.SWbemDateTime")

datetime.Value = "20030313004910.000000+000"
wscript.echo datetime.GetVarDate(false)

Q How can I get the name of the system drive using JScript?

A From Windows Script Host (WSH), you can do the following:

var shell = new ActiveXObject("WScript.Shell");
WScript.Echo("The System Drive is " +
shell.ExpandEnvironmentStrings("%SystemDrive%"));
That will work if that's the only property you want to discover about the system drive. Here's an alternate solution that solves the more general problem of querying all info about the system drive:
var SystemFolder = 1;
var FSO = new
ActiveXObject("Scripting.FileSystemObject");
var Folder = FSO.GetSpecialFolder(SystemFolder);
var drive = Folder.Drive.DriveLetter;
Now you can extract all the info you want about the drive—its name, path, date created—everything.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers: Rhett Attwood, Chris Beatie, Jason Cooke, Dan Grass, Rob Hawthorne, Priya Lakshminarayanan, Eric Lippert, Srikanth Mandadi, Pat Miller, Dare Obasanjo, Michael Sharps, Michael Whalen, Lisa Wollin, Mike Wyvel.


Data Points: SQL Server User-defined Functions

Data Points
SQL Server User-defined Functions
John Papa

Contents
Scalar UDFs
Common UDFs and Nesting
Table-valued UDFs
It's a Wrap

Auser-defined function (UDF) is a prepared code segment that can accept parameters, process some logic, and then return some data. According to SQL Server Books Online, UDFs in SQL Server™ 2000 can accept anywhere from 0 to 1024 parameters, although I must confess I have never tried to pass 1024 parameters into a UDF. Another key characteristic of UDFs is that they return a value. Depending on the type of UDF, the value can be used by the calling routine to continue processing its data. Thus, if a UDF returns a single value (a scalar value), the calling routine can use that value anywhere a standard variable or a literal value can be used. If a UDF returns a rowset, the calling routine can loop through the rowset, join to it, or simply select columns from it.

While most programming languages have supported functions for a while now, UDFs were only introduced with SQL Server 2000. Stored procedures and views have been available in SQL Server much longer than UDFs, but each of these objects has their niche in SQL Server development. Stored procedures are great for processing complex SQL logic, securing and controlling access to data, and returning a rowset to a calling routine whether that routine is a Visual Basic®-based program or another Transact-SQL (T-SQL) batch. Unlike views, stored procedures are compiled, making them ideal candidates to represent and process frequently run SQL statements. Views are great for controlling access to data, but they do it differently than stored procedures. Views are limited to only certain columns and rows from the underlying SELECT statement that generated the view. Thus a view is often used to represent a commonly used SELECT statement that may join several tables, employ a WHERE clause, and expose specific columns. Views are often found in the FROM clause of a SQL statement joined to other tables and views.

At their core, UDFs resemble both views and stored procedures. Like views, UDFs can return a rowset that can be used in a JOIN. Therefore, when a UDF returns a rowset and accepts parameters, it's like a stored procedure that you can join to, or a parameterized view. But, as I will demonstrate, UDFs can be this and much more.

There are two main types of UDFs: scalar value-returning UDFs and table value-returning UDFs. Within table value UDFs you'll find UDFs that return inline tables and multistatement tables (see Figure 1). In the following sections I'll take a look at each.

Figure 1 Main UDF Return Types (Close Figure 1)

Return Type | Main Features
Scalar | Similar to functions in other languages. Returns a single value of a scalar datatype. Can return user-defined datatypes.
Inline table | Returns a rowset of the SQL Server table datatype. Extrapolates the structure of the rowset from the SELECT statement itself. Can be SELECTed from or JOINed to. Allows multiple statements to define the UDF.
Multistatement table | Explicitly defines the structure of the table to return. Defines column names and datatypes in the RETURNS clause. Able to house more complicated and numerous T-SQL logic blocks.

Scalar UDFs

Scalar value-returning UDFs are most similar to what many programming languages refer to as functions. They return a single value consisting of a scalar datatype such as integer, varchar(n), char(n), money, datetime, bit, and so on. UDFs can also return user-defined datatypes (UDDTs) if they are based on a scalar datatype. With UDFs that return either inline or multistatement tables, a rowset can be returned via the table datatype. However, not all datatypes can be returned from UDFs. For example, a UDF cannot return a value of any of these datatypes: text, ntext, image, cursor, or timestamp.

Scalar datatype-returning UDFs can be used in various situations to make the code more maintainable, reusable, and less complex. This can be very useful when the same segment of T-SQL code is used in several places, perhaps by several stored procedures and batch SQL statements. For example, let's say several parts of an application need to find whether a product must be reordered. In each of the places this is required, the code could check the reorder level and compare it to the units in stock plus the number of units on order. However, since this code is used in several places, a UDF could be used instead to reduce the code blocks and make it easier to maintain this function in case it ever needs to change. Such a UDF might look something like the code in Figure 2 and could be called with the following SQL statement:

SELECT    ProductID,
    ReorderLevel,
    UnitsInStock,
    UnitsOnOrder,
    dbo.fnNeedToReorder(ReorderLevel, UnitsInStock, UnitsOnOrder)
AS sNeedToReorder
FROM    Products

Figure 2 Reorder Level UDF (Close Figure 2)
CREATE FUNCTION fnNeedToReorder (@nReorderLevel INT, @nUnitsInStock INT,
                                 @nUnitsOnOrder INT)
    RETURNS VARCHAR(3)
AS
BEGIN
    DECLARE @sReturnValue VARCHAR(3)

    IF ((@nUnitsInStock + @nUnitsOnOrder) - @nReorderLevel) < 0
        SET @sReturnValue = 'Yes'
    ELSE
        SET @sReturnValue = 'No'

    RETURN @sReturnValue
END
GO

In Figure 2, the fnNeedToReorder UDF performs the calculation and returns the appropriate value. This could have been accomplished via a CASE statement inside the SELECT clause, but the code is much more compact when a UDF is used instead. Plus it's easier to propagate to other places that may require the same logic. Assuming that there are several sections of an application that need to determine whether to reorder products, the UDF in Figure 2 really becomes valuable as it makes the application easier to maintain when the logic changes. For example, it doesn't make a lot of sense to reorder a product that has been discontinued. Thus, by changing the UDF in order to account for this business rule, the logic is changed in one place (see Figure 3) and can be run with the following code:

SELECT    ProductID,
    ReorderLevel,
    UnitsInStock,
    UnitsOnOrder,
    dbo.fnNeedToReorder(ReorderLevel, UnitsInStock, UnitsOnOrder,
                        Discontinued) AS sNeedToReorder
FROM    Products

Figure 3 Reorder Level UDF, Version 2 (Close Figure 3)
CREATE FUNCTION fnNeedToReorder (@nReorderLevel INT, @nUnitsInStock INT,
                                 @nUnitsOnOrder INT, @bDiscontinued BIT)
    RETURNS VARCHAR(3)
AS
BEGIN
    DECLARE @sReturnValue VARCHAR(3)

    IF @bDiscontinued = 1
        SET @sReturnValue = 'No'
    ELSE
        IF ((@nUnitsInStock + @nUnitsOnOrder) - @nReorderLevel) < 0
            SET @sReturnValue = 'Yes'
        ELSE
            SET @sReturnValue = 'No'

    RETURN @sReturnValue
END
GO

Notice that the UDF is called using the two-part name of object owner and object name. The object's owner is required when using a UDF that returns a scalar datatype value. Granted, by adding the fourth parameter (Discontinued) to the UDF, all of the places that call the UDF must also be changed. For easier maintenance, I could rewrite the UDF to retrieve the data itself using the ProductID for each row, as shown in Figure 4. This technique is easier to maintain because it does not require any of the calling routines to change how the UDF is called when the logic changes—as long as the data can be pulled in from the current Products table row. However, to gain this maintainability there is a performance trade-off. The UDF in Figure 4 has to retrieve a row from the Products table for every row that is returned from the calling routine. Since the calling routine is retrieving every row from the Products table already, if the table has 77 rows, the code will execute 77 SELECT statements, one for each row returned from the main SELECT statement. While each SELECT is selecting based on the primary key field (ProductID) and hence is very fast, performance can be adversely affected when the rowset is very large or the SELECT statement is less efficient. The code in Figure 4 can be called by using the following SQL snippet:

SELECT    ProductID,
    ReorderLevel,
    UnitsInStock,
    UnitsOnOrder,
    dbo.fnNeedToReorder(ProductId) AS sNeedToReorder
FROM    Products

Figure 4 Reorder Level UDF by ProductID (Close Figure 4)
CREATE FUNCTION fnNeedToReorder (@nProductID INT)
    RETURNS VARCHAR(3)
AS
BEGIN
    DECLARE @sReturnValue VARCHAR(3)

    SELECT    @sReturnValue =
        CASE Discontinued
            WHEN 1
                THEN 'No'
            ELSE
                CASE
                    WHEN ((UnitsInStock + UnitsOnOrder) -
                           ReorderLevel) < 0
                        THEN 'Yes'
                    ELSE    'No'
                END
        END
    FROM    Products
    WHERE   ProductID = @nProductID

    RETURN @sReturnValue
END
GO

An alternative to using this function in a SELECT statement is to create a computed column in the Products table called NeedToReorder. This column would be defined not as a datatype but as the return value of the fnNeedToReorder UDF from Figure 3. To add this column, I can alter the Products table as follows to indicate that the column should be computed:

ALTER TABLE Products
    ADD  NeedToReorder AS dbo.fnNeedToReorder(ReorderLevel,
    UnitsInStock, UnitsOnOrder, Discontinued)
This way the column is defined in the table itself, but is automatically calculated using the UDF.

Common UDFs and Nesting

Thus far I have shown several ways to tackle the same issue using a UDF that returns a scalar value. There are other useful applications of UDFs including functions that are not readily available in T-SQL. One example is a specialized formatting function. For instance, phone numbers are commonly stored without their formatting characters in char(10) columns that represent the area code and phone number (assuming it's a United States number). A UDF could be used to retrieve the phone number in a formatted structure (see Figure 5). Thus, retrieving and formatting a phone number is as simple as this:

SELECT dbo.fnCOM_FormatTelephoneNumber ('3335558888')

Figure 5 fnCOM_FormatTelephoneNumber (Close Figure 5)
CREATE FUNCTION fnCOM_FormatTelephoneNumber (@sPhone CHAR(10))
    RETURNS VARCHAR(15)
AS
BEGIN
    DECLARE @sPhoneFormat VARCHAR(15)

    IF LEN(@sPhone) < 10
        SET @sPhoneFormat = @sPhone

    ELSE SET @sPhoneFormat =     '(' + LEFT(@sPhone, 3) + ') ' +
                    SUBSTRING(@sPhone, 4, 3) + '-' + RIGHT(@sPhone, 4)
    RETURN @sPhoneFormat
END

Any common function can be created using this technique to augment the regiment of functions available in SQL Server. Another example is a function that formats a date to the MM/DD/YYYY format with leading zeros:

CREATE FUNCTION fnCOM_StandardDate (@dtDate DATETIME)
    RETURNS VARCHAR(10)
AS
BEGIN
    RETURN
        dbo.fnCOM_2Digits (CAST(MONTH(@dtDate) AS VARCHAR(2))) + '/' +
        dbo.fnCOM_2Digits (CAST(DAY(@dtDate) AS VARCHAR(2))) + '/' +
        CAST(YEAR(@dtDate) AS VARCHAR(4))
END
The fnCOM_StandardDate UDF accepts a datetime value and returns a varchar(10) in a MM/DD/YYYY format. Pretty simple, of course, and if your application requires a particular format often, this technique could make it easier to maintain. One key component to notice in the previous code is the use of a nested UDF. The fnCOM_StandardDate UDF calls the fnCOM_2Digits UDF (shown in the next sample) twice, both times to put a leading zero in front of days or months less than 10.
CREATE FUNCTION fnCOM_2Digits (@sValue VARCHAR(2))
    RETURNS VARCHAR(2)
AS
BEGIN
    IF (LEN(@sValue) < 2)
        SET @sValue = '0' + @sValue

    RETURN @sValue
END

UDFs can be nested within each other as long as the inside UDF is created first. One catch with nesting functions is that built-in functions that are nondeterministic, such as the getdate function, cannot be nested inside of another UDF (otherwise, a SQL Server error is raised). A nondeterministic function is one which may not return the same result when called multiple times with exactly the same parameters. The getdate function falls into this category since every time it is called, it returns the new current date and time. Another commonly used nondeterministic built-in function is the NewID function. It is also nondeterministic as it always returns a unique GUID and, as such, the NewID function is not allowed to be nested within a UDF.

Table-valued UDFs

Within the category of table-valued UDFs there are two sub-types: inline table value-returning UDFs and multistatement table value-returning UDFs. UDFs that return inline tables return a rowset via the SQL Server table datatype. They are defined with a single SELECT statement making up the body of the function. Inline table value-returning UDFs cannot contain additional T-SQL logic outside of the SQL SELECT statement that defines the table it will return. However, they are simpler to create than UDFs that return multistatement tables since they do not have to define the exact table structure to be returned. UDFs that return inline tables extrapolate the structure of the rowset from the SELECT statement itself. Thus, the columns that the UDF will return are determined by the columns in the SELECT list. The following code shows the fnGetEmployeesByCity UDF, which accepts a city and returns a table containing all employees' first name, last name, and address:

CREATE FUNCTION fnGetEmployeesByCity (@sCity VARCHAR(30))
    RETURNS TABLE
AS
RETURN
    (
        SELECT    FirstName, LastName, Address
        FROM    Employees
        WHERE    City = @sCity
    )
GO
This inline table value-returning UDF can be selected from or even joined to because it returns a rowset via the table datatype, as shown here:
SELECT * FROM dbo.fnGetEmployeesByCity('seattle')

Notice that the UDF is called using the two-part name of object owner and object name. However, the object's owner is not required (but is acceptable) when using a UDF that returns a table datatype value. Table-valued UDFs are quite flexible in that they can be used like a prepared and parameterized view (if one existed). In table-valued UDFs you can use parameters, achieve the performance of a prepared query, and join or select from the resulting rowset (or table in this case).

Although this type of UDF is compact, it is important to remember that if additional logic needs to be added to the UDF, it will have to be converted to a multistatement table value-returning UDF. Also, inline table value-returning UDFs cannot have an ORDER BY clause on the SELECT statement either (unless it is used in conjunction with the TOP clause).

A UDF that returns multistatement tables explicitly defines the structure of the table to return. It does so by defining the column names and datatypes right in the RETURNS clause. Thus it takes a bit more code to get it set up than an inline table value-returning UDF. However, it has several advantages over inline table value-returning UDFs including the ability to house more complicated, numerous T-SQL logic blocks. As their name suggests, multistatement table value-returning UDFs allow multiple statements to define the UDF. Thus statements such as control of flow, assignments, cursors, SELECTS, INSERTS, UPDATES, and DELETES are allowed and can all exist in a single UDF. So, as opposed to UDFs that return inline tables, their multistatement brethren are not limited to a single SELECT statement nor are they prohibited from ordering the returning rowset.

Figure 6 shows how to rewrite the inline table value-returning UDF from the code snippet I just showed as a multistatement table value-returning UDF. Thus the multistatement type can do anything the inline type can do. A more complicated use of a UDF that returns multistatement tables could involve retrieving all employees by city, but if no customers match the given city then a dummy row is returned where the Address field is filled with "No matching employees found in the specified city," as shown in Figure 7.

Figure 7 fnGetEmployeesByCity3 (Close Figure 7)
CREATE FUNCTION fnGetEmployeesByCity3 (@sCity VARCHAR(30))
    RETURNS @tblMyEmployees TABLE
    (
        FirstName VARCHAR(20),
        LastName VARCHAR(40),
        Address VARCHAR(120)
    )
AS
BEGIN
    INSERT    @tblMyEmployees
    SELECT   FirstName, LastName, Address
    FROM     Employees
    WHERE    City = @sCity
    ORDER BY LastName

    IF NOT EXISTS (SELECT * FROM @tblMyEmployees)
        INSERT @tblMyEmployees (Address)
            VALUES ('No matching employees found in the specified city')

    RETURN
END
GO
Figure 6 fnGetEmployeesByCity2 (Close Figure 6)
CREATE FUNCTION fnGetEmployeesByCity2 (@sCity VARCHAR(30))
    RETURNS @tblMyEmployees TABLE
    (
        FirstName VARCHAR(20),
        LastName VARCHAR(40),
        Address VARCHAR(120)
    )
AS
BEGIN
    INSERT   @tblMyEmployees
    SELECT   FirstName, LastName, Address
    FROM     Employees
    WHERE    City = @sCity
    ORDER BY LastName
    RETURN
END
GO

It's a Wrap

There are some other key factors that can help create a powerful UDF of any type, one of which is recursion. Recursion is supported with UDFs such that a UDF can call itself from within itself. Basically, recursion is just nesting a UDF except that the UDF you are nesting is the same one you are in. This can be very useful in certain situations including when you're creating a UDF that must compute a factorial or evaluate each character in a string. There is a limited depth of 32 levels of recursion in SQL Server 2000, after which an error is raised.

It is also important to point out that a UDF can be bound to the schema of the underlying objects to which it refers. To do this, the UDF must be created using the WITH SCHEMABINDING clause. If the UDF is created this way and someone attempts to alter one of the underlying objects' schema without first removing the schema binding, an error will be generated and raised. Using this option will help you ensure that no UDFs break inadvertently due to changes in an underlying object's schema.

When evaluating UDFs it is vital to consider the balance between performance and maintainability. While UDFs can reduce the amount of common code, be used as part of a common function library, can promote shorter code blocks, and are generally easier to maintain than additional versions of the same SQL logic, it would be reckless to use a UDF without first taking into consideration any of the drawbacks.

It would be a bad idea to use a UDF if performance suffers tremendously. For example, assume that there is a UDF that performs a SQL SELECT statement that takes one second to execute. If this UDF is used in a SELECT or a WHERE clause it will be executed for every row. Thus the time the main query takes to execute could increase drastically depending on such factors as the number of rows evaluated and returned and the types of indices in place. Before using a UDF in this type of situation, carefully weigh the options and do some performance testing. However, using a UDF that performs a calculation such as the one shown in Figure 3 barely affects the performance of query. As with any tool, when used properly and evaluated accordingly prior to going live, UDFs offer great convenience and maintainability.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive. You can reach him at mmdata@microsoft.com.


Cutting Edge: Custom Provider Controls

Cutting Edge
Custom Provider Controls
Dino Esposito

Code download available at: CuttingEdge0311.exe (139KB)
Contents
Derive a New Class
Providers
The ToolTip Provider Control
HelpProvider and ErrorProvider
The IExtenderProvider Interface
Writing a Custom Extender Control
Extending the ToolTip Extender
Summing It Up

Sometimes I find the Windows® Forms controls in Visual Studio® .NET to be lacking a particular feature that's indispensable to the application I'm building. While I'm not saying that the set of Windows Forms controls is inadequate, a general-purpose toolbox can't possibly meet all user requirements. Let's consider the TextBox control, for instance. It's relatively easy to agree on a minimum set of properties and methods that this control has to support. However, there is a large set of extended properties that would be very useful. For example, you might want the ability to change the background color when the control gets the focus, an input mask, or perhaps an associated autocomplete list. Of course, I'd rather have a lightweight control that can be extended than a bloated component that has all the features I'd ever need.

In Windows Forms, inheritance is just one option for building extended versions of existing controls. Another option involves provider controls, which is the subject of this month's column. Provider controls add new properties to existing controls without the need for inheritance. The idea is that you use an external extender control to provide new properties and behaviors to existing controls. The implementation of such extensions doesn't add weight to the base control; it's a lightweight alternative to inheritance. Provider controls don't replace inheritance; rather, they extend and complement it in many ways.

Derive a New Class

Suppose you want a textbox that changes its background color when it has focus. One option you have is to derive a new class. Figure 1 shows a simple class that inherits from TextBox and wires up the GotFocus and LostFocus events. The AutoColorTextBox class changes the background color when the control gets the focus and restores the original color when the LostFocus event is fired. You can easily assign different colors to each textbox. All you need to do is place instances of the AutoColorTextBox class on the form and set the background color when the control gets focus:

private void Form1_Load(object sender, EventArgs e)
{
   autoColorTextBox1.SelectedBackColor = Color.Cyan;
   autoColorTextBox2.SelectedBackColor = Color.Yellow;
   autoColorTextBox3.SelectedBackColor = Color.LightGreen;
}

Figure 1 The Original Control (Close Figure 1)
using System;
using System.Drawing;
using System.Windows.Forms;

namespace MsdnMag.ExtenderLib
{
    public class AutoColorTextBox : TextBox
    {
        public AutoColorTextBox()
        {
            SelectedBackColor = this.BackColor;
            m_oldBackColor = this.BackColor;
            this.GotFocus += new EventHandler(AutoColorTextBox_GotFocus);
            this.LostFocus += new
                EventHandler(AutoColorTextBox_LostFocus);
        }

        public Color SelectedBackColor;
        private Color m_oldBackColor;

        private void AutoColorTextBox_GotFocus(object sender, EventArgs e)
        {
            if (this.BackColor != SelectedBackColor)
            {
                m_oldBackColor = this.BackColor;
                this.BackColor = SelectedBackColor;
            }
        }

        private void AutoColorTextBox_LostFocus(object sender, EventArgs e)
        {
            if (m_oldBackColor != this.BackColor)
            {
                this.BackColor = m_oldBackColor;
                m_oldBackColor = this.BackColor;
            }
        }
    }
}

Inheritance is an excellent choice if you need to customize the behavior and the appearance of controls. But what if you want to add another feature to your TextBox class? Suppose you want the textbox to accept only numbers and display all negative numbers in red. Figure 2 shows this new version of the control. The class AutoNumericTextBox inherits from the previously defined AutoColorTextBox. As long as you need both features in all textboxes, inheritance is fine.

Figure 2 The Modified Control
Figure 2 The Modified Control

However, what if you need to use extensions as individual attributes? As these controls are currently designed, all the features are loaded even if you need only the numeric mode (see Figure 3). An alternative scheme, in which extensions are managed individually, is at odds with the core idea of inheritance. With inheritance an object has all the features of the class. So if you want to be able to combine the basic features of the TextBox class with any possible extensions like auto-color and numeric mode, you need to build one of the following classes: TextBox+AutoColor, TextBox+Numeric, or TextBox+AutoColor+Numeric. Scale this pattern to meet the complexity (and the quantity) of real-world classes and you will understand why inheritance is not always the best option for extending controls .

Figure 3 Loading All Features (Close Figure 3)
using System;
using System.Drawing;
using System.Windows.Forms;

namespace MsdnMag.ExtenderLib
{
    public class AutoNumericTextBox : AutoColorTextBox
    {
        public AutoNumericTextBox()
        {
            Value = 0.0f;
            NumericMode = false;
            m_oldForeColor = this.ForeColor;
            this.TextAlign = HorizontalAlignment.Right;
            this.LostFocus += new
                EventHandler(AutoNumericTextBox_LostFocus);
        }

        public double Value;
        private Color m_oldForeColor;
        public bool NumericMode;

        private void AutoNumericTextBox_LostFocus(object
            sender, EventArgs e)
        {
            if (NumericMode)
            {
                Value = Convert.ToDouble(this.Text);
                if (Value <0)
                {
                    this.ForeColor = Color.Red;
                    this.Font = new Font(Font.Name, Font.Size,
                        FontStyle.Bold);
                }
                else
                {
                    this.ForeColor = m_oldForeColor;
                    this.Font = new Font(Font.Name, Font.Size);
                }
            }
        }
    }
}

Providers

A provider control lets you extend existing controls without creating new specific classes. The code that performs the extension (the GotFocus and LostFocus handlers) is loaded into an external class that plugs into the base class at run time. The excellent design-time support in Visual Studio .NET makes the presence of provider controls go almost unnoticed.

To get the numeric code and auto-color features, you need to write two provider controls; one implementing auto-color and one implementing the numeric mode. On the final form, place only the providers that you really need. This way, no unnecessary code is loaded. If you need to combine two or more extensions you can do it by binding more providers to the same control.

Provider controls prove to be extremely useful in another scenario as well. Consider again the extension that changes the background color of a TextBox control. What if you want to add that feature to RichTextBox and ListBox controls, too? Without providers, your only option would be to create two new classes.

The beauty of provider controls is that they allow you to extend multiple types of base controls. The ultimate role of providers is to add custom properties to existing controls. Extended properties show up in the Visual Studio .NET property grid. At run time the Windows Forms infrastructure yields control to the provider for the actual implementation of the extended properties. In doing so, the extender control gets a reference to the extendee and can check the type and the name as well as read and update the state.

The ToolTip Provider Control

Before I discuss how to build custom provider controls, let's review the predefined extenders that come with the Windows Forms toolbox: the ToolTip, HelpProvider, and ErrorProvider controls. All of them are displayed in the component tray area of the Visual Studio .NET IDE, as shown in Figure 4. Each provider control can have its own set of properties that affect the way in which the control works. The control-specific properties have nothing to do with the extensions that are added to bound controls.

Figure 4 Provider Control Icons
Figure 4 Provider Control Icons

The ToolTip control lists properties that let you configure the timing of the popup text, and the ErrorProvider control lets you change the icon used to represent the error message and make the text blink on demand. Finally, the HelpProvider control needs a compiled help file to work.

Figure 5 ToolTip Property
Figure 5 ToolTip Property

Now that you've added a few extender controls to the form, select any other control in the form and scroll to its set of properties. You may be surprised to find that all controls in the form now feature an extra ToolTip property (see Figure 5). The exact name of the property in the grid is "ToolTip on toolTip1," where toolTip1 is the name of the extender control. Set this property for a textbox and build the project. The control on the form should now display a tooltip. Select a button and repeat the operation. The button also has a "ToolTip on toolTip1" property which, when set, causes a tooltip to appear when the mouse pointer hovers over the button. This means that the ToolTip extender control can extend virtually any control you may have on a form with the sole exception of another ToolTip control. Each control that can appear on a form is automatically given a ToolTip property which, if set, causes a popup description to appear at run time.

In this way, all controls that belong to the form share the same ToolTip object, which maintains an internal table of control/caption pairs. Like many other Windows Forms controls, the ToolTip control is a wrapper built around a Win32® window-based control. A Win32 tooltip window lives behind the managed ToolTip object. This window is created when the managed ToolTip object is instantiated and configured for style and delay. Next, a couple of internal tables are added—one to store control/caption associations and one to store control/region pairs.

If you're familiar with Win32 ToolTip controls you know that a single instance can be used to display popup descriptions at different screen locations. In other words, the way in which the ToolTip control works in Windows is significantly different from HTML tooltips, which are attributes bound to an individual element. The Visual Basic® 6.0 model is like the Web scenario. Visual Basic 6.0, in fact, gives each control a ToolTipText property. Don't judge this behavior by its appearance, however. The Visual Basic 6.0 mechanism is not really different from the Windows Forms model. In the Visual Basic 6.0 IDE, the ToolTipText property is an extended property that the ActiveX® container adds to any hosted control. Under the hood of this seemingly simple programming model, only one Win32 tooltip window is created and used to display text for each constituent element that features a non-empty caption. How can you get a tooltip window to display text whenever the mouse pointer stops over a given control? It's simple—use regions.

A tooltip region is a rectangular area that the tooltip is called to monitor. Whenever the mouse enters the area and stops there for the specified period of time, the ToolTip moves its window there, sets the caption, and pops it up. In Windows Forms, each control that has a non-empty caption also is given an entry in the region table. The region contains the bounds of the control area.

HelpProvider and ErrorProvider

The HelpProvider control gets into the game when the user presses the F1 key while the focus is on a form control. HelpProvider adds three properties to each control: HelpNavigator, HelpKeyword, and HelpString. HelpNavigator determines what happens when the user requests some help. The property accepts values from the HelpNavigator enumeration and provides access to specific elements of the help file. HelpKeyword indicates the keyword or the topic to be searched, while the HelpString is displayed through a tooltip, as shown in Figure 6. The HelpString property is independent from any help file and displays the tooltip when F1 is pressed.

Figure 6 Some Help Text
Figure 6 Some Help Text

The ErrorProvider displays an optionally blinking icon beside any control with an error message (see Figure 7). The properties added to each control respect the graphical styles of the icon—IconAlignment and IconPadding. By default, the icon is placed to the right of the control and centered vertically. No pixels are left as padding between the border of the control and the icon unless you set IconPadding to a nonzero value.

Figure 7 Error Icon
Figure 7 Error Icon

The ErrorProvider control adds a third entry to each control's property grid—Error. The Error property contains the error message that the control will display. Unlike tooltips, which are seldom determined at run time, error messages are a dynamic property. This reveals an important difference between extender controls and inherited classes. The properties added to each control—say, the Error message—are not really new properties that you can set programmatically. You can set extended properties at design time because Visual Studio .NET performs this trick, not because real properties are dynamically added to controls. The following code doesn't compile and the reason is obvious—the Error property is not part of the programming interface of a TextBox, but you can set it at design time:

txtAge.Error = "Must be > 18";

So how can you set an error message or a tooltip caption dynamically? You do it using a method on the extender control. The previous line of code must be rewritten as follows:

errorProvider1.SetError(txtAge, 18);

You set the ToolTip text property at run time using nearly identical syntax:

toolTip1.SetToolTip(txtAge, "Enter a major age");
Is there a pattern behind this? An extender control must supply a pair of GetXxx/SetXxx methods for each Xxx property it adds to extendees. Using these methods, a control can access both properties at run time. The prototype of such methods is shown here:
public string GetError(Control control);
public void SetError(Control control, string value);
In these signatures the string type changes according to the actual type of the extended property.

Overall, extender controls are important for a couple of reasons. They provide an alternative model to extend existing controls which is orthogonal to class inheritance. Second, using extender controls wisely can significantly limit the amount of code you need to write for each form. What is the common foundation upon which extender controls must base their functionality? It's the IExtenderProvider interface.

The IExtenderProvider Interface

IExtenderProvider defines only one method—CanExtend. The interface is defined as follows:

public interface IExtenderProvider
{
    bool CanExtend(object extendee);
}

Visual Studio .NET calls CanExtend to determine which objects in a container should receive the extender properties. Any component that provides extender properties must implement IExtenderProvider:

public class SimpleTextBoxExtender : Component, IExtenderProvider
{
   •••
}

Aside from the fact that it implements the IExtenderProvider interface, an extender is primarily a control. For this reason, in addition to implementing the interface you must make it inherit the base functionality of a control. However, if you inherit it from Control, then the control can't be dropped in the component tray area—only on the form. To make it show in the component tray, use Component as the base class. In both cases, though, the overall behavior of the extender doesn't change much.

An extender provider class must be marked with a [ProvideProperty] attribute. The constructor of the ProvideProperty attribute class takes two arguments. The first parameter is the name of the property to add and the second is the type of the object to which you provide the property.

The following code snippet defines an extender that adds the SelectedBackColor property to all textboxes on the form:

[ProvideProperty("SelectedBackColor", typeof(TextBox))]
class SimpleTextBoxExtender: Component, IExtenderProvider
{
   •••
}
Note that you can also declare the property as an extended property for virtually any component—using IComponent or Control instead of TextBox in the attribute. In this case, the implementation typically will include features that make it usable only with a specific category of controls.

You implement the CanExtend method so that it returns true for each control to which the extender wants to add properties. Here's a possible implementation for a simple TextBox extender:

public bool CanExtend(object extendee)
{
   return (extendee is TextBox);
}

A similar implementation also catches any control that inherits from the TextBox class, including the custom classes I built earlier. Figure 8 shows the source code of a very simple extender control. It implements a background color change when the control is focused. In particular, the extender defines the pair of GetSelectedBackColor and SetSelectedBackColor methods. They can be called by any textbox to programmatically get or set the background color. In addition, the control features a public property, SelectedBackColor, which appears in the property grid and provides the default color. Notice that the property won't show up in the property grid if you don't explicitly implement it using the get/set accessors:

private Color m_SelectedBackColor;
public Color SelectedBackColor
{
    get {return m_SelectedBackColor;}
    set {m_SelectedBackColor = value;}
}

Figure 8 Extender Control (Close Figure 8)
namespace Samples
{
    [ProvideProperty("SelectedBackColor", typeof(TextBox))]
    public class SimpleTextBoxExtender : Component,
        IExtenderProvider
    {
        public SimpleTextBoxExtender()
        {
            InitializeComponent();

            // Use a hashtable to track selected colors for
            // each extended control
        }

        public bool CanExtend(object target)
        {
            return (target is TextBox);
        }

        private Color m_SelectedBackColor;
        public Color SelectedBackColor
        {
            get {return m_SelectedBackColor;}
            set {m_SelectedBackColor = value;}
        }

        private Color backupBackColor;
        public Color GetSelectedBackColor(Control control)
        {
            return SelectedBackColor;
        }

        public void SetSelectedBackColor(Control control, Color
            selColor)
        {
            TextBox t = (TextBox) control;
            SelectedBackColor = selColor;
            t.GotFocus += new EventHandler(TextBox_GotFocus);
            t.LostFocus += new EventHandler(TextBox_LostFocus);
        }

        private void InitializeComponent()
        {
            SelectedBackColor = Color.Cyan;
        }

        private void TextBox_GotFocus(object sender,
            EventArgs e)
        {
            TextBox t = (TextBox) sender;
            backupBackColor = t.BackColor;
            t.BackColor = SelectedBackColor;
        }

        private void TextBox_LostFocus(object sender,
            EventArgs e)
        {
            TextBox t = (TextBox) sender;
            t.BackColor = backupBackColor;
        }
    }
}

If you use this control in a sample application, it works just fine. However, when coded this way the control is too simple and not particularly functional. The main drawback is that the background color is unique for all textboxes. Furthermore, hooking events is a delicate operation that needs more attention. Try using any of the derived textboxes with the extender and you'll see that confusion results. Serious extenders need to maintain a table of bound controls and store settings individually.

Writing a Custom Extender Control

Let's enhance the simple TextBox extender to provide better support to each extendee. The extender will now add two properties—the background and foreground colors (see Figure 9).

Figure 9 Adding Background and Foreground Color (Close Figure 9)
[ProvideProperty("SelectedBackColor", typeof(TextBox))]
[ProvideProperty("SelectedForeColor", typeof(TextBox))]
public class TextBoxExtender : Component, IExtenderProvider
{
   private Hashtable Extendees;

   public TextBoxExtender()
   {
       InitializeComponent();

       // Use a hashtable to track selected colors
       // for each extendee
       this.Extendees = new Hashtable();
   }

   •••
}

Each extendee control is assigned an information class that's structured as follows:

public class TextBoxInfo
{
   public Color SelectedBackColor;
   public Color OldBackColor;
   public Color SelectedForeColor;
   public Color OldForeColor;
   public bool  EventsWired;
}
This class tracks the colors to use, but also the colors to restore and wires the GotFocus and LostFocus events only once, which is indicated by the EventsWired property. The code for the Get/Set pairs gets slightly more complicated. Let's tackle the Get accessor first:
public Color GetSelectedBackColor(Control control)
{
   // Retrieve related info
   TextBox t = (TextBox) control;
   TextBoxInfo info = (TextBoxInfo) Extendees[t];

   return info.SelectedBackColor;
}

The control parameter references the actual extendee control—a TextBox in this case. Once you get a reference to the underlying control, you could also filter out controls based on the name or any other property you can access. You use the control reference as the key to access the hashtable and retrieve the corresponding TextBoxInfo structure. After that, returning the background color to use is child's play.

Who's creating the entry in the hashtable and when? The Set accessor of one of the properties is responsible for inserting the control in the table. The control won't be added if a similar entry already exists in the hash table. Figure 10 shows the source code of the SetSelectedBackColor method.

Figure 10 SetSelectedBackColor Method (Close Figure 10)
public void SetSelectedBackColor(Control control, Color
    selColor)
{
    TextBoxInfo info;
    TextBox t = (TextBox) control;
    if (!Extendees.ContainsKey(t))
        info = new TextBoxInfo();
    else
        info = (TextBoxInfo) Extendees[t];

    // Store the new value
    info.SelectedBackColor = selColor;

    // If not already done, wire events up
    if (!info.EventWired)
    {
        t.GotFocus += new EventHandler(TextBox_GotFocus);
        t.LostFocus += new EventHandler(TextBox_LostFocus);
        info.EventWired = true;
    }

    // Add to the table
    if (!Extendees.ContainsKey(t))
        Extendees[t] = info;
}

Extending the ToolTip Extender

The ToolTip class is sealed and can't be inherited to add more features such as the balloon style. In Windows XP and newer operating systems, the ToolTip window has been given the TTS_BALLOON style to display the caption in a balloon-style popup window. At first, deriving a new extender class from the existing ToolTip provider and adding support for the balloon style seems like a relatively easy task. The first problem you run into is that the ToolTip class is sealed; that is, it's not further inheritable. However, by using aggregation instead of inheritance you can devise a custom wrapper extender that exploits the underlying capabilities of the ToolTip class.

The idea is to create a new instance of the ToolTip class in the extender's constructor and call the GetToolTip and SetToolTip methods in the get and set accessors of the new ToolTip property.

[ProvideProperty("MyText", typeof(TextBox))]
public class MyToolTip : Component, IExtenderProvider
{
   private ToolTip _toolTip;
   public MyToolTip()
   {
      toolTip = new ToolTip();
   }
   •••
}

The MyToolTip class embeds a dynamically created instance of the ToolTip class as a private property and exposes a MyText extended property. The implementation of this property passes through the implementation of the text property on the original ToolTip class:

public string GetMyText(Control control)
{
   return _toolTip.GetToolTip(control);
}

public void SetMyText(Control control, string caption)
{
   // TODO :: Enter your changes...

   _toolTip.SetToolTip(control, caption);
}

By implementing this code, you end up with your own custom extender, making it work on top of the existing ToolTip component. A good reason for extending the ToolTip is to use a customized ToolTip control. Balloon and multiline styles are not supported by Windows Forms ToolTips and the only way to add these features is through a custom ToolTip extender. Unfortunately, of the three built-in extenders the one you need—the ToolTip component—is sealed and can't be inherited. That's why you must resort to the trick based on control aggregation.

However, the main obstacle on the road to full customization of the Windows Forms ToolTips lives elsewhere. The ToolTip control inherits from Component, not from Control. There's controversy over which way is better. If you inherit a class from Component, the class behaves like a control tray component, which would be great for extenders. If you derive the class from Control, the resulting component can't be placed in the IDE tray area. However, it will inherit the Handle property from the base class.

The Handle property represents the HWND handle of the underlying Win32 control. Since Windows Forms are based extensively on Win32 controls, they end up creating a window whenever a Control object is instantiated. To customize the appearance and behavior of a control, you need to resort to Win32 styles and messages. And how can you set them if you can't access the underlying window handle?

The managed ToolTip control does have a Handle property defined as an IntPtr member. Recall that IntPtr is the .NET Framework type that maps a Win32 handle such as HWND, HGLOBAL, or HKEY. The difficulty lies in the fact that the ToolTip's Handle property is declared as private and is thus inaccessible due to its protection level. The ToolTip class could be further extended using aggregation, but the extensions you can really add in practice are limited to anything you can do without knowing the HWND of the underlying ToolTip window.

Summing It Up

Extender controls represent an alternate route to control inheritance. By writing extenders you can achieve two goals. First, you can implement a finer level of granularity and decide what features to add to a class without creating a new class and inheriting features at run time. Second, you can add the same features to many control types using a single extension class. You could add properties to textboxes and comboboxes using a unique extender; if you used inheritance, that would require two distinct classes. In addition, while inheritance represents a pure coding approach, extenders try to shift some of the required burden at design time. Extenders reduce the amount of code you need to write and promote a model of programming that is more declarative and attribute based.

Remember, of course, extenders do not replace inheritance, but rather complement it and contribute to making your programming toolset richer than ever.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Programming Microsoft ASP.NET (Microsoft Press, 2003), he spends most of his time teaching classes on ADO.NET and ASP.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com.


The XML Files: XML in Microsoft Office Word 2003

The XML Files
XML in Microsoft Office Word 2003
Aaron Skonnard

Contents
XML Support in Word 2003
Introduction to WordML
Working with WordML
Custom Schemas
Schema Validation
Saving Documents
Applying Transforms
Summing It Up

Several years ago I began a new book project and decided to write the entire project in XML. I figured that producing the book in XML would allow me to reuse the content elsewhere, transform the content into different output formats, and make it easy to manipulate with myriad XML technologies including XPath, XSLT, the various XML APIs, and possibly even XQuery.

When I started to investigate how I might accomplish this, I opened Microsoft® Office Word 2000 and began to experiment. I was looking for ways to both author the chapters with the rich editing and formatting capabilities of Word and define a mapping between the Word document and an XML structure. This would have allowed me work in the familiar Word environment to produce the XML content. As I experimented with the Save as HTML functionality, the only technique available at the time for making Word produce a marked-up document, it quickly became apparent that a Save as XML option was needed. So I was back to good old Emacs to write the XML by hand. Thousands of angle brackets later, I swore never to do it again.

Having the entire book in XML has been extremely valuable, but the road was too painful. Luckily, Microsoft Office Word 2003 is now available.

XML Support in Word 2003

The new XML support in Word 2003 is one of its most exciting and powerful features. XML is no longer an afterthought; Word has been completely designed around XML from the ground up. It supports a native XML vocabulary called the Word Markup Language (WordML). More details on WordML can be found at Microsoft Office Word 2003 Preview.

Microsoft Office Word 2003 introduces the Save as XML command that produces WordML documents. Later, when you double-click on an XML document produced by Word, the Windows loader automatically associates the file with Word. WordML is powerful and flexible enough to capture all of the rich editing and formatting of a Word document with full round-tripping. If you create a typical document in Word, save it as WordML, and then later read it back in; the document is guaranteed to look like the original.

Here's the basic structure of a WorldML document:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?mso-application progid="Word.Document"?>
<w:wordDocument
    xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml" ...
>
   <!-- WordML structure goes here -->
</w:wordDocument>
Notice that it's just an XML document. The root element is named wordDocument from the http://schemas.microsoft.com/office/word/2003/wordml namespace (I'll use the prefix "w" to refer to this namespace in this column). When you double-click on this file from Windows Explorer, Windows® automatically determines that this is a WordML document (by inspecting the mso-application processing instruction at the top) and subsequently launches Word to process it.

The introduction of WordML is probably the most significant change in Microsoft Office Word 2003, but it's definitely not the only one. Developers can also attach custom XML Schema definitions and XSLT transformations to Word documents. Developers can mark up content with elements from the attached schemas, making it possible to inject meaningful business-specific markup that simplifies processing down the road. When you save a document in Word, it can validate the document against the attached schema and apply a custom XSLT transformation during the process. The remainder of this column looks at some of these compelling new features in more detail.

Introduction to WordML

The WordML schema was designed to mirror the information found in a traditional .doc file. The schema for WordML is part of the Microsoft Word XML Content Development Kit (CDK) Beta 2, which you can download from MSDN at Microsoft Office 2003 Downloads. The root element of a WordML document is always w:wordDocument. w:wordDocument contains several other elements that represent the complete Word document structure including properties, fonts, lists, styles, and the actual document body that contains sections and paragraphs, as illustrated in Figure 1.

Figure 1 WordML Basic Structure (Close Figure 1)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?mso-application progid="Word.Document"?>
<w:wordDocument
 xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml"
 xmlns:o="urn:schemas-microsoft-com:office:office"
 ...
>
   <o:DocumentProperties>
      <!-- generic office document properties -->
   </o:DocumentProperties>
   <w:fonts>
      <!-- fonts used in document -->
   </w:fonts>
   <w:styles>
      <!-- named styles used in document -->
   </w:styles>
   <w:docPr>
      <!-- Word-specific document properties -->
   </w:docPr>
   <w:body>
      <!-- document content -->
   </w:body>
</w:wordDocument>

Figure 2 shows a Word document with typical formatting including named styles (such as Heading 1), inline styles (bold and italics), and a simple body.

Figure 2 Doc in WordML
Figure 2 Doc in WordML

The o:DocumentProperties element for the document shown in Figure 2 contains the document properties available for all Office documents such as title, author, last author, creation date, last saved date, and so on (see Figure 3). The o:DocumentProperties element is from a different namespace than WordML since it applies to all Office documents and will be shared across other Office XML vocabularies such as ExcelML.

Figure 3 Office Document Properties (Close Figure 3)
<o:DocumentProperties>
  <o:Title>Sample WordML Document</o:Title>
  <o:Author>Aaron Skonnard</o:Author>
  <o:LastAuthor>Aaron Skonnard</o:LastAuthor>
  <o:Revision>2</o:Revision>
  <o:TotalTime>1</o:TotalTime>
  <o:Created>2003-08-14T12:44:00Z</o:Created>
  <o:LastSaved>2003-08-14T12:44:00Z</o:LastSaved>
  <o:Pages>1</o:Pages>
  <o:Words>10</o:Words>
  <o:Characters>59</o:Characters>
  <o:Company>DevelopMentor</o:Company>
  <o:Lines>1</o:Lines>
  <o:Paragraphs>1</o:Paragraphs>
  <o:CharactersWithSpaces>68</o:CharactersWithSpaces>
  <o:Version>11.5329</o:Version>
</o:DocumentProperties>

The w:fonts and w:styles elements contain the font and style information used in the document. Every font or style used in the document will be represented using an XML element within one of these elements. Figure 4 illustrates the font and style information for the document you saw in Figure 2.

Figure 4 Document Font and Style Information (Close Figure 4)
<w:fonts>
  ...
  <w:font w:name="Arial">
    <w:panose-1 w:val="00000400000000000000"/>
    <w:charset w:val="00"/>
    <w:family w:val="Auto"/>
    <w:pitch w:val="variable"/>
    ...
  </w:font>
  ...
</w:fonts>
<w:styles>
  <w:style w:type="paragraph" w:styleId="Normal">
    <w:name w:val="Normal"/>
    <w:rsid w:val="0052557F"/>
    <w:rPr>
      <w:rFonts w:ascii="Arial" w:h-ansi="Arial"/>
      <wx:font wx:val="Arial"/>
      <w:sz w:val="24"/>
      <w:sz-cs w:val="24"/>
      <w:lang w:val="EN-US"/>
    </w:rPr>
  </w:style>
  <w:style w:type="paragraph" w:styleId="Heading1">
    <w:name w:val="heading 1"/>
    <wx:uiName wx:val="Heading 1"/>
    <w:basedOn w:val="Normal"/>
    <w:next w:val="Normal"/>
    <w:rsid w:val="0052557F"/>
    <w:pPr>
      <w:pStyle w:val="Heading1"/>
      <w:keepNext/>
      <w:spacing w:before="240" w:after="60"/>
      <w:outlineLvl w:val="0"/>
    </w:pPr>
    <w:rPr>
      <w:rFonts w:ascii="Arial" w:h-ansi="Arial"
                w:cs="Arial"/>
      <wx:font wx:val="Arial"/>
      <w:b/>
      <w:b-cs/>
      <w:kern w:val="32"/>
      <w:sz w:val="32"/>
      <w:sz-cs w:val="32"/>
    </w:rPr>
  </w:style>
    ...
</w:styles>

The w:docPr element contains the Word-specific properties for the given document such as view and zoom settings, as illustrated in the following snippet:

<w:docPr>
  <w:view w:val="print"/>
  <w:zoom w:percent="100"/>
  <w:doNotEmbedSystemFonts/>
  <w:proofState w:spelling="clean" w:grammar="clean"/>
  <w:attachedTemplate w:val=""/>
  <w:defaultTabStop w:val="720"/>
  <w:characterSpacingControl w:val="DontCompress"/>
  ...
</w:docPr>
And finally, w:body is where the document content goes. The w:body element contains sections, and sections contain paragraphs and tables. Paragraphs and tables are modeled using the w:p and w:tbl elements, respectively. Paragraphs and tables ultimately consist of text run elements (w:r), which can be annotated with additional properties and styles. Figure 5 illustrates the w:body element for the document shown in Figure 2.

Figure 5 Document Body (Close Figure 5)
<w:body>
  <wx:sect>
    <wx:sub-section>
      <w:p> <w:pPr> <w:pStyle w:val="Heading1"/> </w:pPr>
        <w:r> <w:t>Sample </w:t> </w:r>
        <w:proofErr w:type="spellStart"/>
        <w:r> <w:t>WordML</w:t> </w:r>
        <w:proofErr w:type="spellEnd"/>
        <w:r> <w:t> Document</w:t> </w:r>
      </w:p>
      <w:p>
        <w:r> <w:t>The </w:t> </w:r>
        <w:r> <w:rPr> <w:i/> </w:rPr> <w:t>quick</w:t> </w:r>
        <w:r> <w:t> brown </w:t> </w:r>
        <w:r> <w:rPr> <w:b/> </w:rPr> <w:t>fox</w:t> </w:r>
        <w:r> <w:t> jumped over the lazy dog.</w:t> </w:r>
      </w:p>
      ...
    </wx:sub-section>
  </wx:sect>
</w:body>

As you can see, all of the formatting in the original Word document is represented somehow in the WordML document. This makes it possible to move data between .doc and .xml files without losing any information.

Working with WordML

The major benefit of having your documents in WordML format is that you can process them using any XML API (DOM, SAX, XmlReader, and so forth) on any platform using any programming language. You can also process WordML documents using higher-level XML services like XPath and XSLT. For example, it would be trivial to open a WordML document and identify all w:p elements marked with the Heading 1 style using the following XPath expression:

//w:p[w:pPr/w:pStyle/@w:val = 'Heading 1']
You can also write XSLT transformations to move between WordML and other text-based formats. For example, the XSLT shown in Figure 6 illustrates one way to transform WordML into a simple HTML structure. Running this transformation against the WordML document shown in Figure 2 produces the HTML document shown in Figure 7. (Another transform can be found at WordML to HTML XSL Transformation.

Figure 6 Simple WordML to HTML Transformation (Close Figure 6)
<xsl:transform version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:w="http://schemas.microsoft.com/office/word/2003/
wordml">

  <xsl:template match="/">
    <html>
      <xsl:apply-templates />
    </html>
  </xsl:template>

  <xsl:template match="w:body">
    <body style="font-family: {//w:style[@w:styleId='Normal']/w:rPr/
      w:rFonts/@w:ascii};">
      <xsl:apply-templates />
    </body>
  </xsl:template>

  <xsl:template match="w:p">
    <p>
      <xsl:apply-templates/>
    </p>
  </xsl:template>

  <xsl:template match="w:pStyle[@w:val='Heading1']">
      <xsl:attribute name="style">font-size:16pt;
      font-weight:bold;</xsl:attribute>
  </xsl:template>

  <xsl:template match="w:r">
    <span>
      <xsl:apply-templates/>
    </span>
  </xsl:template>

  <xsl:template match="w:rPr">
    <xsl:attribute name="style">
      <xsl:apply-templates/>
    </xsl:attribute>
  </xsl:template>

  <xsl:template match="w:t/text()">
    <xsl:value-of select="." />
  </xsl:template>

  <xsl:template match="w:i">font-style:italic;</xsl:template>
  <xsl:template match="w:b">font-weight:bold;</xsl:template>
  <xsl:template match="text()" />

</xsl:transform>
Figure 7 Doc Converted to HTML
Figure 7 Doc Converted to HTML

In addition to making it easier to process Word documents, WordML also makes it easier to generate Word documents. Since they're just XML, you can generate Word documents using any of the standard techniques available for producing XML. For example, it's trivial to write ASP.NET pages that generate WordML documents on the fly, a technique that's illustrated in Figure 8.

Figure 8 Generating WordML with ASP.NET (Close Figure 8)
<%@ Page language="c#" %>
<% Response.ContentType="text/xml"; %>
<?mso-application progid="Word.Document"?>
<w:wordDocument ...>
...
  <w:body>
    <wx:sect>
      <wx:sub-section>
        <w:p>
          <w:pPr>
          <w:pStyle w:val="Heading1" />
        </w:pPr>
        <w:r>
         <w:t>Hello <%=Request.QueryString["name"]%></w:t>
        </w:r>
...
Figure 9 ASP.NET Page
Figure 9 ASP.NET Page

This ASP.NET page produces the same document, only the heading now contains "Hello [name]", where [name] is replaced with the value supplied in the query string. Browsing to this ASP.NET page with a query string of "?name=Aaron" produces the document shown in Figure 9.

Custom Schemas

In addition to WordML, Microsoft Office Word 2003 also includes support for custom XML schema definitions (XSDs), making it possible to attach one or more custom schemas to a given Word document. You can then annotate the document with the elements found in the attached schemas. This allows you to inject business-related markup into your documents so you can process documents around business identifiers instead of the more generic WordML markup.

For example, consider an XML document that contains information about a new employee. You can annotate this document with elements found in the employee schema shown in Figure 10. To accomplish this you must first attach the schema to the XML document in Word.

Figure 10 Employee Schema Definition (Close Figure 10)
<?xml version="1.0" encoding="utf-8" ?>
<xs:schema targetNamespace="urn:employees"
   elementFormDefault="qualified"
   xmlns="urn:employees" xmlns:mstns="urn:employees"
   xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
    <xs:simpleType name="SSN">
        <xs:restriction base="xs:string">
            <xs:pattern value="\d{3}-\d{2}-\d{4}" />
        </xs:restriction>
    </xs:simpleType>
    <xs:simpleType name="Phone">
        <xs:restriction base="xs:string">
            <xs:pattern value="\d{3}-\d{3}-\d{4}" />
        </xs:restriction>
    </xs:simpleType>
    <xs:complexType name="Name" mixed="true">
        <xs:sequence>
           <xs:element name="First" type="xs:string" />
           <xs:element name="Middle" type="xs:string" />
           <xs:element name="Last" type="xs:string" />
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name="Employee" mixed="true">
        <xs:all>
            <xs:element name="ID" type="SSN" />
            <xs:element name="Name" type="Name" />
            <xs:element name="Phone" type="Phone" />
        </xs:all>
    </xs:complexType>
    <xs:element name="Employee" type="Employee"/>
</xs:schema>

You attach an XML Schema definition to a Word document by selecting Tools | Template and Add-Ins from the menu. The dialog that comes up has an XML Schema tab where you can manage your schema library and choose the schemas you want to attach to this particular document (see Figure 11). You can also indicate if Word should validate documents and whether it should even allow users to save invalid documents.

Figure 11 Managing Schemas
Figure 11 Managing Schemas

After you've attached the schema, you can start annotating the document with the elements found in it. The XML Structure pane in Word (displayed on the right-hand side) lets you inject custom elements from the attached schemas into the document. The XML Structure pane also gives you an interface to the custom elements currently found throughout the document by displaying the logical tree structure of these elements within the pane.

Figure 12 shows the XML Structure pane and how you can select a custom element to insert in the document. In this example, I'm selecting the Employee element from the urn:employees namespace into the document. When I double-click on a custom element, Word wraps the element around the currently highlighted text. The first time you insert a custom element, Word asks you if it should surround the entire document, which is what you'll usually want.

Figure 12 Inserting a Custom Element into a Doc
Figure 12 Inserting a Custom Element into a Doc

After you do this, Word will display the XML tags in the document so you can see where they appear. Figure 12 illustrates what the document looks like after inserting the Employee element around the entire document. If you don't want to see the visual XML tags, you can hide them by deselecting the "Show XML tags in the document" checkbox.

Figure 13 Tree Structure of Embedded Elements
Figure 13 Tree Structure of Embedded Elements

As you insert additional elements into the document, Word displays the logical tree structure of the custom elements at the top of the XML Structure pane. This view only shows the embedded custom elements, making it easy to select the text within them (as illustrated with the Last element in Figure 13).

Schema Validation

Once you insert a custom element, Word will begin validating the content (assuming you selected that option when you attached the schema). The purple line appearing down the left side of the document (in Figure 12) indicates that the Employee element's content is currently invalid (because it's missing some required child elements, in this case). The yellow icon next to the Employee element in the XML Structure pane indicates the same thing. Right-clicking on either of these items displays the problem.

When you put the cursor within the Employee element, Word automatically determines which child elements are allowed under Employee according to the schema and displays them at the bottom of the XML Structure pane. In this example, Word shows the ID, Name, and Phone elements, making it easy to select some text and apply them quickly (see Figure 12).

Word also validates text content against the simple type definitions found in your schema. For example, the schema in Figure 10 contains two simple types: one called SSN and the other called Phone. Both of these simple types restrict strings using a pattern facet (a regular expression). Figure 14 illustrates simple type validation in action. Notice that a purple line shows up underneath the text if it isn't within the simple type's value space. If you right-click on it in this case, Word indicates that the text doesn't follow the required pattern (\d{3}-\d{2}-\d{4}). All of this happens on the fly while you're typing.

Figure 14 Type Validation
Figure 14 Type Validation

Once you've added all of the required elements and made the text conform to the simple type definitions in play, the document should look like Figure 13. You know the document is valid when all of the purple lines disappear. By default, Word doesn't let you save docs with validation problems as XML. However, selecting the "Allow saving as XML even if not valid" option (see Figure 11), makes it possible to save docs even though they're not completely valid yet. When you save a doc with custom elements you have to decide whether to include the WordML markup in the output.

Saving Documents

You have two options when saving a Word document with custom elements. The default option is to save the document as WordML with the custom elements nested throughout the tree. The other option is "Save data only", which removes WordML markup and only persists the custom element tree structure.

For example, Figure 15 illustrates where the Employee and ID elements would appear in the WordML for the document shown in Figure 13. One thing to notice is that according to the schema, the Employee element isn't officially allowed to have child elements other than ID, Name, and Phone—so the child w:p element makes the document invalid. You may ask yourself how Word can consider the document to be valid, since there weren't any purple indicators before saving. It turns out Word is smart enough to determine validity, ignoring the presence of WordML markup since you can choose to exclude it.

Figure 15 WordML with Custom Elements (Close Figure 15)
<w:body>
  <wx:sect>
    <wx:sub-section>
      <ns0:Employee>
        <w:p>
          <w:pPr>
            <w:pStyle w:val="Heading1"/>
          </w:pPr>
          <w:r>
            <w:t>New Employee Record</w:t>
          </w:r>
        </w:p>
        <w:p/>
        <w:p>
          <w:r>
            <w:t>Social Security Number: </w:t>
          </w:r>
          <ns0:ID>
            <w:r>
              <w:t>555-55-5555</w:t>
            </w:r>
          </ns0:ID>
        </w:p>
Figure 16 Saving As Data Only
Figure 16 Saving As Data Only

When you choose "Save data only" (see Figure 16) Word removes the WordML markup and only saves the custom elements found in the document. Doing this, however, causes you to lose any special formatting you may have applied to the document. Figure 17 shows what the employee document looks like when saved this way. Notice it's a much simpler and cleaner XML document that completely conforms to the attached schema.

Figure 17 Doc Without Formatting
Figure 17 Doc Without Formatting

Applying Transforms

When saving a document, you might want to generate a completely different format altogether (perhaps not even XML). To accommodate this, Word makes it possible to apply an XSLT transformation during the save process. You can specify a transform by selecting the Apply transform option available in the Save As dialog (see Figure 16). Simply choose an XSLT from disk and Word will run the transformation and output the result every time you save the document.

You can associate multiple transformations with a given schema through the Schema Library, creating what's known as a Word solution. This makes the transformations available for all documents that implement the given schema. When you have multiple XSLT transformations available for a given document, Word makes them available to you in the XML Document pane (located on the right), allowing you to easily switch between views of the same document within Word.

Word also supports using transformations when you open a document. Word provides an Open as XML option in the Open dialog that lets you choose the particular transformation you'd like to use. When you select this option, Word opens the result of applying the transformation instead of the original document that you selected to open.

Summing It Up

Microsoft Office Word 2003 is miles ahead of its predecessors when it comes to XML support. The native support for WordML makes it possible to build powerful and flexible content solutions without leaving the comfort of your XML APIs. Plus, the built-in support for XML Schema, real-time validation, and different XSLT techniques makes working with custom XML schemas and output formats a breeze.

In addition to the core XML functionality described here, Microsoft Office Word 2003 is also completely programmable through Visual Basic® for Applications and the Microsoft Word XML CDK Beta 2. Microsoft Office Word 2003 will revolutionize the way organizations work with the documents that drive their business. For a complete example, check out the DocLibrary (Contoso, LLC) sample that's included in the CDK (Microsoft Office 2003 Downloads).

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard is an Assistant Professor and Director of .NET Projects at Northface University, where he's working to establish the finest university in the world for software developers. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://www.skonnard.com.


Advanced Basics: SQL Server Metadata

Advanced Basics
SQL Server Metadata
Ken Spencer

Code download available at: AdvancedBasics0311.exe (135KB)

Q Frequently when I am working on a project, I must find the name of a particular field in a table in SQL Server™ 2000. Other times I must find the exact name of a field and table in which it is located. Still other times I need to get the datatype of a particular field. How can I accomplish these tasks easily?

A When I've needed the name of a field or a field's datatype, I've often opened the table I thought it was in and gone poking around. This process is not very efficient because the fields in a table are not in alphabetical order and as a result it's not easy to find a field, especially when the table has a large number of fields. After many rounds of this kind of trial and error searching, it dawned on me—why not build a tool that does the searching? That tool is the subject of this month's column.

The first version of the tool's interface is shown in Figure 1, which shows where I search for a field called Region in all tables in the Northwind sample database. Version 1 of the tool allows you to search for a field in either all tables or a specific table. You can also search for part of the field name (the default) or look for an exact match by unchecking the Contains box. This allows you to vary the search, as the output shows, to look for a word in a field name or find an exact field.

Figure 1 SQL Server Search Tool
Figure 1 SQL Server Search Tool

Let's see how this tool was constructed. The first bit of code for the tool, shown in Figure 2, is used to execute the required SQL. I also added the following global variables at the form level:

Dim localServer, localDB, localUser, localPassword As String
Dim localConnectionString As String
Dim dsTables As DataSet

Figure 2 SQL Routines (Close Figure 2)
Public Function RunSQLWithDataSet(ByVal sSQL As String, Optional _
    ByVal TableName As String = "") As DataSet
    Dim oDataSet As New DataSet
    Dim oDataAdapter As SqlClient.SqlDataAdapter

    Try

        If TableName = "" Then
            TableName = "Table1"
        End If

        'Create New DataAdapter
        oDataAdapter = New _
        SqlClient.SqlDataAdapter(sSQL, localConnectionString)

        'Fill Data Set from dataadapter
        oDataAdapter.Fill(oDataSet, TableName)
        'Set return value of function
        Return oDataSet
    Catch e As Exception
        Throw New Exception( _
            "An exception occured in RunSQLWithDataSet", e)
    Finally
    End Try
End Function

Public Function RunSQLScalar(ByVal sSQL As String) As String

    Dim oCommand As New SqlClient.SqlCommand
    Dim oConnection As New _
        SqlClient.SqlConnection(localConnectionString)
    Dim ret As String
    Try

        oConnection.Open()

        oCommand.Connection = oConnection

        oCommand.CommandText = sSQL

        ret = oCommand.ExecuteScalar

        Return ret
    Catch e As Exception
        Throw New Exception("An exception occured in RunSQLScalar", e)
    Finally
    End Try
End Function

To get access to the database, you must provide the user with a way to select the database they want to query. The form named frmDBInfo asks the user for the database login information. To display the form, frmSelectDB calls the SelectDBandLoadTables routine shown in Figure 3. This routine simply displays the form and retrieves the information from the user. Then to actually retrieve the tables, it calls the GetTableNames routine, shown here:

Function GetTableNames() _
    As DataSet
    Dim SQL As String
    Dim ds As DataSet

Figure 3 SelectDBandLoadTables Routine (Close Figure 3)
Sub SelectDBandLoadTables ()
    'Allow user to select database

    Dim frm As New frmDBInfo

    Me.SuspendLayout()

    frm.Location = Me.Location

    If localServer <> "" Then
        frm.txtServer.Text = localServer
    End If
    If localDB <> "" Then
        frm.txtDatabase.Text = localDB
    End If
    If localUser <> "" Then
        frm.txtUser.Text = localUser
    End If
    If localPassword <> "" Then
        frm.txtPassword.Text = localPassword
    End If

    frm.ShowDialog()

    If frm.DialogResult = DialogResult.OK Then
        'load stored procs from database
        localServer = frm.txtServer.Text
        localDB = frm.txtDatabase.Text
        localUser = frm.txtUser.Text
        localPassword = frm.txtPassword.Text

        localConnectionString = "server=" & localServer & _
            ";database=" & localDB & ";uid=" & _
            localUser & ";pwd=" & localPassword & ";"

        Application.DoEvents()
        Try

            dsTables = GetTableNames()

            cboTables.DataSource = dsTables.Tables(0)
            cboTables.ValueMember = "Name"
            cboTables.DisplayMember = "Name"

        Catch exc As Exception
            StatusBar1.Text = "An error occurred."
        End Try
    Else

    End If

    frm.Close()

    Me.ResumeLayout()
End Sub

The real work is done in the select statement:

SQL = "SELECT Name FROM sysobjects "
SQL &= "WHERE (type = 'U') AND (status > 0) "
SQL &= "order by Name"
This statement pulls the table names from the sysobjects table in the selected database (later I'll look at another way to do this). There is a tremendous amount of useful data in sysobjects and other system tables that can be extracted for your own purposes. Executing this SQL statement yields a list of all the tables which are returned in a DataSet:
  ds = RunSQLWithDataSet(SQL)
  Return ds
The DataSet's table(0) is then loaded into the combobox. The table is also retained in the dsTables DataSet for later use.

Now, let's turn to another routine that I'll need to complete the search: GetTableDetails (see Figure 4). This routine uses the SQL query shown here to retrieve the details of a particular table, including field names, datatype, and maximum length:

 SQL = "select '" & TableName & "' as TableName, " _
    SQL &= "COLUMN_NAME, DATA_TYPE, " _
    SQL &= "CHARACTER_MAXIMUM_LENGTH  from " _
    SQL &= "INFORMATION_SCHEMA.COLUMNS where " _
    SQL &= "table_name = '" & TableName & "' "

Figure 4 GetTableDetails (Close Figure 4)
Function GetTableDetails(ByVal TableName As String) As DataTable
    Dim ds As DataSet
    Dim rw, workrow As DataRow
    Dim dt As New DataTable("TableInfo")
    Dim localColumn, localDataType As String
    Dim localLength As Integer
    Dim localIdentity As Boolean
    Dim localOutput As String
    Dim SQL As String

    SQL = "select '" & TableName & "' as TableName, " _
        SQL &= "COLUMN_NAME, DATA_TYPE, " _
        SQL &= "CHARACTER_MAXIMUM_LENGTH  from " _
        SQL &= "INFORMATION_SCHEMA.COLUMNS where " _
        SQL &= "table_name = '" & TableName & "' "

    ds = RunSQLWithDataSet(SQL, "RawTableInfo")

    dt.Columns.Add("TableName", Type.GetType("System.String"))
    dt.Columns.Add("ColumnName",Type.GetType("System.String"))
    dt.Columns.Add("DataType", Type.GetType("System.String"))
    dt.Columns.Add("Length", Type.GetType("System.Int64"))
    dt.Columns.Add("Identity", Type.GetType("System.Boolean"))

    For Each rw In ds.Tables("RawTableInfo").Rows
        If Not IsDBNull(rw.Item("Column_Name")) Then
        localColumn = rw.Item("Column_Name").ToString
        If Not IsDBNull(rw.Item("DATA_TYPE")) Then
            localDataType = rw.Item("DATA_TYPE").ToString
          End If
          If Not IsDBNull(rw.Item("CHARACTER_MAXIMUM_LENGTH")) Then
            localLength = CInt( rw.Item("CHARACTER_MAXIMUM_LENGTH") _
               .ToString)
        Else
            localLength = 0
        End If

        If LCase(localDataType) = "int" Then
            localIdentity = CheckColumnIdentity(TableName, localColumn)
        Else
            localIdentity = False
        End If

        workrow = dt.NewRow()
        workrow("TableName") = TableName
        workrow("ColumnName") = localColumn
        workrow("Datatype") = localDataType
        workrow("Length") = localLength
        workrow("Identity") = localIdentity
        dt.Rows.Add(workrow)
        End If
    Next
    Return dt
End Function

GetTableDetails is called when the user clicks the Search button (called cmdSearch). More on this soon.

One more function that I need to look at is CheckColumnIdentity.

Function CheckColumnIdentity(ByVal TableName As String, _
    ByVal ColumnName As String) As Boolean
    Dim SQL As String
    SQL = "SELECT COLUMNPROPERTY( OBJECT_ID('" & " _
    SQL &= "TableName & "'),'" & ColumnName & _
    SQL &= "','IsIdentity')"
    If CInt(RunSQLScalar(SQL)) = 1 Then
        Return True
    Else
        Return False
    End If
End Function
This function takes a table name and column name as parameters and returns a Boolean value, indicating whether the column is an identity column. It uses COLUMNPROPERTY to determine if the column is an identity column. You can also use this property to determine if a column allows nulls, and so on.

Now, let's put this together by looking at the cmdSearch Click event code. The first few lines of code define the variables I'll need:

Dim dt, dttemp As DataTable
Dim ds As New DataSet
Dim tabletosearch As String
Dim rw As DataRow
Dim IsMatch As Boolean = False

The following two lines set tabletosearch to the name of the table to search and clears the output textbox:

tabletosearch = cboTables.Text
txtOutput.ResetText()

The next few lines control how the search is handled. If chkAllTables is True, then all of the details for each table are loaded into the dt table. If chkAllTables is False, then dt is loaded with the details for only the selected table:

If chkAllTables.Checked Then
    For Each rw In dsTables.Tables(0).Rows
        dttemp = GetTableDetails(rw("Name").ToString)
        ds.Merge(dttemp)
    Next
    dt = ds.Tables(0)
Else
    dt = GetTableDetails(tabletosearch)
End If

Now that the tables are loaded, I can perform the search. The search is handled inside the For Each loop, which moves through all rows in the dt table:

For Each rw In dt.Rows
The next line sets IsMatch to False as the default for each field. IsMatch is a flag that determines whether to output the field:
IsMatch = False

Next, if chkContains is True, then InStr is used to determine if a match exists. If chkContains is False, then the Else clause checks for an exact match. If a match is found, IsMatch is set to True:

If chkContains.Checked Then
    If InStr(UCase(rw("ColumnName").ToString), _
        UCase(txtSearchField.Text)) > 0 Then
        IsMatch = True
    End If
Else
    If UCase(rw("ColumnName").ToString) = UCase(txtSearchField.Text) Then
        IsMatch = True
    End If
End If
Now that the test is complete, I can output the data if IsMatch is True. The values in the various fields in the datatable are output to txtOutput, as shown here:
If IsMatch Then
    txtOutput.Text &= rw("TableName").ToString _
        & " : " & rw("ColumnName").ToString & " - "
    txtOutput.Text &= rw("DataType").ToString _
        & " (" & rw("Length").ToString & ")"

    If CBool(rw("Identity").ToString) Then
        txtOutput.Text &= " Identity"
End If
        txtOutput.Text &= vbCrLf
    End If
Next
That's it. There is not a lot of code to this tool, but it sure is handy.

The information schema views provided by SQL Server 2000 are used to retrieve some of the metadata in this application. These views provide a wealth of metadata about SQL Server databases. Just for fun, I tried using a few different views and found some really cool features. The following SQL statement pulls a list of the tables and views in the database, sorted by table name:

select *
from INFORMATION_SCHEMA.Tables order by table_type, table_name

This SQL query pulls information about the stored procedures in the database, as shown in the following:

select *
from INFORMATION_SCHEMA.ROUTINES
where ROUTINE_TYPE='PROCEDURE'

And if that wasn't cool enough, check this out. The following code pulls the names of all the stored procedures and the code to create them:

select routine_name, routine_definition
from INFORMATION_SCHEMA.ROUTINES
where ROUTINE_TYPE='PROCEDURE'

The first three rows returned for the Northwind sample database are shown in Figure 5.

Figure 5 Sprocs and Code Used to Create Them (Close Figure 5)

Stored Procedure Name (routine_name)
Ten Most Expensive Products
SQL to create it (routine_definition)
CREATE PROCEDURE "Ten Most Expensive Products" AS
    SET ROWCOUNT 10
    SELECT Products.ProductName AS
        TenMostExpensiveProducts, Products.UnitPrice
        FROM Products
        ORDER BY Products.UnitPrice DESC
Stored Procedure Name (routine_name)
Employee Sales by Country
SQL to create it (routine_definition)
CREATE PROCEDURE "Employee Sales by Country"
@Beginning_Date DateTime, @Ending_Date DateTime AS
    SELECT Employees.Country, Employees.LastName,
        Employees.FirstName, Orders.ShippedDate,
        Orders.OrderID, "Order Subtotals".Subtotal
        AS SaleAmount
        FROM Employee
Stored Procedure Name (routine_name)
dt_removefromsourcecontrol
SQL to create it (routine_definition)
CREATE PROCEDURE dbo.dt_removefromsourcecontrol AS
    SET nocount ON declare @iPropertyObjectId int
    SELECT @iPropertyObjectId = (SELECT objectid
        FROM dbo.dtproperties
        WHERE property = 'VCSProjectID')
    EXEC dbo.dt_droppropertiesbyid

Now, let's brainstorm a minute. What can I do with this code? I can put it into my search routine and use InStr to search for particular fields. This would allow me to search not only tables but also stored procedures for the fields. Of course, I could add Views rather easily as well.

One change I would make to the SQL code is shown here:

select routine_name, routine_definition
from INFORMATION_SCHEMA.ROUTINES
where ROUTINE_TYPE='PROCEDURE' and left(routine_name,3) <> 'dt_'
order by routine_name
This version filters out the dt_ procedures system procedures. It also sorts the output by procedure name. You should note that the order by clause here will not work if you run this query against Master db because group by and order by can't handle rows longer than 8060 bytes.

There are quite a few ways you can query a table about itself. I have only scratched the surface of the possibilities here.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencer works for 32X Tech (http://www.32X.com), where he provides training, software development, and consulting services on Microsoft technologies.


The ASP Column: ATL Server Versus ASP.NET

The ASP Column
ATL Server Versus ASP.NET
George Shepherd

Code download available at: ASPColumn0311.exe (534KB)
Contents
What Exactly is ASP.NET?
How is ATL Server Different?
ATL Server Versus ASP .NET
ASPX Files Versus SRFs
The "Intrinsic" Objects
Managing UI Elements
Session State
Conclusion

AWeb server's job is to accept incoming HTTP requests and return something useful to the caller (whether the caller is human or, in the case of Web Services, a machine). Windows® includes a mature infrastructure for handling requests—IIS and related extensions. However, programming IIS from scratch can be rather tedious and often error prone. A better way to manage HTTP requests is to work with one of the frameworks that sit on top of IIS. This month I'll compare the two main technologies for creating Web applications for Windows: ASP.NET and ATL Server. Each framework has some specific advantages and disadvantages. In this installment, I'll focus on managing a Web-based UI with ASP.NET. Next time I'll focus on building Web Services and other features using the two frameworks.

What Exactly is ASP.NET?

ASP.NET is a library of classes designed to handle HTTP requests. In addition to a class library, ASP.NET includes several IIS components for managing requests. These components include an ISAPI DLL named ASPNET_ISAPI.DLL and a worker process named ASPNET_WP.EXE. ASP.NET also installs new mappings in IIS, redirecting file requests for ASPX, ASCX, ASHX, and ASMX to ASPNET_ISAPI.DLL. From there, ASPNET_ISAPI.DLL directs the request to the ASPNET_WP.EXE, where ASP.NET loads the classes required to service the request.

ASP.NET has a convenient object model centering around a managed class named HttpContext. If you've ever written a standard ISAPI DLL, you're aware of the information contained in the EXTENSION_CONTROL_BLOCK passed into an ISAPI DLL's HttpExtensionProc. In managing a request, the ISAPI DLL picks apart that structure to get such information as the context ID, the query string, and functions to read and write to the client. ASP.NET wraps all this information in the HttpContext class. ASP.NET also includes the basic class infrastructure for managing Web-based UIs (through the System.Web.UI.Page class), and for managing Web Services (through the System.Web.Services.WebService class and the [WebMethod] attribute).

ASP.NET is object oriented. Every request passing through an ASP.NET application is handled by a class that implements an interface named IHttpHandler. This makes for a very extensible architecture. You may choose to take advantage of the ASP.NET page architecture or Web Services architecture, or you may write your handling logic from scratch. Figure 1 shows the path an ASP.NET request takes.

Figure 1 The Lifecycle of an ASP.NET Request
Figure 1 The Lifecycle of an ASP.NET Request

The ASP.NET UI-handling architecture centers around the System.Web.UI.Page class (represented by ASPX files). ASPX files may include straight HTML code and server-side controls. When ASP.NET encounters a server-side control (via the "runat=server" property), it instantiates a class representing the control (for example, the Button and the ListBox controls). An ASP.NET page is basically processed as a tree of these server-side controls (literal text and tags on the page are packaged as a LiteralControl)—everything else is a server-side control. When the page is asked to render itself, it simply walks the control tree, telling each node in the tree to render itself. ATL Server works somewhat differently.

How is ATL Server Different?

ATL Server is a C++ template library for creating ISAPI DLLs. When IIS was first making its way in the world, developers had to write ISAPI extensions from scratch or from MFC's ISAPI DLL classes. Building ISAPI DLLs using raw C++ or MFC code required extensive hand coding. For example, the MFC ISAPI didn't include a forms-based architecture for developers. Any HTML tags that ended up on the client had to be emitted by hand.

ATL Server combines a forms-based approach with the runtime speed and flexibility of C++. A Web site built using ATL Server is made up of three basic components: the Server Response File (SRF), an application DLL, and the requisite ISAPI extension. The SRF is a new file type which ATL Server installs in IIS. The SRF mapping points IIS to your application's ISAPI DLL, which in turn directs processing to one or more of your application DLLs. SRFs include a special new tag syntax which basically calls entry points in your application DLLs. Figure 2 shows the path an ATL Server-based request takes through the system.

Figure 2 Path of an ATL Server-based Request
Figure 2 Path of an ATL Server-based Request

ATL Server applications are comprised of a collection of DLLs (ISAPI extensions and application extensions) and HTML-generation templates called SRFs (mentioned earlier). This architecture clearly separates the app presentation from the application logic. A page's presentation is defined by an SRF containing HTML and special tags. These tags call into your ATL Server application DLLs.

As with most Web applications that target the Windows platform, ATL Server applications are built upon ISAPI DLLs. An ATL Server project includes a single ISAPI extension DLL that handles the raw request. ATL Server apps also include one or more application DLLs for more refined request processing. Classes for handling requests derive from CRequestHandlerT and contain your own specialized code for handling the tags in SRFs.

The handler classes contain dictionaries to associate request handler classes with request handler DLLs and associate replacement methods with SRF tags. In addition to the replacement dictionaries, CRequestHandlerT contains methods and member variables for accessing standard Web application elements such as form variables, cookies, request streams, and response streams.

When a browser surfs to a .srf URL via HTTP, IIS knows to open the site with the ATL Server application's ISAPI DLL. The ATL Server app then opens the SRF and loads the application's DLLs (if they weren't already loaded). The application then passes the request to the default request handler class, which parses the SRF looking for specially marked tags. Each time a tag appears in the SRF, the application calls a replacement method in a handler class residing in one of the application DLLs. The replacement methods generate the output to the browser dynamically. Figure 2 shows the path of a request through an ATL Server application.

ATL Server Versus ASP .NET

While ATL Server and ASP.NET are both based upon the ISAPI architecture, they handle requests very differently. To illustrate the differences, let's look at an example application that will collect the name of an individual and his or her development preferences. I'll show how to develop a user interface and how to use session state. In the next installment of this column I'll examine some of the other features (such as caching and browser capabilities) and how Web Services work on each framework. Figure 3 compares some of the features of the two frameworks.

Figure 3 Comparing Features of ASP.NET and ATL Server (Close Figure 3)

Feature | ASP.NET | ATL Server
Recognized file types | ASPX, ASCX, ASMX, ASHX, SOAP, ASAX | SRF
Underlying architecture | IIS | IIS
View state | Handled automatically | Requires intervention
Session state | Off, in-proc, state server, SQL Server | Off, in-proc, OLE DB
Cache support | Managed cache | BLOB support
Web Service support | [WebMethod] | Implements class members

ASPX Files Versus SRFs

Both ASP.NET and ATL Server introduce new file extensions for the ISAPI architecture. ASP.NET introduces the file types ASPX, ASMX, ASCX, and ASHX, among others. Each of these file types has a corresponding managed class within the ASP.NET framework. In the case of ASPX files, that class is System.Web.UI.Page. The Page class is responsible for rendering a UI-based page. Figure 4 shows a simple ASPX file.

Figure 4 Simple ASPX File (Close Figure 4)
<%@ Page language="c#"
  Codebehind="WebForm1.aspx.cs"
  Inherits="DevPreferencesASPNET.WebForm1" %>
  <HTML>
    <form id="Form1" method="post" runat="server">
      <asp:label id="Label1" runat="server">
        Type Name Here:</asp:label>
      <asp:textbox id="TextBoxName" runat="server"/>
      <asp:Label id="Label3" runat="server">
        Type Age Here:</asp:Label>
      <asp:textbox id="TextboxAge" runat="server"/>
      <asp:dropdownlist id="DropDownList1" runat="server">
        <asp:ListItem Value="I hate COM">
         I hate COM</asp:ListItem>
        <asp:ListItem Value="I love C#">
         I love C#</asp:ListItem>
        <asp:ListItem Value="I love ease of development">
         I love ease of development</asp:ListItem>
        <asp:ListItem Value="I love .NET">
         I love .NET</asp:ListItem>
        <asp:ListItem Value="Auto memory management">
         Auto memory management</asp:ListItem>
      </asp:dropdownlist>
    <asp:label id="Label2" runat="server/>

    <asp:button id="ButtonSubmitMe" runat="server"
       Text="SubmitMe"/>
    <asp:label id="LabelFeature" runat="server"/>
    <asp:Label id="Label4" runat="server">
      What's your favorite .NET Feature?
     </asp:Label>
  </form>
</HTML>

The key things to notice about the ASPX file is the Inherits directive near the top of the page and the "runat=server" attributes with the buttons, labels, and dropdownlist tags. These tags represent server-side controls, for which Visual Studio® places corresponding classes in the codebehind file.

By contrast, SRFs contain mostly standard, plain-vanilla HTML tags. ATL Server does not have a server-side component architecture. However, it does introduce the notion of server response flags. These are special tags denoted by double curly braces ({{}}). When the ATL Server request architecture encounters a server response tag, it expects to find a corresponding handler function within the application DLLs. Figure 5 shows a simple SRF that displays roughly the same user interface as the ASPX example in Figure 4.

Figure 5 Simple SRF File (Close Figure 5)
<html>
  <head>
  </head>
  <body>
    <form method="post" action="DevPreferencesATLServer.SRF">
    <P> {{handler DevPreferencesATLServer.dll/Default}}
    This is a test: {{Hello}}</P>
    <P>Type name here: <INPUT id="Text1" type="text"
       name="TextBoxName"></P>
    <P>Type age here: <INPUT id="Text2" type="text" name="TextBoxAge"></P>
    <P> {{ShowNameAndAge}} <br> </P>
    <h2>What's your favorite ATL Server feature?</h2>
    <P><SELECT id="SelectFavoriteFeatures" name="SelectFavoriteFeatures">
       {{ShowFeatureSelection}} </SELECT></P>
    {{ShowFavoriteFeature}}
    <P> <INPUT id="Submit1" type="submit"
        value="Submit Me" name="Submit1"> </P>
    </form>
  </body>
</html>

The most important things to notice in this file are the special tags surrounded by double curly braces. For example, near the top of the SRF, you'll see the handler tag that specifies the default handler for the application ({{handler...}}). This tells the ATL Server application which DLL to load to find the functions to be called by the response tags. The other response tags refer to entry points in the application DLL.

The "Intrinsic" Objects

Both ASP.NET and ATL Server include similar intrinsic Request and Response objects, not unlike classic ASP. In ASP.NET, they are represented by the HttpRequest and HttpResponse classes. In ATL Server, they're represented by the CHttpRequest and the CHttpResponse classes. They serve generally the same purposes within each framework. The Request object encapsulates such items as the request parameters and request URL. The Response object includes functionality for outputting text to the client. For example, to insert "Hello World" into the output stream of an ASP.NET-based request, just call Response.Write, like so:

protected void HelloWorld()
{
  Response.Write("Hello World");
}

To output "Hello World" to the client of an ATL Server-based application, use the CHttpResponse object, like so:

[tag_name(name="Hello") ]
HTTP_CODE OnHello(void) {
  m_HttpResponse << "Hello World!";
  return HTTP_SUCCESS;
}
Notice how the OnHello function is called using the server response tag in Figure 5. (The tag looks like this: {{Hello}}.)

Managing UI Elements

Each of these frameworks takes a different approach to managing UI elements. As mentioned before, UI support in ASP.NET centers around the server-side control model. The presentation code (the ASPX file) declares server-side elements on the page using the "runat=server" attribute. As long as a corresponding class is declared in the codebehind class, accessing the controls programmatically is a breeze. For example, the code in Figure 4 shows several server-side control elements (the submit Button, the TextBox elements, and the DropDownList, for example). The codebehind page declares Button, TextBox, and DropDownList classes as members of the page, making the UI elements available programmatically. To find the data that's in the TextBoxName element, just access the Text property, like so:

String s = this.TextBoxName.Text;

ASP.NET server-side controls also have the advantage of tracking view state automatically. When the browser initiates a round-trip to the server, UI elements like dropdown listboxes and radio buttons retain their state consistently. For example, the last item selected in the dropdown listbox is the item that's shown. You don't need to write any special code to make the control behave correctly.

ATL Server has no such control model. The UI may be managed only through the server response tags. To populate the dropdown list, the ATL Server example includes a server response function to populate it (see the {{ShowFeatureSelection}} tag in Figure 5). Figure 6 shows the server response function that inserts the items in the dropdown list.

Figure 6 Managing a Dropdown Listbox within ATL Server (Close Figure 6)
[tag_name(name="ShowFeatureSelection")]
HTTP_CODE OnShowFeatureSelection()
{
  const CHttpRequestParams&
    reqParms=m_HttpRequest.GetFormVars();

  CString strFeature = reqParms.Lookup("SelectFavoriteFeatures");

  this->m_HttpResponse << "<option ";
  if(strFeature == "I Love COM") {
    this->m_HttpResponse << " selected ";
  }
  this->m_HttpResponse << "> I Love COM </option>";

  this->m_HttpResponse << "<option ";
  if(strFeature == "I Love C++") {
    this->m_HttpResponse << " selected ";
  }
  this->m_HttpResponse << "> I Love C++ </option>";

  this->m_HttpResponse << "<option ";
  if(strFeature == "I Love speed") {
    this->m_HttpResponse << " selected ";
  }
  this->m_HttpResponse << "> I Love speed </option>";

  this->m_HttpResponse << "<option ";
  if(strFeature == "I Love Control") {
    this->m_HttpResponse << " selected ";
  }
  this->m_HttpResponse << "> I Love Control </option>";

  this->m_HttpResponse << "<option ";
  if(strFeature == "ATL Server works better with my COM components") {
    this->m_HttpResponse << " selected ";
  }
  this->m_HttpResponse << "> ATL Server works better with my
                             COM components </option>";

  return HTTP_SUCCESS;
}

ATL Server does not track view state in the same way that ASP.NET does. Keeping the dropdown listbox consistent requires the kind of code shown in Figure 6. The code examines the parameters coming in on the query string to find out which item was selected. The rendering code ensures that the item selected by the user includes the "selected" attribute within the option tag.

Session State

Session state management in ASP.NET is extremely convenient. The ASP.NET worker process handles the grungy details. When a session is started by a new client, ASP.NET automatically allocates a new Session object—a dictionary containing name-value pairs. The System.Web.UI.Page class includes a reference to the current client's session information. Accessing the client's state simply involves accessing the session dictionary with an indexer. Figure 7 shows the code necessary for creating new session variables and accessing them later using ASP.NET.

Figure 7 Managing Session State in ASP.NET (Close Figure 7)
private void ButtonSubmitMe_Click(object sender, System.EventArgs e)
{
  Session["Name"] = this.TextBoxName.Text;
  Session["Age"] = this.TextboxAge.Text;
  SetLabel();
}

protected void SetLabel()
{
  string s = (string)Session["Name"];
  if(s != null)
  {
    this.Label2.Text = "Hello " + s;
  }
  else
  {
    this.Label2.Text = "Who are you?";
  }

  object oAge = Session["Age"];
  if(oAge != null)
  {
    this.Label2.Text += " You're age " + oAge.ToString();
  }
  else
  {
    this.Label2.Text += " How old are you?";
  }
}

Managing session state in ATL Server is a bit more complex. While the ISAPI DLL generated by the ATL Server Wizards includes a session manager, you still need to write the code for creating and accessing session objects. The code in Figure 8 shows how to create and access session variables in ATL Server.

Figure 8 Managing Session State in ATL Server (Close Figure 8)
HTTP_CODE ValidateAndExchange()
{
  // Get the ISessionStateService from the ISAPI extension
  HRESULT hr;
  hr=m_spServiceProvider->QueryService(__uuidof(ISessionStateService),
                                       &m_spSessionSvc)
  if(FAILED(hr))
    return HTTP_FAIL;

  CCookie sessionCookie;
  sessionCookie = m_HttpRequest.GetSessionCookie();
  if(sessionCookie.IsEmpty()){//no Session
    char newSessionID[MAX_SESSION_KEY_LEN];
    DWORD temp=MAX_SESSION_KEY_LEN;
    m_spSessionSvc->CreateNewSession(newSessionID,&temp,&m_spSession);
    CSessionCookie cookieSession(newSessionID);
    m_HttpResponse.AppendCookie(cookieSession);
  } else {
    CStringA strSessionID;
    sessionCookie.GetValue(strSessionID);
    HRESULT hr;
    CComVariant vAge;
    CComVariant vName;
    hr = m_spSessionSvc->GetSession(strSessionID,&m_spSession);
    hr = m_spSession->GetVariable("Age", &vAge);
    if(SUCCEEDED(hr)) {
      m_nAge = vAge.lVal;
    } else {
      OnGetAge();
    }
    hr = m_spSession->GetVariable("Name", &vName);
    if(SUCCEEDED(hr)) {
      m_strAge = vName.bstrVal;
    } else {
      OnGetName();
    }
  }
  return HTTP_SUCCESS;
}

The ValidateAndExchange function was generated by the ATL Server Wizard. This method is called at the beginning of each request in much the same way the ASP.NET Page class includes an Init and a Load event. ValidateAndExchange acquires an interface to the ISAPI DLL's session manager (via QueryService). If there isn't already a session cookie in the headers, the method creates a new session and adds a session cookie to the response. If this is a continuation of an existing session, the code uses the session information to populate the name and age variables. Notice that the session variables are represented by COM variants.

Conclusion

This month I took a quick survey of ASP.NET and ATL Server. These frameworks represent the current state-of-the-art in processing HTTP requests on the Windows platform. Each of these frameworks builds upon the time-tested ISAPI architecture that emerged nearly a decade ago. ASP.NET and ATL Server map their file extensions to specific DLLs. The ASP.NET DLL is ASPNET_ISAPI.DLL. The ATL Server DLL is generated by ATL Server, but for the most part it's boilerplate code. One advantage ATL Server has in this respect is that you may actually review (and change) the raw ISAPI request-handling code for your particular application.

While you may use either framework to write generally equivalent applications, the standard trade-offs apply here. It's usually very straightforward to develop an ASP.NET application using a managed language and the ASP.NET framework. However, you are at the mercy of the framework (though ASP.NET mitigates the flexibility issue by being very extensible). On the other hand, once your ATL Server application has gotten past IIS, it consists mostly of a bunch of DLL calls—a very high-performance proposition. ATL Server applications are written in C++, so you have a great deal of control, along with the requisite responsibility.

Next time I'll compare some other features of ASP.NET and ATL Server, including caching state, security, and Web Services.

Send your questions and comments for George to asp-net@microsoft.com.

George Shepherd specializes in software development for the .NET Framework. He is the author of Programming with Microsoft Visual C++.NET (Microsoft Press, 2002), and coauthor of Applied .NET (Addison-Wesley, 2001). He teaches seminars with DevelopMentor and is a contributing architect for Syncfusion's .NET tools.


Bugslayer: Google from Visual Studio .NET

Bugslayer
Google from Visual Studio .NET
John Robbins

Code download available at: Bugslayer0311.exe (669KB)
Contents
CommandBars from Hell
F1 Monitoring?
The Real Google Add-in
Implementation Highlights
Wrap Up
The Tips

One key to maximizing your software development efforts is finding answers to your coding challenges as quickly as possible. When it comes to searching for those answers on the Internet, all roads lead to Google, as you undoubtedly know. Not only is Google the most comprehensive search engine, it's also the fastest.

Knowing this, it amazes me how few developers have taken the time to read the docs for Google. Google can return a ton of hits for any query you make. To make the search really efficient, the trick is to tell Google how to scope the search so that instead of getting a billion results, you get the 100 best.

In order to scope things better, Google has set up special search groups for specific topics, including information for Microsoft, Apple Macintosh, BSD, Linux, and the U.S. Government. For instance, when you search from http://www.google.com/microsoft.html, your query will be run only on sites that have predominantly Microsoft content. To use the specific sites, point your browser at http://www.google.com/options/specialsearches.html.

In addition, you can limit your search to a single site. In the search edit control enter "site:" without the quotes followed by the site you want to search, or use the advanced search page. To search MSDN® Online for example, enter "site:msdn.microsoft.com" before your search string.

I'd strongly encourage you to read the complete Google search documentation, which you can find at http://www.google.com/about.html. The answers to all of your development problems are out there on the Internet, but without smart searching, you'll look for them forever.

OK, so my two Google tips alone are worth the price of this column, but there's more. Because I'm fundamentally lazy, I don't like opening a browser and typing "site:msdn. microsoft.com" into Google's search box every time I have a question. I want to click a button to be able to do the Google searches I've mentioned, along with Usenet groups, Google Directory, and whole-Web searches from right inside Visual Studio® .NET. Hey, it's nothing that a nice little add-in couldn't handle. For a preview of what it's going to look like, check out Figure 1.

Figure 1 Google Add-in
Figure 1 Google Add-in

My original intent was that this column would be short, sweet, and give you a utility to search the Internet but I ran into some very strange limitations in the add-in model and had to do some considerable poking around to circumvent them. My hope is that I'll save you a lot of trouble (and a lot of cursing at the computer) when you set out to develop your own add-ins.

For the rest of this column, I'm going to assume that you're already familiar with the basics of how add-ins work and how to write them. If you want to get up to speed, read the January 2002 Bugslayer column or the Visual Studio .NET documentation itself.

CommandBars from Hell

My original design had the entire Google add-in placed in a single toolbar. The first control would be a combobox that would mimic the existing Find combobox by saving past searches. The following controls would be all buttons that would take the text from the combobox. The order of those buttons would be: Search MSDN Online, Search Microsoft sites, Search Google groups, Search Google Directory, and Search the Web.

CommandBars are the type for toolbars and menus in Visual Studio. The documentation for CommandBars indicates they come from the Microsoft Office DLL, MSO.DLL. The Visual Studio .NET documentation does not directly address CommandBars and simply refers to the Microsoft® Office documentation. The problem with this approach is that the documentation implies that the Visual Studio .NET CommandBars are as capable as the Office CommandBars but, as you'll see in just one moment, they are not. This means you're left to muddle through the Visual Basic® for Applications examples in the Office XP Developer documentation.

As I already had some experience wrestling with Visual Studio .NET CommandBars, I realized my first task would be to prototype the combobox so I could figure out its limitations. I suspect this is only a problem with the add-in model and may be fixed with the Visual Studio Integration Program (VSIP) model, but I haven't checked, mainly because writing VSIP code is more involved and can't be done as managed code. Since most of you are going to be sticking with the add-ins, it's good to finally find out what the limitations are.

In the source code for this month's column you can find my CommandBar testing application, called ToolBarTest, which is written in Visual Basic .NET (see the link at the top of this article). You may want to compile and install the add-in so you can see the limitations for yourself. The installation is accomplished by running the TOOLBARTEST.REG file that's part of the project.

Adding a combobox is as simple as calling the Add method of the Controls property in a toolbar. As you can see in the downloadable code, I only create and populate the toolbar when the IDTExtensibility2.OnConnection method is called and the ext_ConnectMode parameter is ext_cm_UISetup, which indicates the add-in is being told to perform its initial user interface setup. As the user interface setup is only supposed to happen once, on subsequent loads of the add-in, I hunt down the already-created toolbar and make it active.

On the second run, my code to find the toolbar worked correctly, but the combobox I created in the first position on the toolbar was no longer there. I was quite perplexed and thought that maybe it was hidden and I needed to "unhide" it. I whipped out a quick macro to enumerate the controls on my toolbar and sure enough the combobox I had added was missing in action. After resetting the CommandPreload option to 1 to force the add-in user interface initialization and rerunning Visual Studio, my combobox came back. Sadly, a restart still lost the combobox.

Those of you who have done some work with CommandBars might be wondering if I mistakenly set the Temporary parameter to true when calling CommandBars.Controls.Add. As you can see, I made sure to add the combobox as a non-temporary control. At this point, I figured that with the add-ins model, it looks like only buttons are saved. My workaround was when the add-in loaded normally in the IDTExtensibility2.OnConnection method, I'd hunt down the CommandBar for my add-in and manually poke the combobox into the first position on the toolbar. You can see that code in Connect.AddComboBox inside ToolBarTest.

While it's odd that the combobox isn't saved for a toolbar, at least you have a partial workaround. The problem occurs when your toolbar is visible but your add-in isn't loaded at startup. In that case, your toolbar won't show up correctly. You can see this demonstrated in the ToolBarTest add-in. While you could tell the user to ensure your add-in is always loaded, that won't always be the case. If you play with ToolBarTest, you'll see that if it's not loaded, clicking on a button will make the combobox magically appear because that's when the add-in has its first chance to add it. This is not ideal from a user interface standpoint.

Since I wasn't coming up with any solution for the disappearing combobox, I turned to handling the CommandBarComboBox.Change event from the combobox, which is the only event supported. I got the event hooked up, stuffed a few items into the combobox, and started testing. Looking at the code, you'll see that whenever the Change notification triggers, I pop up a message box that indicates that the event occurred and notes the text that is in the combobox. Additionally, I hooked up the IDTCommandTarget.Exec method to also pop up a message box with a different title along with the text in the combobox. At first, everything looked great in that I could select an item with the mouse from the combobox dropdown and the CommandBarComboBox.Change event was getting triggered appropriately.

Now I had to see what would happen with items entered into the combobox edit control. I found that entering a new value and pressing ENTER properly pops the Change notification—the correct action. Also, clicking on any of my buttons caused the execution message box to pop up with the appropriate tests. Things quickly went downhill from there.

As I wanted to mimic the standard toolbar's Find combobox, I entered some new text and immediately clicked on one of my buttons. Annoyingly, the entered text was wiped out and the first item in the combobox list replaced the text! That sure looked like a bug and suggested to me that the Visual Studio .NET add-in model doesn't want you using comboboxes in your CommandBars. No matter what I tried, there was no way I was going to get the combobox to stop changing the text. To make matters worse, changing the text in the combobox and clicking anywhere outside the toolbar also caused the entered text to be replaced and a Change notification for the replaced text to be sent.

Doing a little Spy++ action, I then looked at windows procedures for both the standard toolbar Find combobox and my toolbar's combobox edit controls. While mine comes from MSO.DLL, the Office XP controls DLL, the Find combobox edit control comes from VSBROWSE.DLL, which is, oddly enough, the Web browsing package. As the Find combobox has normal combobox behavior and its windows procedure comes from a different DLL, it's obvious that the default combobox is not quite right.

Although the editable comboboxes weren't working, the straight selection comboboxes worked fine, so I started on a quest to see if I could still salvage my idea of a single toolbar. Because the Office XP Developer documentation indicates that there are actually many different controls you can put on a CommandBar, I tried to use different MsoControlType controls. I learned that other than for buttons or comboboxes, CommandBars.Controls.Add will always throw an exception when you try to add the control. I would have thought that edit controls would have been allowed as well.

In the end, without a huge subclassing effort, add-ins can only safely put buttons and selection comboboxes on the toolbar. I was rather disappointed because I can see lots of uses for editable comboboxes on toolbars. The good news is that at least I suffered through all the CommandBar frustrations so I know exactly what can and can't be placed on them.

F1 Monitoring?

I wanted to work on something else for a while, so I thought it might be interesting if my Google add-in commands could be assigned to keys so you could use the Google search directly from the editor. The ultimate situation would be if I could somehow get the same list that the Dynamic Help window shows when you're editing. That way my add-in would be searching Google for the exact same text that a normal F1 command uses. Additionally, I wouldn't have to do any work to parse out the current keyword from the edit window either.

After poking through the MSDN Help, I ran across the "Adding Custom Links to the Dynamic Help Window" section, which has quite a bit of helpful information. One interesting tidbit mentioned in the section on debugging help topics you add to Dynamic Help was how you can see how Visual Studio .NET was looking at the topics. In the registry key, HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\7.0\Dynamic Help, add a new string value of "Display Debug Output in Detail" and set its value to "YES". After you restart Visual Studio, you'll see some additional details in the Dynamic Help window, as shown in Figure 2.

Figure 2 More Details
Figure 2 More Details

As you move the focus to different areas, you'll see that the values under Active Context change as appropriate. In fact, it's rather interesting to click around the IDE to see what values show up.

After spending some time poking through the Visual Studio .NET automation documentation, I stumbled into the ContextAttributes collection, which contains all of the attributes associated with a global context or window's context in the Dynamic Help window. As I played around with some macros to access the individual ContextAttribute values, such as the one in Figure 3, I figured out that the DTE object contains the keyword list when focus is in a source window.

Figure 3 Reading the ContextAttributes Collection (Close Figure 3)
Public Sub ListDTEAttrs()
    Dim Str As New StringBuilder

    Str.Append("**DTE High Priority ContextAttributes" & vbCrLf)

    Dim CA As ContextAttribute

    DTE.ContextAttributes.Refresh()

    Dim x As ContextAttributes
    x = DTE.ContextAttributes.HighPriorityAttributes

    DTE.ContextAttributes.HighPriorityAttributes.Refresh()
    Str.Append("  HP Count : " & x.Count & vbCrLf)
    For Each CA In x
        Try
            Str.Append("CA Name : " & CA.Name & vbCrLf)
            Try
                Dim o As Object
                For Each o In CA.Values
                    Str.Append("   Value : " & o.ToString() & vbCrLf)
                Next
            Catch ex As System.Exception
                Str.Append("   Unable to get values" & vbCrLf)
            End Try
        Catch ex As System.Exception
            Str.Append("Unable to get value item" & vbCrLf)
        End Try
    Next

    Str.Append("**DTE ContextAttributes" & vbCrLf)

    ' Only do the keywords.
    CA = DTE.ContextAttributes.Item(1)
    Try
        Str.Append("CA Name : " & CA.Name & vbCrLf)
        Try
            Dim o As Object
            For Each o In CA.Values
                Str.Append("   Value : " & o.ToString() & vbCrLf)
            Next
        Catch ex As System.Exception
            Str.Append("   Unable to get values" & vbCrLf)
        End Try
    Catch ex As System.Exception
        Str.Append("Unable to get value item" & vbCrLf)
    End Try

    Dim ow As New OutputPane("CATest")
    ow.Clear()
    ow.WriteLine(Str.ToString())

End Sub

My plan was to write my add-in commands to look at the active window. If it was anything other than a source code window, I'd defer to the normal Help.F1Help command. When I ran the macro similar to the one in Figure 3 on multiple projects in order to get a feeling for the patterns in the ContextAttributes collection, I noticed two patterns. The first was that for various projects the F1 keyword—Active Context as listed in the dynamic help—was not always in the first position in the array, as I would have suspected. In fact, in Visual Basic .NET projects, it seemed to dance all over the place in the array in no particular pattern at all.

After another read of the docs, I ran into the HighPriorityAttributes property of the DTE object that seemed to imply that it should contain the F1 keywords. After adding code to check the property HighPriorityAttributes, which is done in the macro for Figure 3, I saw that for source code windows, the HighPriorityAttributes.Count was always zero. This was a setback in my plan since I wanted to execute the Google search on the same F1 keyword.

The second pattern I noticed in the ContextAttributes collection was that for Visual Basic .NET-based applications, most of the keywords started with "vb." Since the keywords are actually keywords specific to the help engine, that's understandable. While I could strip off the "vb.", I figured it wasn't worth the effort since I couldn't get the F1 keyword.

At this point, my two main ideas for the Google add-in were stymied. Since I couldn't get the Visual Studio .NET CommandBar to properly handle comboboxes and could not ensure I was going to search on the correct F1 keyword, I needed to step back and see if I could find another way of getting Google into the IDE.

The Real Google Add-in

If I couldn't get my Google add-in into a CommandBar, I was positive I could make it a hosted tool window. In case you're not familiar with hosting managed controls into tool windows, I'd encourage you to visit the "Automation Samples for Visual Studio .NET" page where you can download the basic ToolWindow add-in. The key part of the sample is a shim C++ COM DLL that takes all the work out of hosting the common language runtime (CLR) and getting your user control loaded into the tool window. As you'll see in a little bit, I took the shim DLL code, added error handling, assertions (so you'll know what's up when something goes wrong), and some additional useful functionality. Since your add-in is getting the shim loaded and telling the shim what to load, you can reuse my improved shim DLL with no code changes whatsoever.

You should take a look back at the screen shot of the Google add-in in action (see Figure 1). I felt that the Google window should be positioned between the Solution/Class dock and the Properties/Dynamic Help dock for easy access. There's nothing stopping you from dragging the Google window into a tab alongside the Dynamic Help window. If you hide the Google window, the View | Other Windows popup has the Google Search Window as the first item. You could also assign a key combination such as Alt+F1, which is available on the default keyboard, to the GoogleAddIn.ViewGoogle command.

Because I can control the combobox in the Google window, I made it behave exactly like the Standard toolbar's Find combobox. Anything you type in the edit control will be added to the combobox and searches are saved between IDE runs. The buttons across the top of the control perform a particular search, as I outlined at the beginning of the column.

When you click one of the search buttons, the add-in will start a browser window according to your Help settings. If you've elected to start the MSDN Help system externally, a full browser window pops up external to the IDE. If you've selected internal help, a browser window opens inside the IDE. You can override the defaults in the options.

As all good add-ins should, the Google add-in has a properly integrated Options dialog property page, as shown in Figure 4. All of the options in the Settings section are self-explanatory, but I just want to mention that the Default Google site dropdown allows you to specify exactly which international Google version you'd like to use. I found all the international sites at http://www.google.com/language_tools.

Figure 4 Google Add-in Options Page
Figure 4 Google Add-in Options Page

The Group limit string text allows you to specify to which groups you'd like to limit your search. The default is to limit searching to the "comp.*" groups. If you'd like to limit your search to just the Usenet groups hosted by Microsoft, you could substitute "microsoft.public.*" as the text.

Implementation Highlights

With the usage for the Google add-in out of the way, I want to spend a little time on a few interesting implementation highlights. If you're new to add-ins and are wondering how I got the Option property page into the Option dialog, make sure you read Leo Notenboom's MSDN Magazine article, "Custom Add-Ins Help You Maximize the Productivity of Visual Studio .NET".

The first puzzle I wanted to tackle was how to control the built-in browser windows in Visual Studio .NET. Poking through the various window types in the vsWindowKind* constant shows there is a vsWindowKindWebBrowser that is a Window that contains the Web browser. It didn't take a rocket scientist to realize that inside the window was an instance of the SHDocVw.WebBrowser you've come to know and love. Unfortunately, I wasn't at all sure how to get the actual Web browser control so that I could call the SHDocVw.WebBrowser.Navigate method.

The Window class has an Object property, which the documentation says returns an interface or object that can be accessed at run time by name. While that description is as about as vague as can be, I thought since I was doing the Google add-in in C#, I'd try casting the returned object as a SHDocVw.WebBrowser and see what happened. After adding a reference to SHDOCVW.DLL to my project, I set up the code in Figure 5.

Figure 5 Get Browser (Close Figure 5)
// Get the first window that's a browser.
Window Win = m_ApplicationObject.
    Windows.Item(Constants.vsWindowKindWebBrowser);
// Force the window to be visible.
Win.Visible = true ;
// Get the Internet Explorer control inside the browser window.
SHDocVw.WebBrowser x = (SHDocVw.WebBrowser)Win.Object ;

Object Zero = 0 ;
Object EmptyString = "" ;
// Navigate to the URL.
x.Navigate ( Args.UrlString, ref Zero, ref EmptyString,
             ref EmptyString ,ref EmptyString  ) ;

While undocumented, the trick of casting the Window.Object value to a SHDocVw.WebBrowser works great. Even nicer is that calling the Navigate method automatically brings up the Web toolbar and puts the URL into the browsing address combobox.

After verifying that I could control the internal Web browser windows inside the IDE, I turned to creating the actual control itself, which is nothing fancy. However, I wanted to get the control completely debugged and tested before ever using it inside the IDE. While it's possible to develop and debug .NET user controls that reside in add-ins, I didn't want that hassle. The appropriately named GoogleControl.DLL contains the UI for the control.

Because I wanted the control to be self-contained, all the logic for building up the Google search URLs is in GoogleControl.BtnBar_ButtonClick. Figuring out the various parameters was simply a matter of performing a few of each kind of search and looking for the common elements. The only moderately interesting design decision I made was that I didn't want the control itself accessing the IDE to open a window and such, so when a particular search button is pressed, the control fires its GoogleSearch event whose argument parameter contains the fully formed URL. That way, the controlling application can do the dirty work of spawning the browser as appropriate.

Little did I know that choosing to use an event was going to cause some serious head scratching when it came to hosting the Google control. As I mentioned earlier, Microsoft supplied a shim control with one of the add-in samples. The whole reason for this shim is that the Visual Studio .NET IDE is an ActiveX® host and the .NET Framework does not create ActiveX controls. Your add-in will create a ToolWindow telling it to host the shim. Once the ToolWindow is created, you access the shim inside the ToolWindow and tell it to host the CLR and create a specific control. It's quite instructional to walk through the hosting code in the CVSNetToolWinShim::HostUserControl2 and CVSNetToolWinShim::FinalConstruct methods in Figure 6. The full implementation is in VSNETTOOLHOSTSHIM.CPP from the VSNetToolHostShim2project.

Figure 6 CVSNetToolWinShim::HostUserControl2 and CVSNetToolWinShim::FinalConstruct (Close Figure 6)
HRESULT CVSNetToolWinShim::
            HostUserControl2 ( IUnknown * pToolWindow  ,
                               BSTR       Assembly     ,
                               BSTR       Class        ,
                               BSTR       SatelliteDLL ,
                               int        ResourceID   ,
                               IUnknown** ppControlObject )
{
    CComQIPtr<EnvDTE::Window> pWindow ( pToolWindow ) ;

    RECT rc ;
    CComPtr<IWin32Window> pIWin32Window ;

    // Try to create the assembly out of the GAC.
    HRESULT hr = m_pDefaultDomain->CreateInstance ( Assembly,
                                      Class, &m_pObjHandle) ;
    if ( FAILED ( hr ) || ( !m_pObjHandle ) )
    {
        // Assume the complete path is passed in and try and
        // to create that one.
        hr = m_pDefaultDomain->CreateInstanceFrom ( Assembly,
                                    Class, &m_pObjHandle  ) ;
        if ( FAILED ( hr ) || ( !m_pObjHandle ) )
        {
            // At least let the caller know something is bad in
            // a debug build.
            ATLASSERT ( FALSE ) ;
            hr = E_FAIL ;
            Reset ( ) ;
            return ( hr ) ;
        }
    }

    hr = m_pObjHandle->Unwrap ( &m_varUnwrappedObject ) ;
    if ( ( m_varUnwrappedObject.vt != VT_DISPATCH ) &&
           ( m_varUnwrappedObject.vt != VT_UNKNOWN ) ||
            ( !m_varUnwrappedObject.punkVal  ) )
    {
        ATLASSERT ( FALSE ) ;
        hr = E_FAIL ;
        Reset ( ) ;
        return ( hr ) ;
    }
    hr = m_varUnwrappedObject.pdispVal->QueryInterface (
            IID_IUnknown , (LPVOID*)ppControlObject ) ;
ATLASSERT ( SUCCEEDED ( hr ) ) ;

    hr = m_varUnwrappedObject.pdispVal->QueryInterface (
            IID_IWin32Window, (LPVOID*)&pIWin32Window ) ;
        ATLASSERT ( SUCCEEDED ( hr ) ) ;
    if ( FAILED ( hr ) )
    {
        Reset ( ) ;
        return ( hr ) ;
    }

    long hWndTemp ;
    hr = pIWin32Window->get_Handle ( &hWndTemp ) ;
// 'type cast' : conversion from 'long' to 'HWND' of greater size
#pragma warning ( disable : 4312 )
    m_hWndForm = (HWND)hWndTemp ;
#pragma warning ( default : 4312 )
    if ( FAILED ( hr ) || ( !m_hWndForm ) )
    {
        ATLASSERT ( FALSE ) ;
        Reset ( ) ;
        return ( hr ) ;
    }
    if ( pWindow.p )
    {
        USES_CONVERSION ;
        CComVariant varPic ;
        CComPtr<IPictureDisp> pPictureDisp ;
        PICTDESC pd ;
        pd.cbSizeofstruct = sizeof ( PICTDESC ) ;
        pd.picType = PICTYPE_BITMAP ;
        HMODULE hModule = LoadLibrary ( W2T ( SatelliteDLL ) );
        pd.bmp.hbitmap = LoadResourceBitmap ( hModule ,
            MAKEINTRESOURCE ( ResourceID ) ,
            &pd.bmp.hpal ) ;
        OleCreatePictureIndirect ( &pd,  IID_IPictureDisp,
            FALSE, (LPVOID*)&pPictureDisp ) ;
        pPictureDisp->QueryInterface ( IID_IUnknown ,
                                   (LPVOID*)&varPic.punkVal ) ;
        varPic.vt = VT_UNKNOWN ;
        pWindow->SetTabPicture ( varPic ) ;
        FreeLibrary ( hModule ) ;
    }

    ::SetParent((HWND)m_hWndForm, m_hWnd);
    ::GetWindowRect(m_hWnd, &rc);
    ::MoveWindow((HWND)m_hWndForm, 0, 0, rc.right-rc.left,
         rc.bottom-rc.top, TRUE);
    ::ShowWindow((HWND)m_hWndForm, SW_SHOW);
    return ( hr ) ;
}

HRESULT CVSNetToolWinShim::FinalConstruct()
{
    CComPtr<IUnknown> pAppDomainPunk;
    HRESULT hr = CorBindToRuntimeEx(NULL, NULL,
        STARTUP_LOADER_OPTIMIZATION_SINGLE_DOMAIN |
        STARTUP_CONCURRENT_GC, __uuidof(CorRuntimeHost),
        __uuidof(ICorRuntimeHost), (LPVOID*)&m_pHost);
    if(FAILED(hr))
    {
        ATLASSERT ( FALSE ) ;
        return hr;
    }
    hr = m_pHost->Start();
    if(FAILED(hr))
    {
        ATLASSERT ( FALSE ) ;
        return hr;
    }
    hr = m_pHost->GetDefaultDomain(&pAppDomainPunk);
    if(FAILED(hr) || !pAppDomainPunk)
    {
        ATLASSERT ( FALSE ) ;
        return hr;
    }
    hr = pAppDomainPunk->QueryInterface(__uuidof(mscorlib::_AppDomain),
             (LPVOID*)&m_pDefaultDomain);
    if(FAILED(hr) || !m_pDefaultDomain)
    {
        ATLASSERT ( FALSE ) ;
        return hr;
    }

    return hr;
}

The Microsoft control, though it was in need of some serious assertions, did the job pretty well. However, with GoogleControl triggering an event when a button was pressed, I realized the Microsoft code wasn't going to help. While it was great at creating the control and letting it run, there was not a way to access the actual hosted CLR control inside the VSNetHostShim ActiveX control. Without access to the control, I couldn't receive the event notification nor call methods on the control itself.

After a few moments of panic, thinking that I was going to have to rethink the Google control implementation, I remembered the System.Runtime.Remoting.ObjectHandle class that allows you to pass an object between application domains while controlling the instance. If I could somehow get the IObjectHandle interface from the VSNetToolHostShim2 returned to my C# code, I'd be all set. The good news is that calling AppDomain.CreateInstance, which is the method that has to be called in CVSNetToolWinShim::HostControl2 to create a .NET object, returns the IObjectHandle interface. The result is that the Microsoft hosting shim already had the IObjectHandle interface inside its implementation, but the necessary methods weren't there to expose it.

All I needed to do was add a new method to the VSNetToolHostShim2 ActiveX control that just did a QueryInterface on the ObjectHandle class member data for the IID_IObjectHandle. You can see the one line of code in the CVSNetToolWinShim::GetObjectHandle method.

Once your .NET written add-in has the ObjectHandle, all it needs to do is call the Unwrap method and cast the return value to the appropriate type and you've got a reference to the hosted control. While I really wanted my CVSNetToolWinShim::GetObjectHandle method to return the ObjectHandle directly, which I would have done by importing MSCOREE.IDL in VSNetToolHostShim2.IDL, the MSCOREE.IDL file contains references to files that don't exist so it does not compile if imported or if MSCOREE.H is included in the IDL file. Consequently, I return IUnknown and you are required to cast the returned value to an ObjectHandle in order to call the Unwrap method.

If the last few paragraphs were a little confusing to you, all you need to do is read Figure 7, which shows the method in the Google add-in that creates the ToolWindow and gets the Google Control hosted properly. For your own tool windows, you can use the code to ensure your hosted controls are properly set up.

Figure 7 Hosting Controls in a ToolWindow (Close Figure 7)
/// <summary>
/// Takes care of ensuring the window is created.
/// </summary>
private void ConstructWindow ( )
{
    try
    {
        // This'll contain the VSNetToolHostShim2 on output.
        Object RefObj = null ;

        // Create the main tool window by loading host shim.
        m_TheToolWindow = m_ApplicationObject.Windows.
            CreateToolWindow ( m_AddInInstance ,
                        "VSNetToolHostShim2.VSNetToolWinShim2",
                        ResConst.ToolWinCaption,
                        ResConst.ToolWinGUID, ref RefObj);

        // Make the window visible.  You must do this before
        // calling the HostUserControl method or things won't
        // get hooked up right.
        m_TheToolWindow.Visible = true ;

        // Get the shim.
        VSNetToolHostShimLib2.IVSNetToolWinShim2 ShimObj =
         (VSNetToolHostShimLib2.VSNetToolWinShim2Class)RefObj ;

        // Get this assembly so I can pass the location to the shim.
        System.Reflection.Assembly CurrAsm =
          System.Reflection.Assembly.GetExecutingAssembly ( ) ;

        // Figure out the path to the the GoogleControl DLL.
        StringBuilder StrCtlDll = new StringBuilder ( ) ;
        String StrTemp = CurrAsm.Location.ToLower ( ) ;
        int iPos =StrTemp.IndexOf(ResConst.AddInName.ToLower
            ( ) ) ;

        StrCtlDll.Append ( CurrAsm.Location.Substring ( 0 , iPos ));
        StrCtlDll.Append ( ResConst.GoogleCtlDLL ) ;

        // Load the managed control into the ActiveX control
        // and have it load the tab bitmap.
        ShimObj.HostUserControl2 ( m_TheToolWindow,
            StrCtlDll.ToString ( ), ResConst.GoogleCtlName,
            m_AddInInstance.SatelliteDllPath,
            ResConst.TabBitmapID);

        // Ask the VSNetShim control for the IUnknown of the
        // hosted control.  From the object handle, I can use
        // casting to get to the actual hosted GoogleControl.
        ObjectHandle oHnd = (ObjectHandle) ShimObj.GetObjectHandle ( ) ;
        GoogleCtl = (GoogleControl)oHnd.Unwrap ( ) ;

        // Set up to handle the events from the control.
        GoogleCtl.GoogleSearch += new GoogleControl.
            GoogleSearchEventHandler(GoogleCtl_GoogleSearch);

        // Initialize the user options.
        InitializeUserOptions (
            m_ApplicationObject.RegistryRoot, GoogleCtl );
    }
    catch ( System.Exception Ex )
    {
      MessageBox.Show ( Ex.Message + "\r\n" + Ex.StackTrace ) ;
        throw Ex ;
    }
}

The good news about all this messing around with the VSNetToolHostShim2 control is that you can use the control exactly as-is for any control you want to host inside Visual Studio .NET. Armed with VSNetToolHostShim2, you can now handle controls that fire events. As I did with the Google control, you can isolate functionality better and just maybe achieve the dream of truly reusable controls that can be plugged into multiple apps.

The final task for the Google add-in was to have it integrate seamlessly into the environment, I wanted to add a Google Search Window option to the View | Other Windows popup. According to my understanding of the documentation, all I had to do was get the "&View" CommandBar, enumerate through to the Other Windows popup menu, and poke my command on the end of the popup menu. I didn't think this would be much of a problem, but I went at it with a bad assumption.

My first step was to write the following macro and enumerate through the items on the "&View" menu and spit out their types:

Sub PrintMenu()
    Dim cw As New CmdWindow

    cw.Clear()
    Dim Cmds As _CommandBars
    Cmds = CType(DTE.CommandBars, _CommandBars)
    Dim FileMenu As CommandBar = Cmds("&View")
    For Each Ctrl As CommandBarControl In FileMenu.Controls
        cw.WriteLine(Ctrl.Caption + vbTab + Ctrl.Type.ToString())
    Next

End Sub
By the way, the CmdWindow is a simple class to encapsulate writing to the Command window and Output panes. The idea for my macro was to enumerate the controls, and output their types so I could see exactly what values I needed.

I was surprised to see that the Other Windows menu was not listed in the output and there were no types other than msoControlButton with each menu item. With a little poking around, I found I was able to enumerate the Other Windows menu on its own so it was easy enough to add my menu option in the correct place. My assumption was that I would have been able to get msoControlPopup types directly from the CommandBar.

Wrap Up

While it was a wild journey, I eventually ended up with an add-in that I use every day to slay development problems that creep up. Google is an amazing search engine and now that I have it in my IDE, I wonder how I lived without it.

The Tips

In my last column, there was only enough room to publish one of the two tips. To make up for the missing tip, this month you have a whopping three! If you have any tips to add to the collection, don't hesitate to send them to me at john@wintellect.com.

Tip 56 The Microsoft Visual Studio team has put together a collection of power toy add-ins for Visual Studio that you might find useful. You can download them at PowerToys for Visual Studio .NET 2003. The Visual Basic Commenter, which lets you put XML comments in Visual Basic .NET, and the Custom Help Builder, which makes it easy to add your own class documentation to the Help System, are both outstanding additions to the IDE.

Tip 57 To hasten the startup of the Visual Studio .NET IDE, get rid of the Start Page. Because the Start Page requires all Web-browsing components to be loaded, you can chop off a considerable amount of startup time by skipping it. To turn off the Start Page, select Options from the Tools menu to bring up the Options dialog. In the Environment/General property page, select "Show empty environment" in the "at Startup" combobox.

Tip 58 Another performance tip is to turn off tracking the active item in the Solution Explorer window. This will keep the Solution Explorer selection from bouncing all over the place when working on different files in a project. From the Tools menu, select Options. In the Options dialog, select the Environment/Projects and Solutions property page and uncheck Track Active Item in Solution Explorer.

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in programming for the .NET Framework and Windows. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at http://www.wintellect.com.


C++ Q&amp;A: Menu Tips in an MFC App

C++ Q&A
Menu Tips in an MFC App
Paul DiLascia

Code download available at: CQA0311.exe (193KB)

Q I'm working on a project that has a long history. When we started, we implemented our own popup menus for various reasons. When tooltips came along, we added this functionality to our menus so that tooltips appear when the user holds the mouse over a menu item. This is important to our users because it explains why a menu item is disabled. As our users became more familiar with the world of Windows® they wanted to have menus that looked like the standard menus. Currently we use CMenu, but we have lost our excellent menu tooltips. How can we implement these in MFC?

Joakim Fagerli

A What a great idea! Figure 1 is the picture worth a thousand words. It shows a little program I wrote, MenuTips, that implements menu tips for any MFC app. Menu tips are especially cool because they eliminate one reason for having a status bar. Even without the status bar, you can still see what each command does. More important, the tip appears next to the menu item where it's more obvious. With today's giant screens, many users don't even realize menu prompts appear in the status bar—it's too far away to notice.

Figure 1 MenuTips
Figure 1 MenuTips

I implemented the menu tips in a class called CMenuTipManager. To implement menu tips in your app, just add a CMenuTipManager to your main window class and call Install when your frame is created:

// in CMainFrame::OnCreate(...)
m_menuTipManager.Install(this);

That's all there is. Now whenever the user highlights a menu item for longer than a second or so, the menu tip manager displays a tip with the command prompt, as shown in Figure 1. CMenuTipManager gets the prompt from your program's string table—the same place MFC looks for status line prompts.

CMenuTipManager uses my world-famous-by-now subclassing class, CSubclassWnd, to trap WM_MENUSELECT messages sent to the main window. As the user highlights different menu items in the main menu, system menu, or any popup menu, Windows sends the owning window a WM_MENUSELECT for each new menu item highlighted. This is your chance in life to provide feedback or do whatever else you want to do. MFC's CFrameWnd::OnMenuSelect handles WM_MENUSELECT to display the command prompt in the status bar. CMenuTipManager traps the same message to display menu tips. Figure 2 shows the relevant code.

Figure 2 MenuTipper (Close Figure 2)

MenuTipper.h
///////////////////////////////////////////////////////////////
// MSDN Magazine — November 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#pragma once
#include "PupText.h"
#include "subclass.h"

//////////////////
// Implement menu tips for any MFC main window. To use:
//
// - instantiate CMenuTipManager in your CMainFrm
// - call Install
// - implement prompt strings the normal way: as resource
//   strings w/ID=command ID.
//
class CMenuTipManager : public CSubclassWnd {
protected:
   CPopupText m_wndTip;    // home-grown "tooltip"
   BOOL m_bMouseSelect;    // whether menu invoked by mouse
   BOOL m_bSticky;         // after first tip appears, show
                           // rest immediately

public:
   int m_iDelay;           // tooltip delay: you can change

   CMenuTipManager() : m_iDelay(1000), m_bSticky(FALSE) { }
   ~CMenuTipManager() { }

   // call this to install tips
   void Install(CWnd* pWnd) { HookWindow(pWnd); }

   // Useful helpers to get window/rect of current active menu
   static CWnd* GetRunningMenuWnd();
   static void  GetRunningMenuRect(CRect& rcMenu);
   CRect GetMenuTipRect(HMENU hmenu, UINT nID);

   // Useful helper to get the prompt string for a command ID.
   // Like CFrameWnd::GetMessageString, but you don't need a
   // frame wnd.
   static CString GetResCommandPrompt(UINT nID);

   // Get the prompt for given command ID
   virtual CString OnGetCommandPrompt(UINT nID)
   {
      return GetResCommandPrompt(nID);
   }

   // hook fn to trap main window's messages
   virtual LRESULT WindowProc(UINT msg, WPARAM wp, LPARAM lp);

   // Call these handlers from your main window
   void OnMenuSelect(UINT nItemID, UINT nFlags, HMENU hMenu);
   void OnEnterIdle(UINT nWhy, HWND hwndWho);
};
MenuTipper.cpp
///////////////////////////////////////////////////////////////
// MSDN Magazine — November 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#include "stdafx.h"
#include "MenuTipper.h"
#include <afxpriv.h> // for AfxLoadString

//////////////////
// This is more or less copied from
// CFrameWnd::GetMessageString, but I want a static function
// that doesn't require a frame window.
//
CString CMenuTipManager::GetResCommandPrompt(UINT nID)
{
   // load appropriate string
   CString s;
   if (s.LoadString(nID)) {
      LPTSTR lpsz = s.GetBuffer(255);
      // first newline terminates prompt
      lpsz = _tcschr(lpsz, '\n');
      if (lpsz != NULL)
         *lpsz = '\0';
      s.ReleaseBuffer();
   }
   return s;
}

//////////////////
// Override CSubclassWnd::WindowProc to hook messages on behalf
// of main window.
//
LRESULT CMenuTipManager::WindowProc(UINT msg, WPARAM wp, LPARAM lp)
{
   if (msg==WM_MENUSELECT) {
      OnMenuSelect(LOWORD(wp), HIWORD(wp), (HMENU)lp);
   } else if (msg==WM_ENTERIDLE) {
      OnEnterIdle(wp, (HWND)lp);
   }
   return CSubclassWnd::WindowProc(msg, wp, lp);
}

//////////////////
// Got WM_MENUSELECT: show tip.
//
void CMenuTipManager::OnMenuSelect(UINT nItemID, UINT nFlags,
    HMENU hMenu)
{
   if (!m_wndTip.m_hWnd) {
      m_wndTip.Create(CPoint(0,0), CWnd::FromHandle(m_hWnd));
      m_wndTip.m_szMargins = CSize(4,0);
   }

   if ((nFlags & 0xFFFF)==0xFFFF) {
      m_wndTip.Cancel();   // cancel/hide tip
      m_bMouseSelect = FALSE;
      m_bSticky = FALSE;

   } else if (nFlags & MF_POPUP) {
      m_wndTip.Cancel();   // new popup: cancel
      m_bSticky = FALSE;

   } else if (nFlags & MF_SEPARATOR) {
      // separator: hide tip but remember sticky state
      m_bSticky = m_wndTip.IsWindowVisible();
      m_wndTip.Cancel();

   } else if (nItemID && hMenu) {
      // if tips already displayed, keep displayed
      m_bSticky = m_wndTip.IsWindowVisible() || m_bSticky;

      // remember if mouse used to invoke menu
      m_bMouseSelect = (nFlags & MF_MOUSESELECT)!=0;

      // get prompt and display tip (with or without timeout)
      CString prompt = OnGetCommandPrompt(nItemID);
      if (prompt.IsEmpty())
         m_wndTip.Cancel(); // no prompt: cancel tip

      else {
         CRect rc = GetMenuTipRect(hMenu, nItemID);
         m_wndTip.SetWindowPos(&CWnd::wndTopMost, rc.left,
            rc.top, rc.Width(), rc.Height(), SWP_NOACTIVATE);
         m_wndTip.SetWindowText(prompt);
         m_wndTip.ShowDelayed(m_bSticky ? 0 : m_iDelay);
      }
   }
}

//////////////////
// Calculate position of tip: next to menu item.
//
CRect CMenuTipManager::GetMenuTipRect(HMENU hmenu, UINT nID)
{
   CWnd* pWndMenu = GetRunningMenuWnd(); //CWnd::WindowFromPoint(pt);
   ASSERT(pWndMenu);
   CRect rcMenu;
   pWndMenu->GetWindowRect(rcMenu); // whole menu rect

   // add heights of menu items until i reach nID
   int count = ::GetMenuItemCount(hmenu);
   int cy = rcMenu.top + GetSystemMetrics(SM_CYEDGE)+1;
   for (int i=0; i<count; i++) {
      CRect rc;
      ::GetMenuItemRect(m_hWnd, hmenu, i, &rc);
      if (::GetMenuItemID(hmenu,i)==nID) {
         // found menu item: adjust rectangle to right and down
         rc += CPoint(rcMenu.right - rc.left, cy - rc.top);
         return rc;
      }
      cy += rc.Height(); // add height
   }
   return CRect(0,0,0,0);
}

//////////////////
// Note that windows are enumerated in top-down Z-order, so the
// menu window should always be the first one found.
//
static BOOL CALLBACK MyEnumProc(HWND hwnd, LPARAM lParam)
{
   char buf[16];
   GetClassName(hwnd,buf,sizeof(buf));
   if (strcmp(buf,"#32768")==0) { // special classname for
                                  // menus
      *((HWND*)lParam) = hwnd;    // found it
      return FALSE;
   }
   return TRUE;
}

//////////////////
// Get running menu window.
//
CWnd* CMenuTipManager::GetRunningMenuWnd()
{
   HWND hwnd = NULL;
   EnumWindows(MyEnumProc,(LPARAM)&hwnd);
   return CWnd::FromHandle(hwnd);
}

//////////////////
// Need to handle WM_ENTERIDLE to cancel the tip if the user
// moved the mouse off the popup menu. For main menus, Windows
// will send a WM_MENUSELECT message for the parent menu when
// this happens, but for context menus there's no other way to
// know the user moved the mouse off the menu.
//
void CMenuTipManager::OnEnterIdle(WPARAM nWhy, HWND hwndWho)
{
   if (m_bMouseSelect && nWhy==MSGF_MENU) {
      CPoint pt;
      GetCursorPos(&pt);
      if (hwndWho != ::WindowFromPoint(pt)) {
         m_wndTip.Cancel();
      }
   }
}

CMenuTipManager is mostly straightforward, but as usual in Windows there are several sticky points that require attention. First are the tooltips themselves: has anyone ever figured out how to use the standard Windows tooltips? Instead, I used the CPopupText class from my September 2000 and June 2001 columns. CPopupText is so simple even a Visual Basic® expert could implement it—if he knew how to type semicolons. All you have to do is instantiate a CPopupText object, call Create and SetWindowText, then CPopupText::ShowDelayed to display the tip in a specified number of milliseconds. CPopupText::Cancel kills the tip. The only hard part is making CPopupText look the same as a normal tooltip. For this, CPopupText uses the menu font and GetSysColor(COLOR_INFOBK), which obtains the (usually pale yellow) system tooltip color. As always, you can download the source code from the link at the top of this article.

The trickiest part of CMenuTipManager is figuring out where to place the tip so it aligns perfectly next to the menu, just to the right of the highlighted item, as shown in Figure 1. The basic idea is to find the location of the menu and do a bunch of arithmetic to add up the heights of all the menu items until you reach the selected menu item. But how do you get the menu's location? This turns out to be a nontrivial problem. As you might guess, the menu is itself a window—but there's no API function to get its handle, so what to do? As I've said many times before, in Windows there's always a way. You simply must not be deterred.

CMenuTipManager has a static helper function, CMenuTipManager::GetRunningMenuWnd, that returns the current running menu window. This function is so useful I decided to make it public. But how does GetRunningMenuWnd work? You might think of calling WindowFromPoint to get the window under the mouse. This would work except for a slight problem: the user can invoke the menu through the keyboard, not the mouse, in which case the cursor could be anywhere, not necessarily over the menu. So instead CMenuTipManager calls ::EnumWindows to enumerate all top-level windows, looking for one with the special class name "#32768" that Windows uses for menus:

static BOOL MyEnumProc(HWND hwnd, LPARAM lParam)
{
   char buf[16];
   GetClassName(hwnd, buf, sizeof(buf));
   if (strcmp(buf,"#32768")==0) { // menu window
      // save hwnd
      return FALSE; // no need to look further
   }
   return TRUE;       // keep looking
}

Since there should be only one menu displayed, the first one MyEnumProc finds is good. Even if for some bizarre reason there were two menus floating around, EnumWindows enumerates the windows in top-down Z-order, so the first one found must be the one that's active. Pretty clever, eh? Once you have the menu window (HWND or CWnd), it's only a matter of pixel math to calculate the exact screen position for the popup tip. CMenuTipManager::OnMenuSelect in Figure 2 shows the details.

What about the tip text? CMenuTipManager provides another helper function, CMenuTipManager::GetMessageString, to get the menu prompt associated with each command. I copied this more or less verbatim from CFrameWnd::GetMessageString. Why duplicate this function? So you can call it without a CFrameWnd. CFrameWnd::GetMessageString should've been static, but whichever friendly Redmondtonian wrote this function apparently didn't notice that CFrameWnd isn't required. And why should you need a frame window to load a resource string? To set the universe straight, I made my version static.

Another bugaboo I discovered when I first implemented menu tips has to do with canceling tips when the user moves the mouse off a menu. For main window menus, if the user moves the mouse away from the menu, Windows sends a WM_MENUSELECT with the parent menu as the handle and the MF_POPUP flag, so it's possible to tell this has happened and hide the tip. For context menus, however, you're out of luck. When the user moves the mouse off a context menu, there's no WM_MENUSELECT message to notify you. Sigh.

No problem; in Windows there's always a way. In this case it turns out that Windows sends a different message: WM_ENTERIDLE. Actually, Windows sends WM_ENTERIDLE any time the program is waiting for input and a dialog or menu is displayed. Windows is even polite enough to pass along the HWND of the dialog or menu—amazing! So all you have to do is compare this HWND to the HWND of the window under the mouse when you get WM_ENTERIDLE. If the HWND of the window under the mouse is the same as the WM_ENTERIDLE HWND, then the mouse is still over the menu; if the HWND of the window under the mouse is some other HWND, the user moved the mouse off the context menu and it's time to cancel the tip. CMenuTipManager::OnEnterIdle in Figure 2 is the function that does it.

Finally, CMenuTipManager uses an internal flag, m_bSticky, to control whether tips are delayed or immediate. When the user first invokes a menu item, you want to wait before showing the tip. But once the first tip appears, you don't want to make the user wait again every time he selects a new menu item. So once a tip is displayed, CMenuTipManager sets m_bSticky = TRUE so it knows to display subsequent tips immediately. Canceling the menu or invoking a command resets m_bSticky to FALSE.

CMenuTipManager displays the same menu prompt whether the menu item is enabled or disabled. To implement the feature in your program that explains why a menu item is disabled, you'll have to modify CMenuTipManager and MFC's prompt mechanism. MFC expects command strings to have the format "Long command prompt explanation\nshort prompt." That is, MFC expects a newline character (\n) separating the long form of the prompt from the short one. MFC displays the long prompt in the status bar and the short prompt in toolbars. You could extend this convention to add a third string used to explain why a command is disabled. You'll have to modify CMenuTipManager::GetResCommandPrompt to take a second argument: which command prompt to get (long/short/disabled), and modify OnGetCommandPrompt to get the disabled prompt if the menu item has the MF_DISABLED flag. I leave this as an exercise for the reader.

Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


Resource File: Threat Model Your Security Risks

Resource File
Threat Model Your Security Risks

In the .NET world of loosely coupled distributed components, sharing sensitive data across networks means increased exposure to attackers hungry for your data. You need to create a tight security model to benefit from the .NET vision of fully functional, distributed computing. Failure to achieve this can lead to disaster. So how do you make sure your application is as secure as it needs to be? Well, you should begin with threat modeling, an iterative approach to assessing the vulnerabilities in your application to find those that are the most dangerous because they expose the most sensitive data. From there, you create a prioritized set of countermeasures to manage your risk.

The best place to learn about threat modeling and its role in the overall architecture and design process is "Improving Web Application Security: Threats and Countermeasures," located on the Microsoft patterns & practices Web site.

There's a six-step approach to creating a threat model. The first is obvious—identify your assets. Every site has some confidential data, from salaries to Social Security numbers. You won't know what hackers want until you've identified the sensitive information on your site.

The next step is to develop an architecture overview. You need to be explicit about what the application is designed to do (use cases), how you plan to architect and design the application to achieve that functionality, and what technologies are required to implement the design. This helps you identify common technology-specific threats and implement solutions to overcome them.

The use cases and architectural model will help you decompose your application, which is the third step. The more you know about your application, the easier it is to uncover threats. This step involves breaking down your application to create a security profile. Accepting the axiom that all data is evil, you should perform validation against all data sent across subsystems. An exhaustive examination of your trust boundaries, data flow, and entry points will ensure that all handoffs are done in a secure manner.

In the fourth step, you identify threats that might affect your system and compromise your assets. To take a methodical approach, you should work your way up the stack: from network threats, through host threats, and then application threats. To assess network threats, investigate how the data passes through router configurations, firewalls, and switches. This is the network-level defense in-depth strategy and you need to determine what it takes to get past each gatekeeper. When investigating the host, examine common configuration categories applicable to all server resources (patches, files, directories, and so on). Finally, refocus on the application. The best way to go deep with your app is to use attack trees, which define a potential attack on your system in a structured and hierarchical manner. "Improving Web Application Security" goes into greater detail on how to create and use attack trees.

In the fifth step, you document each threat—the description of the threat, the target of the attack, the risk of the attack, the techniques likely to be used to perpetrate the attack, and a strategy to manage your risk. For example, when dealing with SQL injection, the target is your database and the technique is that the attacker types a command into a textbox that is automatically added into a T-SQL command without client-side validation. To counter this threat, use regular expressions to validate the user name, and use a parameterized query to access the database.

So far you've been in data collection mode, determining every possible hole in your app. Step six is to prioritize. Addressing every conceivable threat is not practical, but you need to do a risk assessment to prioritize and address the most important ones. This requires some simple math: the probability of occurrence multiplied by the potential damage that would occur. The good news is that this is a simple way to prioritize; the bad news is that this is a simple way to prioritize. It is a bit subjective given the meticulous approach to identifying the risks. There are more thorough approaches for risk assessment.

Microsoft uses the DREAD model to assess risk with a greater granularity than the simple math already described. DREAD is an acronym that defines five key attributes used to measure each vulnerability: Damage potential, Reproducibility, Exploitability, Affected users, and Discoverability. "Improving Web Application Security" details how Microsoft uses DREAD to prioritize and mitigate risk.

These six steps complete the process. You are now ready to properly implement your security strategy. Threat modeling is an indispensable approach to avoiding the disasters that could make for front-page news.