 MSDN Magazine June 2005

MSDN Magazine June 2005

Hackers Beware: Keep Bad Guys at Bay with the Advanced Security Features in SQL Server 2005

Get a peek at the new security features in SQL Server 2005 from a developer's point of view. While there are lots of admin enhancements, there are also plenty of dev-specific security improvements you can take advantage of, such as endpoint authentication and support for the security context of managed code that executes on the server. Here Don Kiely elucidates.

Don Kiely Know Thy Code: Simplify Data Layer Unit Testing using Enterprise Services

If you want to employ unit testing and test-driven development techniques in your database application development process, you'll have different factors to consider than you do when you're not involving a database. For example, you have to maintain a consistent state within the database and be able to roll back transactions when necessary. This article shows you how to get the best of unit testing for your database apps in a safe, usable manner.

Roy Osherove Say What?: Make Sense of Your Web Feedback using SQL Server 2005

By now, you're certainly collecting a variety of data from your Web site. But once you collect it, what do you do with it. For it to be meaningful, this data must be categorized and analyzed. The SQL Server 2005 data mining functionalities can help you get the most out of those reams of data you collect, as Peter Pyungchul Kim explains.

Peter Pyungchul Kim Mine Your Business: AMO Lets You Dig Deeper into Your Data from Your Own Applications

Analysis Management Objects (AMO) for SQL Server 2005 facilitates client-side custom programming as Decision Support Objects (DSO), the object model in SQL Server 2000 Analysis Services. With AMO, a whole range of data mining questions can be answered at the operational level. This means that sales and marketing departments get answers more quickly and thus can make informed decisions.

Liu Tang and Paul Bradley Make It Snappy: Juice Up Your App with the Power of Hyper-Threading

In this article, the author explores the hyper-threading technology found on newer Intel Pentium 4 processors and demonstrates how adding parallelism to your code can improve performance on hyper-threaded machines. He covers advanced optimizations for hyper-threading and shows a number of useful patterns. Code samples are in C#, but you can apply the same principles in other languages as they apply to both managed and unmanaged applications.

Yaniv Pessach Get A Raise: Discover a Series of Fortunate Event Handlers in Visual Basic

You may use them every day, but how much do you really know about events? What's going on when you add an event handler to a class? In this article, Ken Getz demonstrates various ways to interact with events and event handlers, and shows how they can solve common problems. Along with the article, you can download two sample applications (one for Visual Basic .NET 2002 and 2003 and one for Visual Basic 2005) and follow along.

Ken Getz Editor's Note: Reader E-mail

Here at MSDN Magazine, we get a lot of mail from readers. As you can imagine, much of it involves fulsome praise for our content, our eye-catching covers, or the handsome photo that now graces this page.

New Stuff: Resources for Your Developer Toolbox

A few months ago I had a large SQL Server™ asset management database into which I had to import data from text files in daily batches. I needed to tweak the tables, add some queries, and generate new reports.

Marnie Hutcheson Web Q&A: Locking Pop-Up Blocker, Mixed Authentication, and More

Edited by Nancy Michell Cutting Edge: Data Repeater Controls in ASP.NET

The ASP. NET Repeater is a basic container control that allows you to create custom lists from any data available to the page. It's a handy control, especially since most ASP. NET pages that display data need to repeat the same kinds of data over and over.

Dino Esposito Advanced Basics: Scaling Up: The Very Busy Background Compiler

One of the features that distinguishes Visual Basic® from the other languages in Visual Studio® is its use of a background compiler (BC). The BC runs from the moment you start up a Visual Basic project until the time you close the last one down.

Matthew Gertz Data Points: XML Features in SQL Server 2000

SQL Server™ 2000 includes several XML features that let you transform relational rowsets into hierarchical XML documents, read XML documents, and bulk load data via XML. For example, you can pass an XML document to a stored procedure, join the XML to some tables and return a rowset, or even modify data in the database.

John Papa Service Station: Techniques for Contract-First Development

In my May 2005 column, I discussed contract-first development and appropriate times to use it (see Service Station: Contract-First Service Development). In this second part, I'm going to focus on some techniques for contract-first development within the ASMX framework.

Aaron Skonnard Extreme ASP.NET: Tools of the Trade: Application Center Test

When you sit down to write an ASP. NET application, how much time do you spend thinking about performance? It's unfortunate, but for most developers performance is an afterthought. Performance planning and design really need to be front and center.

Rob Howard Test Run: The Analytic Hierarchy Process

Most software testing takes place at a relatively low level. Testing an application's individual methods for functional correctness is one example. However, some important testing must take place at a very high level—for example, determining if a current build is significantly better overall than a previous build.

James McCaffrey Security Briefs: Customizing GINA, Part 2

GINA, the Graphical Identification and Authentication component, is a part of WinLogon that you can customize or replace. Last month I introduced GINA customization; this month, I'm going to drill down to implement each of the GINA entry points.

Keith Brown Wicked Code: Power Programming Tips for ASP.NET 2.0

In the February 2005 issue, I introduced five lesser-known features of ASP. NET 2. 0 that have the potential to make a significant impact on the security, performance, and robustness of your code (see Wicked Code: Five Undiscovered Features on ASP.

Jeff Prosise Pure C++: CLR Generics Versus C++ Templates

Visual Studio® 2005 brings the type parameter model of generic programming to the Microsoft® . NET Framework. C++/CLI supports two type parameter mechanisms—common language runtime (CLR) generics and C++ templates.

Stanley B. Lippman C++ at Work: Counting MDI Children, Browsing for Folders

Paul DiLascia { End Bracket }: It's Not Just Fonts

Creating a typeface that's truly readable on a screen is much more complex than people realize. The average core font in Windows®, for instance, has about 25,000 lines of executable code as well as the outline and other data.

Bill Hill


Hackers Beware: Keep Bad Guys at Bay with the Advanced Security Features in SQL Server 2005

Hackers Beware

Keep Bad Guys at Bay with the Advanced Security Features in SQL Server 2005

Don Kiely
This article is based on a prerelease version of SQL Server 2005. All information contained herein is subject to change.
This article discusses:
  • Granular permissions and the principle of least privilege
  • Controling metadata visibility
  • Separation of users and schemas
  • Execution context and encryption in the database
This article uses the following technologies:
SQL Server 2005, Security
Code download available at: SQLServerSecurity.exe (115KB)
Contents
Permissions
Catalog Views and Metadata Visibility
User/Schema Separation
Execution Context
Encryption
Conclusion

I
t's an increasingly hostile world for your databases and client applications. Every day, attackers develop clever new attacks to compromise your valuable data. Fortunately, SQL Server™ 2005 offers strong new security features that directly address principles such as defense in depth and least privilege. As explained in SQL Server Books Online, Microsoft has implemented a number of security initiatives, which include reducing the attack surface area and making it easier to securely deploy SQL Server and databases, while providing better security tools and docs about maintaining high security in a changing security landscape.

In this article I'll explore the most interesting security enhancements in SQL Server 2005 from a developer's viewpoint. I covered admin security features in the Spring 2005 issue of TechNet Magazine. But there are plenty of dev-specific security enhancements I can explore, such as endpoint authentication and support for the security context of managed code that executes on the server. I'll focus only on the core relational database engine. Supporting systems such as Reporting Services, Analysis Services, and so on have their own security infrastructures that deserve whole articles to cover them adequately.

Permissions

SQL Server 2000 and earlier contained a reasonable scheme for assigning permissions to users, logins, and other principals. But when you attempt to lock down those versions of SQL Server, you're working with fairly blunt tools rather than the sharp scalpels necessary to implement strong security. Often you have to assign a user to a fixed role that has the permission the user needs as well as a dozen others that the user doesn't need. This is a gross violation of the principle of least privilege, in which each user or other principal has exactly the permissions needed, no more and no less.

The grantable permissions in SQL Server 2005 are far more specific than in its predecessors. Virtually any object has a variety of permissions grantable to virtually any principal. SQL Server still makes heavy use of roles at the server and database levels, but it is no longer necessary to add a user to a role when all the user needs is limited access to specific resources.

In the security terminology of SQL Server 2005, a principal is any individual, group (role), or process that can request access to a protected resource and be granted permission to access it. As in previous versions of SQL Server, a principal can be defined in Windows® or can be based on a SQL Server login with no corresponding Windows principal. Figure 1 shows the hierarchy of SQL Server 2005 principals, excluding fixed server and database roles. It also shows how logins and database users can be mapped to security objects. As in previous versions, in SQL Server 2005 the scope of the influence of the principal depends on the scope of its definition. Thus, a Windows-level principal is more encompassing than a SQL Server-level principal, which in turn is more encompassing than a database-level principal. And every database user automatically belongs to the fixed Public role.

Figure 1 SQL Server 2005 Principals Hierarchy (Close Figure 1)

Windows-level principals
Windows domain login
Windows local login
Windows group
SQL Server-level principals
SQL Server login
SQL Server login mapped to a Windows login
SQL Server login mapped to a certificate
SQL Server login mapped to an asymmetric key
Database-level principals
Database user
Database user mapped to a Windows login
Database user mapped to a certificate
Database user mapped to an asymmetric key
Database role
Application role
Public role

Also notable is that objects can be secured by granting or denying permissions. Figure 2 lists the securable objects in SQL Server 2005. At the server level, you can secure network endpoints to control the communication channel into and out of the server, as well as databases, bindings, and roles and logins. At the database and schema level, virtually every object you can create is securable, particularly within the schema object used to contain database objects.

Figure 2 Securable Objects in SQL Server 2005
Figure 2 Securable Objects in SQL Server 2005

The set of fixed server and database roles is unchanged from SQL Server 2000, so you can still take advantage of those predefined bundles of permissions when users or applications require all or most of the defined permissions. Under the principle of least privilege, however, you'll rarely want to use roles that give a principal excess permissions. Although it will be a bit more work discovering the permissions required for a principal and assigning them, it can result in a far more secure database environment.

For a sense of the number of permissions available in SQL Server, you can view the builtin_permissions catalog view:

SELECT * FROM sys.fn_builtin_permissions(default)
In the April CTP this returns more than 180 rows representing all of the permissions; I'll discuss a few of the more important ones.

The CONTROL permission confers ownership rights to the grantee, including the ability to grant permissions on the securable object. CONTROL rights on one level of the hierarchy of securable objects implies those same rights for all contained objects.

The ALTER ANY entity permission confers the right to create, alter, and drop any object of the entity type. For example, the ALTER ANY DATABASE permission at the server level allows the grantee to alter any database. The ALTER ANY ASSEMBLY permission within the scope of a specific database allows the grantee to make changes to any assembly.

IMPERSONATE ON login or IMPERSONATE ON user allows the grantee to impersonate the specified server login or database user so that any operations are performed using the impersonated security credentials. This permission is critical for the EXECUTE AS feature I'll cover later in this article.

SQL Server 2005 still uses the familiar GRANT, DENY, and REVOKE scheme for assigning or refusing permissions on a securable object. The GRANT statement has been greatly expanded to cover new permission options, such as the scope of the grant and whether the principal is able to grant the permission to other principals. Cross-database permissions are not allowed. In order to grant cross-database permissions, you'll have to create a duplicate user in each database and separately assign each database's user the permission. Like earlier versions of SQL Server, activating an application role suspends other permissions for the duration that the role is active.

Granting a particular permission can convey the rights of other permissions through implication. SQL Server Books Online contains the T-SQL for an ImplyingPermissions user-defined function that assembles the hierarchy list from the sys.fn_builtin_permissions catalog view and identifies the depth of each permission in the hierarchy. After adding ImplyingPermissions to the master database, I executed this statement to produce a hierarchical list of permissions, passing in the object and permission type:

SELECT * FROM master.dbo.ImplyingPermissions('schema', 'alter')

It might be difficult at first to appreciate just how granular permissions can be in SQL Server 2005. Consider the number and types of principals available and the number of securable objects in the server and a typical database. Consider also the sheer number of available permissions and the covered and implied permissions. Creating an application is going to require much more detailed analysis of its security needs and careful control of permissions on all objects. Nevertheless, there are still some actions that do not have a grantable permission. For example, TRUNCATE TABLE requires the ALTER permission on the table, which confers the power to do more than just delete a table's rows.

Catalog Views and Metadata Visibility

One of the benefits of the highly granular permissions scheme in SQL Server 2005 is that it affords better protection of the metadata necessary to a modern database engine. SQL Server has long supported rich metadata exposure through INFORMATION_SCHEMA views defined in the SQL-92 specification, system tables at the server and database level, and a vast number of built-in system stored procedures. You could even change the underlying state of the server and its databases by modifying the system tables, albeit with great risk to system stability. Microsoft constantly reminded us that the underlying system tables could change at any time, so using them was at our own risk.

In SQL Server 2005, Microsoft has created a new set of catalog views to expose all of the metadata throughout the server. The new sys schema encapsulates the metadata views. The catalog views are read-only, which eliminates some of the hacks that were possible in earlier versions of SQL Server. Many of the views have names similar to the old system tables to make it easier to find the information you want. For example, the sysobjects metadata is now exposed through the sys.objects view. To keep us all honest, the view definitions are physically hidden in an undocumented location with no way to directly access their definitions (unless someone hacks a server and finds them). The message is nevertheless clear: use the catalog views to get at all of the server and database data and forget about previous hacks. Stick to the stored procedures, T-SQL, and other formal ways of configuring and tweaking SQL Server. It's all for the sake of security and stability.

From a security standpoint, a benefit of using views to expose metadata is that the data returned from a catalog view is filtered according to the permissions of the user context under which the data is requested. For example, it used to be that a user who had permission to access the sysobjects table in a database—and most users did in many applications—could run a query against the table and see every available object in the database, whether or not she had access to each object. In SQL Server 2005, the user has to have some minimal access to a table, such as SELECT permission, to see the table. If the user had access to the database but not to any of its objects, a query against sys.objects returns no rows.

To see how this works, run the following code as an admin:

USE master
CREATE LOGIN User1 WITH password = 'myPassword'

-- Use the database of your choice
USE AdventureWorks
CREATE USER User1

EXECUTE AS LOGIN = 'User1'
SELECT * FROM sys.objects
-- Because the user has no permissions in the db, no rows are returned
REVERT

The code starts by creating a User1 login on the server, then adds a user mapped to that login to the sample AdventureWorks database. Then it changes the execution context to User1 and retrieves data from sys.objects. Nothing is returned from the catalog view, because while User1 has access to the database, it has no permissions on any objects. The REVERT statement reverts the execution context to the admin user.

In order to return data, you must give the user a couple of permissions. The following code grants SELECT permission on the Contact table and EXECUTE permission on the dbo.uspGetBillOfMaterials stored procedure:

-- Give User1 rights to a table and stored procedure
GRANT SELECT ON Person.Contact TO User1
GRANT EXECUTE ON dbo.uspGetBillOfMaterials TO User1

-- Execute again as User1
EXECUTE AS LOGIN = 'User1'
SELECT * FROM sys.objects
REVERT

When you execute the catalog view as User1, you get nine rows from the sys.objects catalog view, including metadata for the Contact table and uspGetBillOfMaterials stored procedure as well as related objects such as table constraints and a trigger. This demonstrates how granting a permission at one level of the object hierarchy grants related permissions on child objects.

As you'd expect, sysadmins and sa can see everything on the server in the system catalog views and database owners can see everything in their databases. The permissions-based filtering also applies to system stored procedures that reveal information about different objects, such as sp_help and sp_helpdb. Since these system stored procedures read the system catalog views, they are filtered according to the permissions of the principal under which the procedures run. You must be careful, though, since the limits on metadata visibility don't yet apply to all of the metadata functions, such as OBJECTPROPERTY.

Most of the old system tables, stored procedures, and views are still available for use, except that they are now all exposed as read-only views. They are there for backward compatibility and do not expose any of the new features of SQL Server 2005. Unless you are supporting old code from an earlier version, it is best to use the new system catalog views.

User/Schema Separation

The ANSI SQL-99 specification defines a database schema as a collection of database objects owned by a single principal that forms a single namespace of objects. A schema contains database objects such as its tables, views, stored procedures, functions, types, and triggers—which is a convenient way of grouping objects so a database can reuse object names and group objects under different owners.

In SQL Server 2000, the owner and schema were indistinguishable, as shown in the top portion of Figure 3. If an admin created a user called Fred in a database, there was also a schema called Fred created automatically that hid behind Fred the user. If Fred the user logged into SQL Server and created a table, Table1, the name of the table would be Fred.Table1. The same held for other objects Fred created, such as Fred.StoredProcedure and Fred.View1. If Fred were a database owner, then objects Fred1 created would be part of the dbo schema instead.

Figure 3 Users and Schemas in SQL Server
Figure 3 Users and Schemas in SQL Server

A problem arose when it would become necessary to change the ownership of objects, such as when Fred leaves the company and George takes over Fred's job. A system admin would have to change ownership of all of the objects owned by Fred. Any T-SQL or client application code that referred to Fred.Table1 would have to be changed to George.Table1 after George took ownership of the table. Depending on the number of objects Fred owned and how many apps had the name embedded in them, this could get complicated.

SQL Server 2005 solves this problem and more closely implements the SQL-99 schema by separating the user from the schema, as shown in the bottom part of Figure 3. When you create a new user called Fred using the new DDL statements, SQL Server no longer automatically creates a schema with the same name. Instead, you must explicitly create a schema and assign ownership. Because all of the database objects are now contained in the MySchema schema, which Fred initially owns, it becomes simple to change ownership of all the schema's objects by simply changing the ownership of the schema to George. Each user can also have a default schema assigned to it, so that any objects referenced by name without the schema reference will be assumed to be in the default schema. So in the bottom part of Figure 3, if user Fred has MySchema as his default schema, he can refer to the table as either MySchema.Table1 or simply as Table1. George, who perhaps does not have a default schema associated with his user name, would have to refer to the table as MySchema.Table1. Any user with no specified default schema has dbo as the default.

Figure 4 shows how this works. The code starts by creating a login called Carol and a user called Carol in the Pubs database with CREATE TABLE permission. The security context of the running code is changed to Carol, and then Carol attempts to create a table. This attempt fails with an error: "Specified schema name 'dbo' either does not exist or you do not have permission to use it." In this case, the problem is that Carol doesn't have permission to create objects in the dbo schema. Since she doesn't have a default schema specified, the CREATE TABLE statement is attempting to create dbo.table1.

Figure 4 Separation of Users and Schemas (Close Figure 4)
USE pubs

-- Create a new SQL Server login using GUI: 'Carol" with
-- password 'carol5', default DB pubs

CREATE LOGIN Carol WITH PASSWORD = 'carol5'
CREATE USER Carol FOR LOGIN Carol
GRANT CREATE TABLE TO Carol

EXECUTE AS LOGIN = 'Carol'
GO

CREATE TABLE table1 (tID int)
-- Error: Don't have permission to create object in dbo schema

-- Revert to admin
REVERT

-- Create schema and link to login
CREATE SCHEMA CarolSchema AUTHORIZATION Carol

EXECUTE AS LOGIN = 'Carol'
GO

CREATE TABLE table1 (tID int)
-- Still an error. Just because there is a schema, doesn't mean
-- it's the default

CREATE TABLE CarolSchema.table1 (tID int)
-- Success at last!

-- Clean up
REVERT

DROP TABLE CarolSchema.table1
DROP SCHEMA CarolSchema
DROP USER Carol
DROP LOGIN Carol

The code then reverts to the admin user who signed in to run this code and creates a CarolSchema, giving ownership of the schema to Carol. Then, again running in Carol's security context, it attempts again to create the table. The attempt again fails with the same error message. Just because Carol has a schema at her disposal doesn't mean that SQL Server will use it by default. She is finally able to successfully create the CarolSchema.table1 when she explicitly uses that object name and a schema that she has permissions to use. The second failed attempt at creating the table, after the CarolSchema existed, would have succeeded had Carol had a default schema set, either when the user Carol was created or by adding it to the user, as shown here:

CREATE USER Carol FOR LOGIN Carol WITH DEFAULT_SCHEMA = CarolSchema
-- or
ALTER USER Carol WITH DEFAULT_SCHEMA = CarolSchema

Execution Context

SQL Server has long supported ownership chaining to ensure that a user who is executing code that accesses data has the proper permissions. As long as the user calling the code had execute permissions and the owner of the code was the owner of the two tables and the view that was accessed, for example, no further permissions were checked and the caller received the requested data. If the ownership chain was broken—if, for example, the owner of the code didn't own the referenced view—the caller's security context was checked.

If the caller had permission to access the view, the data was returned. If she didn't, the data would not be returned. This is how SQL Server 2000, and previous versions, works; this is fine when you want to give users access to data through a stored procedure or function when they didn't have the underlying rights to the data. Ownership chaining has some limitations. One is that it only applies to data manipulation operations and not to dynamic SQL.

But maybe you don't want to use the caller's permissions to validate access to data. What if you want to create a stored procedure that uses another user's security context to check permissions as part of a carefully designed security framework to protect data? You didn't really have any easy options before SQL Server 2005, which introduces the concept of execution context. Now, when you define stored procedures, data manipulation triggers, and user-defined functions (except inline table-valued), you can use the EXECUTE AS clause to specify which user's permissions SQL Server uses to validate access to objects and data referenced by the procedure. For example, you can specify that the permissions associated with the creator of the code are always used to access the data. The following is an example that runs the code with a specific user's permissions. In this case, the user ec must have SELECT permission on the titles table, since the execution context is that of ec:

USE pubs

CREATE LOGIN ec WITH PASSWORD = 'ecpassword'
CREATE USER ec FOR LOGIN ec

CREATE PROCEDURE GetTitlesEC(@Table varchar(40))
WITH EXECUTE AS 'ec'
AS
    EXECUTE('SELECT * FROM ' + quotename(@Table))
GO

SQL Server 2005 provides four execution context options. EXECUTE AS CALLER specifies that the code is executed in the context of the caller of the module. The caller must therefore have execute permission on the module as well as access permissions on all of the underlying objects. SQL Server only checks permissions for broken ownership chains, so if the owner of the code also owns the underlying objects, only the module's execute permission is checked. This is the default execution context in order to ensure backward compatibility.

EXECUTE AS 'user_name' specifies that the code executes in the security context of the specified user. This is a great option if you don't want to rely on ownership chaining and instead create a user with the necessary permissions to run the code.

EXECUTE AS SELF is a shortcut notation for specifying that the module executes under the security context of the user creating or altering the module. SQL Server saves the actual user name associated with the module rather than SELF.

EXECUTE AS OWNER specifies that the security context is that of the current owner of the module. If no owner is specified, the context of the containing schema's owner is used. The owner must be a singleton account of a specific user and not a role. This is a great option when you want to be able to change the module's owner without changing the module itself.

There are a few limitations to changing the execution context of code. The creator of the module must have IMPERSONATE permissions for the specified user. You never need to have this permission to impersonate yourself. You also cannot drop the specified user from the database until the execution context of all modules has been changed to no longer use that user. Ownership chaining doesn't apply to dynamic SQL executed within the module. Like SQL Server 2000, the execution context must have permissions on the underlying objects accessed by the dynamic SQL.

Encryption

The best protection for a valuable resource is layer upon layer of security—a principle known as defense in depth. An attacker has to pierce one layer after another before attaining the prize. Far too often, hackers have been able to compromise network security, then machine security, then database security, gaining free access to the valuable data.

Encrypted data in SQL Server is a final line of defense. Even if an attacker successfully gains access to a database, she must still decrypt the data. Given today's strong encryption algorithms, without a decryption key the attacker faces a nearly insurmountable challenge. Encryption is never free, of course. You might be tempted to encrypt all the data stored in SQL Server, including publicly accessible data such as your product catalog. But that could incur a massive performance hit because encryption is processor-intensive. Multiply the processing cycles required to encrypt or decrypt one short string by 10 million rows of data in a table, and you can easily bring a server to its knees. You should consider the level of protection required, and use encryption only when protecting a field is worth the cost incurred.

SQL Server 2005 provides a variety of key types and encryption algorithms. For symmetric keys, you can use RC4, RC2, the DES family, and the AES family of algorithms. For asymmetric keys, it provides RSA, and for certificates it uses the Internet Engineering Task Force's X.509 V1 standard.

The hardest part of encryption is key management—keeping secrets secret. If an attacker obtains the symmetric key used to encrypt your data, he can access your data and can change it with abandon. SQL Server 2005 lets you manage keys yourself, or you can have SQL Server manage them for you, using a hierarchy of encryption objects to protect various types of keys at different scopes within the server (as shown in Figure 5).

Figure 5 Encryption Hierarchy in SQL Server 2005
Figure 5 Encryption Hierarchy in SQL Server 2005

At the top of Figure 5 is the Service Master Key, the mother of all keys and certificates in SQL Server. The Service Master Key, a symmetric key that directly or indirectly encrypts all keys on the server, is created automatically when you install SQL Server. If it is compromised, an attacker can eventually crack every key in every database. For that reason, it's protected by the Data Protection API (DPAPI) in Windows. You can access it using the service account name under which SQL Server is running.

SQL Server takes care of managing the Service Master Key for you, although you can perform certain maintenance tasks on it to dump it to a file, regenerate it, and restore it from a file using the T-SQL statements shown in Figure 6.

Figure 6 Maintaining the Service Master Key (Close Figure 6)
-- Export the service master key to store in safe location
-- The file is encrypted using the supplied password as the key
DUMP SERVICE MASTER KEY TO FILE = 'c:\servicemasterkey.dat'
    PASSWORD = 'S3@fBZir2D^P$x5P&tNr^uR!@wGW'
-- Reload the service master key
LOAD SERVICE MASTER KEY FROM FILE = 'c:\servicemasterkey.dat'
    PASSWORD = 'S3@fBZir2D^P$x5P&tNr^uR!@wGW'

-- Regenerate the service master key. Be careful! Decrypts all keys
-- encrypted with old master key then encrypts them with the new key,
-- which can require a lot of processing time.
ALTER SERVICE MASTER KEY REGENERATE

The ALTER SERVICE MASTER KEY statement also has options to change the recovery option and the service account used to encrypt the key in DPAPI. Most of the time, though, you won't need or want to make any changes to the key.

Within the scope of a database, the database master key is the root encryption object for all keys, certificates, and data in the database. Each database can have a single master key; you'll get an error if you try to create a second key.

You must create a database master key via CREATE MASTER KEY T-SQL before using it with a user-supplied password:

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'WeZ#6hv*XHq#akAEaqcr7%CUP3aQ'
The resulting key is encrypted with triple DES and stored twice. One storage location is in the sys.symmetric_keys database table, encrypted by the supplied password, and the second is the sys.databases table in the master database, encrypted using the Service Master Key. Such duplicate storage allows automatic opening of the master key.

Detaching a database with an existing master key and moving it to another server can cause a problem, since the new server's Service Master Key will be different. ALTER MASTER KEY has options to drop encryption by the Service Master Key on the old server and then add it back on the new server. Otherwise, the master key will always have to be explicitly opened before use.

Once the database master key exists, you can use it to create any of three types of keys, depending on the encryption you need:

  • Asymmetric keys are used for public key cryptography with a public and private key pair.
  • Symmetric keys are used for shared secrets where the same key both encrypts and decrypts data.
  • Certificates are essentially wrappers for a public key.

Figure 5 shows how these keys and certificates can be used to encrypt other keys and data. Asymmetric keys can encrypt symmetric keys and data, symmetric keys can encrypt other symmetric keys and data, and certificates can encrypt symmetric keys and data. Symmetric keys can also be created using a custom password if you want to handle key management yourself. Of course, all of these are ultimately encrypted using the database master key.

The certificate is the same sort of object that is used in the context of digital signatures when talking about signing a document, e-mail, or file. The certificate object wraps the public key used in asymmetric encryption with plenty of options for how it is created. But because SQL Server takes care of all the details, it hides the steps needed to create certificates in other environments.

You can have SQL Server create a certificate for its own use or import a certificate created by any trusted certificate authority, such as Microsoft Certificate Server or VeriSign. Options include setting a password to use to encrypt the private key or letting SQL Server use the database master key, as well as setting the creation and expiration dates. You can import the certificate from a file or read it from a signed executable or loaded .NET assembly. The following code creates two certificates, one that is associated with User1 and does not expire, and another for User2 that expires at the end of 2005:

-- Create a certificate associated with User1, encrypted with the
-- database master key. The database master key must already exist.
CREATE CERTIFICATE User1Certificate
    AUTHORIZATION User1 WITH subject = 'Certificate For User1'
GO
-- Create a certificate for User2 that is limited
CREATE CERTIFICATE User2Certificate
    AUTHORIZATION User2 WITH subject = 'Certificate For User2',
    EXPIRY_DATE = '12/31/2005',
    ENCRYPTION_PASSWORD = 'q%dsabciJ&#QZk#wM5G!WB36z5m7'

Once the certificate exists, you can encrypt data using one of the new T-SQL functions that supports encryption, EncryptByCert. Imagine that you have a table called Customer defined like this:

CREATE TABLE Customer (
    CustId int,
    name nvarchar(30),
    City varchar(20),
    CreditCardType varbinary(300),
    CreditCardNumber varbinary(300),
    Notes varbinary(4000))
GO
Encrypted data is binary, so the table uses varbinary fields to hold it. Because the encrypted data includes the digital certificate, the field lengths must be rather large, even to hold simple strings like "Visa" and a credit card number. Here is how you can insert a row of data into this table:
INSERT INTO Customer VALUES (1, 'Don Kiely', 'Fairbanks',
    EncryptByCert(Cert_ID('User1Certificate'), 'Visa'),
    EncryptByCert(Cert_ID('User1Certificate'), '1234-5678-8765-4321'),
    EncryptByCert(Cert_ID('User1Certificate'),
        'This customer is a real flake. Don''t trust him!'))

EncryptByCert takes two parameters, the certificate ID, which you can obtain with the Cert_ID function, the name you gave the certificate, and the data to be encrypted.

You can see the benefit of encrypting data when you select data from the table. Running a SELECT * FROM Customer in SQL Server Management Studio gives the result shown in Figure 7. Because the INSERT statement uses multiple calls to the Encrypt function, each field can be encrypted using different methods.

Figure 7 SELECT on Table with Encrypted Fields (Close Figure 7)

To decrypt the data as part of a SELECT statement, use DecryptByCert as shown here:

SELECT CustID, Name, City,
    CONVERT(VARCHAR,
        DecryptByCert(Cert_ID('User1Certificate'),
            CreditCardType)) AS CardType,
    CONVERT(VARCHAR,
        DecryptByCert(Cert_ID('User1Certificate'),
            CreditCardNumber)) AS CardNumber,
    CONVERT(VARCHAR,
        DecryptByCert(Cert_ID('User1Certificate'),Notes)) AS Notes
FROM Customer
Because DecryptByCert returns varbinary data, you usually need to convert it to some other data type, here a varchar. This returns the clear text, as shown in Figure 8.

Figure 8 SELECT Using DecryptByCert
Figure 8 SELECT Using DecryptByCert

A very common type of encryption, one performed with asymmetric keys, is frequently called public key encryption. Asymmetric keys in SQL Server 2005 use the RSA algorithm with key sizes of 512, 1,024, or 2,048 bits. You can encrypt the private key generated either with a password you specify or with the database master key. You can also import the keys from a disk file, an executable file, or a .NET assembly loaded in memory. The following code shows a couple of different ways to create the keys. The first uses a secret managed by the user and the second uses the database master key managed by SQL Server:

-- Create an asymmetric key pair with the private key protected by a
-- user-supplied password.
CREATE ASYMMETRIC KEY User1AsymmetricKey
    AUTHORIZATION User1
    WITH ALGORITHM = RSA_2048
    ENCRYPTION BY PASSWORD = 'AVeryVerySecretPassword'

-- Create another that is protected by the database master key
CREATE ASYMMETRIC KEY User2AsymmetricKey
    AUTHORIZATION User1
    WITH ALGORITHM = RSA_2048

Once the keys are available in the database, you can use the EncryptByAsymKey function to encrypt the data. Like the EncryptByCert function, the first parameter is the ID of the key you want to use, which you can obtain with the AsymKey_ID function. There is no need to use the same key for every encrypted field in a record, although that's the approach used in Figure 9.

Figure 9 Encrypting Data (Close Figure 9)
INSERT INTO Customer VALUES (2, 'Joe Public', 'North Pole',
    EncryptByAsymKey(AsymKey_ID('User1AsymmetricKey'), 'MasterCard'),
    EncryptByAsymKey(AsymKey_ID('User1AsymmetricKey'),
        '8765-4321-1234-5678'),
    EncryptByAsymKey(AsymKey_ID('User1AsymmetricKey'),
        'A total sucker. Sell him up!'))

INSERT INTO Customer VALUES (3, 'Jane Doe', 'Ester',
    EncryptByAsymKey(AsymKey_ID('User2AsymmetricKey'), 'Discover'),
    EncryptByAsymKey(AsymKey_ID('User2AsymmetricKey'),
        '9999-8888-7777-6666'),
    EncryptByAsymKey(AsymKey_ID('User2AsymmetricKey'),
        'Favors shoes of every style and color.'))

Use the DecryptByAsymKey function to decrypt each column's data, once again, using the AsymKey_ID function with the name of the asymmetric key used to encrypt the data. In Figure 10, User1AsymmetricKey was created with a password, so you need to pass that password as the third field of the DecryptByAsymKey function call. You'll usually need to use the CONVERT function to convert the varbinary data to human-readable (see Figure 10).

Figure 10 Decrypting with DecryptByAsymKey (Close Figure 10)
-- View the decrypted data. Have to pass in the password used to protect
-- the private key.
SELECT CustID, Name, City,
    CONVERT(VARCHAR,
        DecryptByAsymKey(AsymKey_ID('User1AsymmetricKey'),
            CreditCardType, N'AVeryVerySecretPassword')) AS CardType,
    CONVERT(VARCHAR,
        DecryptByAsymKey(AsymKey_ID('User1AsymmetricKey'),
            CreditCardNumber, N'AVeryVerySecretPassword')) AS CardNumber,
    CONVERT(VARCHAR,
        DecryptByAsymKey(AsymKey_ID('User1AsymmetricKey'),
            Notes, N'AVeryVerySecretPassword')) AS Notes
FROM Customer WHERE CustID = 2

For data encrypted with User2AsymmetricKey (see Figure 11), which was created with the database master key, there is no need to pass in any credentials because SQL Server is managing the key for you. Hopefully you are starting to see that it is far easier to let SQL Server manage keys for you.

Figure 11 Decrypting without Passing Credentials (Close Figure 11)
-- View CustID 3. No need to pass password since protected with database
-- master key.
SELECT CustID, Name, City,
    CONVERT(VARCHAR,
        DecryptByAsymKey(AsymKey_ID('User2AsymmetricKey'),
            CreditCardType)) AS CardType,
    CONVERT(VARCHAR,
        DecryptByAsymKey(AsymKey_ID('User2AsymmetricKey'),
            CreditCardNumber)) AS CardNumber,
    CONVERT(VARCHAR,
        DecryptByAsymKey(AsymKey_ID('User2AsymmetricKey'),
            Notes)) AS Notes
FROM Customer WHERE CustID = 3

If SQL Server creates the asymmetric key pair, you can get the public key for transmitting to others with the sys.asymmetric_keys catalog view, which exposes the public key in the public_key field. The following code returns the public keys associated with each asymmetric key pair you create:

SELECT Name, Public_Key FROM sys.asymmetric_keys

Certificates and asymmetric keys provide industrial-strength encryption for SQL Server data, particularly when you need to move the data outside of the database and protect it en route. But if you need to protect the data on the server and not worry about it in other locations, symmetric keys are a good choice. This is a fairly common database scenario, where data is stored securely, decrypted on the server, sent to the client and displayed to an authorized user. Symmetric keys require far fewer processing cycles than other forms of encryption. Because it is so well-suited to database applications, symmetric key encryption in SQL Server is quite flexible.

Creating symmetric keys is similar to creating certificates and asymmetric keys. The CREATE SYMMETRIC KEY statement has several options. One option is how the key is itself encrypted when stored in the database. Referring back to Figure 5, you can protect a symmetric key using a certificate, asymmetric key, user-specified password, or another symmetric key. Others include the encryption algorithm used, whether to use a user-supplied passphrase to generate the key, and an optional identity phrase. This phrase generates a GUID that is used to tag data encrypted with a temporary key. SQL Server supports the majority of the most widely used symmetric key algorithms, including DES, triple DES, and AES. The algorithm must be installed in Windows on the server.

When considering the options for creating symmetric keys and whether to manage the key yourself or let SQL Server do it for you, it is important to consider how the optional password is protected. If you specify a password to encrypt the key, triple DES is used to derive a key from the password, no matter what algorithm you elect to use to encrypt the data with the resulting symmetric key. It is quite possible, then, to use stronger encryption for data than that used to protect the password protecting the key.

Figure 12 shows a few of the many options for creating symmetric keys. The first example creates a key with triple DES secured by an existing certificate, giving User1 ownership of the resulting key. Notice that you don't have to explicitly open the certificate in order to use it, unlike a symmetric key. The second sample creates a similar symmetric key, but instead of a certificate, the user has chosen to manage the key by supplying a password. The third sample omits the AUTHORIZATION clause so that the key is owned by dbo, and uses the RC4 algorithm. This sample uses an existing asymmetric key to protect the new symmetric key. The last sample creates another symmetric key owned by dbo that uses DESX to encrypt data, but is itself protected by another symmetric key. Before you use the existing symmetric key to encrypt the new key, you have to explicitly open the existing key.

Figure 12 Create Symmetric Keys in SQL Server 2005 (Close Figure 12)
-- Create a key using Triple DES secured by a certificate and associate
-- it with User1
CREATE SYMMETRIC KEY User1SymmetricKeyCert
    AUTHORIZATION User1
    WITH ALGORITHM = TRIPLE_DES
    ENCRYPTION BY CERTIFICATE User1Certificate

-- Create a key using Triple DES secured by a password and associate
-- it with User 2
CREATE SYMMETRIC KEY User2SymmetricKeyPwd
    AUTHORIZATION User2
    WITH ALGORITHM = TRIPLE_DES
    ENCRYPTION BY PASSWORD = 'AGreatPassword'

-- Create a key using RC4 secured by an asymmetric key and owned by dbo
CREATE SYMMETRIC KEY GenericSymmetricKeyAsym
    WITH ALGORITHM = RC4
    ENCRYPTION BY ASYMMETRIC KEY User2AsymmetricKey

-- Create a key using DESX secured by another symmetric key and
-- owned by dbo
OPEN SYMMETRIC KEY User1SymmetricKeyCert
    USING CERTIFICATE User1Certificate
CREATE SYMMETRIC KEY GenericSymmetricKeySym
    WITH ALGORITHM = DESX
    ENCRYPTION BY SYMMETRIC KEY User1SymmetricKeyCert
CLOSE SYMMETRIC KEY User1SymmetricKeyCert

Inserting and selecting data encrypted with symmetric keys is similar to the method used for certificates and asymmetric keys, but is a bit more involved, as you can see in the sample code download (available on the MSDN®Magazine Web site). The code uses the EncryptByKey and DecryptByKey functions to convert the data, along with the Key_GUID function to retrieve the ID of the specified key. You have to explicitly open a symmetric key before using it, since you can't create such a key that is encrypted by the database master key. The sample creates and uses a GenericSymmetricKeySym key. Because this key is itself protected by the symmetrical key User1SymmetricKeyCert, you have to first open the symmetric key that protects the key you want to use. This demonstrates that you can have multiple symmetric keys open in a database at any time. SQL Server knows which key to use to decrypt the data; you don't have to specify it in the DecryptByKey function.

One of the interesting benefits of the SQL Server 2005 implementation of encryption is that it provides a way to let users see only their own data. By using the AUTHORIZATION clause as I've done in many of the samples in this article, you can give ownership of the key or certificate to a user who can then grant rights to other users to see the data.

Conclusion

SQL Server 2005 has an unprecedented level of support for security in a database engine. It provides a rich set of features developers can use to create secure applications that can access data on a secure server. Highly granular permissions let you practice the security principle of least privilege, granting a user just the permissions necessary to do a job and no more, preventing attacks that take advantage of bloated permissions. These and other security enhancements in SQL Server 2005 provide all the tools you need to weather today's sophisticated attacks on your data.

Don Kiely, MVP, MCSD, is a senior technology consultant and developer who focuses on security, particularly in the development of distributed Windows Forms and ASP.NET applications. He also writes about technology, speaks at conferences, and trains others. Reach Don at donkiely@computer.org.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Know Thy Code: Simplify Data Layer Unit Testing using Enterprise Services

Know Thy Code

Simplify Data Layer Unit Testing using Enterprise Services

Roy Osherove
This article discusses:
  • Unit testing principles
  • Database state management
  • Enterprise Services and COM+ 1.5 basics
  • Services Without Components
This article uses the following technologies:
COM+ 1.5, C#, Visual Studio, Testing
Contents
The State of Database Testing
Keeping Consistent State
Database Restore
ADO.NET Transaction Object
COM+ Transaction—Inherit ServicedComponent
Simple, Easy, and Painless
COM+ 1.5
Unit Testing in Visual Studio 2005 Team System
Conclusion

D
espite all the hype surrounding unit testing and Test Driven Development (TDD) many developers don't see them as useful processes that are applicable to the production of real-world applications. One reason is that once you get down to the intricacies of application development, all of the easy testing you've heard so much about can vanish. You may find yourself writing unit tests that are hard to implement against real code. This is often the case in database testing, where TDD can quickly lose its appeal, and force you to revert your processes to the previous real-world methodologies your team employed. Here, I'll present some techniques to make your testing process easier, even when developing real-world applications, using unit testing.

The State of Database Testing

Most well-architected data-driven systems have some sort of data access layer (DAL). This could be a specific class or even a separate class library project that is responsible for all the communications with the specific database. When I refer to database unit testing, I am actually talking about writing unit tests that exercise and drive the development of that specific DAL component.

Testing DALs is trickier than checking simple logic and algorithmic code. There are two possible ways to run unit tests against a DAL. The first is to replace the database API that your DAL is talking to with an object of your own creation that mimics the database's interface, but does not actually perform any tasks. This method is often referred to as "mocking" or "stubbing." Usually, a mock of an object has specific behavior and expects various actions to be performed on the replaced object, while a stub is merely a silent replacement for something that is not used to test logical actions. A stub just helps you decouple the tested object from interacting with expensive or time-consuming resources. Mocking usually means that you are performing interaction testing, in which you test interactions between objects. For more information on mock objects, see Mark Seemann's article in the October 2004 issue of MSDN®Magazine, available at Unit Testing: Mock Objects to the Rescue! Test Your .NET Code with NMock.

Figure 1 State-Based Testing
Figure 1 State-Based Testing

The second testing method involves your DAL calling the real database behind the scenes, which entails various potential problems. This usually means that you are performing state-based testing. You don't test logical calls between objects (namely, your DAL and the ADO.NET infrastructure), but simply perform actions and check the state of your tested objects afterward to make sure they fit your requirements. Figure 1 shows how all the objects and layers take part in the test. For more on state-based testing, see Martin Fowler's article at Mocks Aren't Stubs.

Keeping Consistent State

There are a number of things you should keep in mind when you're writing unit tests.

Always know the initial state before testing A test should always start with a predictable initial state. For example, if you're testing functionality that deletes rows from a specific database table, make sure that before the test begins you have known data in the table that you can delete in your test. This means that before each test you need to create your own initial state, be it data in the database or fresh instances of tested objects. You cannot rely on anything but the current running test to do this.

This is usually accomplished with the aid of a setup method. Think of the setup method as a constructor that is invoked for each test case in your test fixture. If you have five test cases, the setup method would be invoked five times—once before each test case starts. With NUnit, creating a start method is done by marking a method with SetUpAttribute. This is the place to set up fresh instances of objects to be tested, thus decoupling each test's state from every other test's state.

Each unit test is independent of the others A test should not depend on any other unit test to do its job, and a test should not be required to run in a specific order. Therefore, you should be able to run each of your unit tests separately in any order and get the same results every single time. The NUnit framework does not guarantee that your tests will be run in any particular order. This means that you will need to write each test as if it were the one and only test in the whole testing project.

A unit test fails or passes in a consistent, predictable manner If a valid test fails, it should continue to fail until you fix the bug that causes the failure. Likewise, if a test passes once, it should always pass until you introduce a change into the system that causes it to no longer pass. Running a unit test several times on the same compiled code and having it fail on some of them and pass on others can both lull you into a false sense of security and frustrate you to no end. Again, always know the initial state before testing.

The problem with testing against a live database is that many unit tests you write will make changes to the state of the database—changes that will be visible to all of the other unit tests that run against the same database. Suppose you have a test that deletes a row in the database. You might have a separate test that relies on the fact that a specific number of rows exists in the database before running (perhaps it checks that a read operation succeeds by checking the number of rows returned), which will fail if it was run right after your deletion test. Since this is a database, possibly shared with other developers and testers, you can't just add fresh objects into your test state using a setup method. This external state dependency has to somehow be consolidated before you run the next test. Creating a new object instance isn't going to help here; you need to actually return the database to its previous state.

All the rules I've mentioned so far underscore how important it is to keep the database state consistent between the tests. You have to undo any changes that are performed against the database in each test, even though it is this rolling back that is the most problematic and can lead to unmaintainable code.

There are several solutions to the problems I've mentioned. Since this article deals with a specific solution in detail I will limit the discussion of the other solutions. Fortunately, they're quite simple.

Database Restore

The perfect way to restore a database to a known state is, of course, to actually restore it from a previously made backup copy. If you run a simple RESTORE DATABASE script before each test, you could ensure the tests will be run each time on the same database. Unfortunately, though, this is quite time-consuming. If you assume that it takes five seconds to do a simple restore of a small SQL Server™ database, and multiply that by a couple hundred tests (each test triggers a setup method that restores the database) the restore process itself would add more than 16 minutes of overhead to all those tests. This is hardly a great way to spend your day, especially if you want to be able to run the tests as you develop your code. However, this solution is handy for specific cases in which you have tests that perform a significant amount of data input into the database (usually in load tests). This is typically not the case with the unit tests I am describing here, which are much simpler (insert/update/delete one or a few rows in the database each test).

Things get a little tricky and performance begins to degrade if you're testing against multiple databases in the same testing project (multiply that restore time by the number of databases and see the frightening result). The same is true if a number of developers are testing against the same database, as each developer's tests will interfere with everyone else's tests.

If you choose not to restore the database completely for each test, you are left with the option of using a transaction against the database operations to roll back your changes. There are several ways you can accomplish this.

ADO.NET Transaction Object

Transactions are a great way to deal with this problem. In his book Test Driven Development with Microsoft .NET (Microsoft Press®, 2004), James Newkirk talks about using transactions to undo the effects unit tests have had on your database. He explains, however, how to use ADO.NET 1.x transactions to achieve this, which in my opinion is not the cleanest way. The idea is that you explicitly open a transaction object inside your test setup method and then send that ITransaction object into your tested object, which uses the open connection to execute the tasks. At the end of the test, you simply call the ITransaction's Abort method to undo any changes made to the database during the test.

This method is overly complex and is hard to implement and maintain for each specific project. It also requires you to modify your tested object's API to receive such a transaction. This is something that may be feasible in some systems but certainly is not possible in many other systems in which the API must remain closed. This is particularly true for systems that use COM+ transactions to achieve their database functionality. Newkirk admits that this isn't the most elegant solution, and I agree. I'm not that keen on changing my whole product's APIs just so their test effects can be rolled back. Of course, there are other ways to pass information into methods other than through method signatures, such as using static fields or thread-local storage (TLS). If you use ADO.NET transactions for your testing purposes, you might consider these as an alternative.

COM+ Transaction—Inherit ServicedComponent

COM+ transactions are a great way to overcome the obstacles inherent in the previous approach. You need to somehow make your unit tests run inside a COM+ transaction and then abort that transaction during your test's teardown method (a teardown method runs at the end of your test, and is denoted in NUnit through the TeardownAttribute). Plus, your tested APIs don't have to change. It should just work. Remarkably, this is one of the easiest things to implement.

To participate in a transaction, you need a class that inherits from the System.EnterpriseServices.ServicedComponent class. Your test fixture class, which tests the DAL object, is the perfect candidate. Simply make it inherit ServicedComponent, thereby allowing it to participate in COM+ transactions using the ContextUtil class found in the System.EnterpriseServices namespace. The convenient thing about using a COM+ ServicedComponent is that to enable transactions, all you have to do is put a simple attribute on top of the class declaration. In this case, you can put up an attribute that tells COM+ that each public method that is invoked on an instance of this class must run inside a new transaction context, as shown here:

[TestFixture]
[Transaction(TransactionOption.RequiresNew)]
public class TransactionalTests:ServicedComponent
{
    [Test]
    public void Insert()
    {
        //insert something into the database
    }
}
Each time you run this specific test fixture, each test will be invoked inside a transaction.

Now all you need is to make sure that after each test is run, the transaction it participated in will be rolled back. This is easily done using a [TearDown] method, which uses ContextUtil.SetAbort to roll back the transaction, as shown in Figure 2.

Alternative Testing Frameworks

There are multiple unit-testing frameworks that replace the NUnit framework and that allow you to run transactional unit tests through a simple attribute called RollbackAttribute. These frameworks use COM+ 1.5 to control the transactional unit tests, and as such they require the same set of operating systems as the ones I've mentioned.

MbUnit MbUnit is a recent and welcome addition to the testing frameworks world. It is written by Jonathan de Halleux and can be downloaded at mbunit.tigris.org. The framework includes a RollbackAttribute that uses the same techniques that have been described in this article.

It also includes a SqlRestoreAttribute that performs a database restore action for each specific test, as mentioned in the first method in the article. MbUnit also comes with many other attributes that can make a test developer's life easier, including RepeatAttribute and PerfCounterAttribute, which allow for testing against minimum or maximum performance counter values. Although it is still in its infancy, MbUnit has many benefits, including the ability to run NUnit tests.

XtUnit XtUnit is an add-on library you can download and use in your own code. It is an elegant way of adding rollback capabilities to your tests, no matter what test framework you are working with. It uses .NET interception to achieve this. It is also extensible, allowing you to easily create attributes of your own to be executed on your tests. You can get more information on this framework at teamagile.com/mainpages/downloads.html.

NunitX NunitX is a variant of the NUnit testing framework, including the GUI and console runners, and has been slightly enhanced to include a new attribute named RollbackAttribute. By applying this attribute to specific unit tests, you run the test inside a transaction context. The framework uses COM+ 1.5 SWC. The same principles you saw here are implemented in the framework. It can be downloaded from www.osherove.com.

Make Your Own Attribute

The latest version of the NUnit framework (2.2.1) includes the ability to extend it with additional attributes. This lets you add your own rollback attribute quite easily for your own tests.

Figure 2 Rolling Back Transactions (Close Figure 2)
[TestFixture]
[Transaction(TransactionOption.RequiresNew)]
public class TransactionalTests:ServicedComponent
{
    [Test]
    public void Insert()
    {
        // Perform your magic against the database
        CategoriesManager mgr = new CategoriesManager();
        int newID = mgr.InsertCategory("MyCategory");
        Assert.IsTrue(newID != 0, "returned ID should be more than zero");
    }

    [TearDown]
    public void Teardown()
    {
        if(ContextUtil.IsInTransaction)
        {
            // Abort the running transaction
            ContextUtil.SetAbort();
        }
    }
}

Any code, including any DAL code that you call, and any changes to the database resulting from the execution of this code, will be rolled back. Figure 3 shows the test flow using a COM+ transaction as a rollback mechanism.

Figure 3 Using a COM+ Transaction to Roll Back Changes
Figure 3 Using a COM+ Transaction to Roll Back Changes

Simple, Easy, and Painless

There are several drawbacks to keep in mind when using this method. Because using serviced components in this fashion means that all the tests in your test fixture will run inside a transaction, then even tests that only perform read operations with no changes to the database will use transactions. And transactions, though not as time-consuming as a full database restore, still incur a cost. To avoid that overhead, you need to write a whole new test fixture just to separate the "write" tests from the "read" tests, which makes your unit testing code more complex.

If the code you are testing has specific requirements that conflict with the COM+ transaction requirements, you must assume no rollback will be available. A few common scenarios cause this conflict. First, you could have code in your DAL object that itself makes use of TransactionOption.RequiresNew. That causes the DAL object to disregard the transaction started by your test fixture and to create a new transaction—one that you cannot control from within your test. Second, you could have code in your DAL object that uses TransactionOption.Disabled or TransactionOption.NotSupported. This means the code will perform its actions outside of any given transaction context; therefore, any changes to the database cannot be rolled back using the transaction context.

Classes that inherit from ServicedComponent are required to be inside a strongly named assembly. That means your tests will have to be signed. Also, any assembly that is referenced by a strongly named assembly is required to be strongly named as well. When I just want to do some simple database tests against a simple proof-of-concept assembly, I often don't have the time or willpower for signing and naming assemblies. Furthermore, there is no way to test an assembly that isn't strongly named without first disassembling it and then reassembling it with a strong name key.

Additionally, it seems that, for whatever reason, the current version of NUnit (version 2.2.1) does not always play well with serviced components, throwing a variety of exceptions when running tests inside the GUI driver. For those using NUnit as a testing harness, this problem makes this method practically unusable in many cases (though it might work just fine on some machines and in some circumstances).

COM+ 1.5

So how can you overcome problems such as the inability to use the NUnit GUI runner to run the tests or the need to strongly name the assemblies? Enter COM+ 1.5. COM+ 1.5 is an enhancement to the COM+ Enterprise Services you know and love. To take advantage of the features I'm about to demonstrate, you need to be running either Windows® XP Service Pack 2 or Windows Server™ 2003 or later. This really shouldn't pose any configuration problems, since the only machines that need COM+ 1.5 in this case are the testing/build/development machines, not the customer machines. If you attempt to use these features in systems running earlier versions of the software than those just mentioned, a nasty PlatformNotSupportedException will be thrown.

Among other things, COM+ 1.5 lets you use Enterprise Services without inheriting ServicedComponent explicitly with a feature called Services Without Components (SWC). Instead, all you have to do is get familiar with two really simple classes that reside in the System.EnterpriseServices namespace: ServiceConfig and ServiceDomain. These two classes let you enlist in transactions and abort them without ever needing to change your inheritance chain (that is—create a class that inherits from ServicedComponent). Figure 4 shows some very simple code that accomplishes exactly what the code in the previous samples did: it makes a test run inside a transaction.

Figure 4 COM+ 1.5 Transaction Code (Close Figure 4)
[SetUp]
public void Setup()
{
    // Enter a new transaction without inheriting from ServicedComponent
    Console.WriteLine("Attempting to enter a transactional context...");
    ServiceConfig config = new ServiceConfig();
    config.Transaction= TransactionOption.RequiresNew;
    ServiceDomain.Enter(config);
    Console.WriteLine("Attempt suceeded!");
}

[Test]
public void Insert()
{
    // Perform your magic against the database
    CategoriesManager mgr = new CategoriesManager();
    int newID = mgr.InsertCategory("MyCategory");
    Assert.IsTrue(newID != 0, "returned ID should be more than zero");
}

[TearDown]
public void Teardown()
{
    Console.WriteLine("Attempting to Leave transactional context...");
    if(ContextUtil.IsInTransaction)
    {
        // Abort the running transaction
        ContextUtil.SetAbort();
    }
    ServiceDomain.Leave();
    Console.WriteLine("Left context!");
    // Trying to access ContextUtil now will yield an exception

}

There's no ServicedComponent in sight; instead, there is a very simple use of the two classes introduced in COM+ 1.5. ServiceDomain is used to enter or leave a transactional context. Within a transactional context, you can use the ContextUtil object at any time to manipulate transactions. The static Enter method takes an argument of the ServiceConfig type, which essentially tells the transactional context the requirements of your component. In fact, it's a complete replacement of the [Transaction] Attribute that you put on top of your class in the previous example. You simply specify a property on the ServiceConfig class that tells the context that you require a new transaction. And the beautiful thing is that you can enter or leave a ServiceDomain at any time, with any code between those two calls able to commit or rollback the current transaction.

You can specify which tests participate in a transaction and which don't. Right now all tests do because SetUp and TearDown are called for each test. But entering and leaving a transaction domain takes time, so it's better if tests that don't need this rollback functionality don't actually do their work inside a transaction. If you just put individual calls in each test case to open and close a transaction domain instead, you'd get better granularity, at the cost of writing more repetitive code. See the "Alternative Testing Frameworks" sidebar for information on frameworks such as XtUnit and MbUnit that provide built-in attributes with rollback abilities.

If your test performs a significant number of writes into the database, you run the risk of making the rollback and the transactional enlistment take a long time. In such cases, using a database restore may be faster or perhaps even preferable.

SWC is the most elegant of the three transaction-based methods I've discussed and allows the most flexibility. It is currently my favorite method for performing unit tests that involve database interactions, although the new System.Transactions namespace in the .NET Framework 2.0 may take its place in my bag of tricks (for more information on System.Transactions, see John Papa's column in the February 2005 issue of MSDN Magazine, at Data Points: ADO.NET and System.Transactions).

Unit Testing in Visual Studio 2005 Team System

Visual Studio® 2005 Team System provides a built-in unit testing framework for testing managed code, available through a new kind of project type: the test project. This is a special project in which you can create unit tests, load tests, and a variety of other kinds of tests, to exercise your application and return results. Every solution that contains a test project also contains a test run configuration file, which stores the settings used for running tests. These settings include the computer on which to run tests (local or remote), the hosting process (such as ASP.NET), and whether to gather code-coverage results. Team System also lets you run tests from either a command line or within the Visual Studio IDE. You can then see the results, along with the reasons for any failed tests, and publish the results so that they can be analyzed by other team members or incorporated into reports.

After you have run unit tests, you can see the unit test code coverage — how much of your code has been exercised by the tests you ran. Code coverage is reported to you in two ways, through statistics for what percentage of your code was actually run and through color coding of the source displayed in Visual Studio. Code that was run is highlighted in green and code not run is highlighted in red.

Two special kinds of unit tests are also available in Team System: data-driven unit tests and ASP.NET unit tests. A data-driven test is a unit test that you configure to be called repeatedly for each row of data from some data source. ASP.NET unit tests are used to exercise code in an ASP.NET application as it responds to page requests. ASP.NET unit tests are hosted inside the process of the ASP.NET application being tested.

Some new attributes in Team System are similar to what you've used in other unit-test harnesses. Instead of NUnit's TestFixtureAttribute, you now have TestClassAttribute, and instead of NUnit's TestAttribute, you have TestMethodAttribute. The same goes for SetUpAttribute and TearDownAttribute, which are replaced by TestInitializeAttribute and TestCleanUpAttribute, respectively.

In Team System, you can automatically generate unit tests from the classes used in the source code you want to test. Generated unit tests contain minimal code, and need to be edited before they produce meaningful test results. However, when you edit unit tests, you are aided by the classes and methods of the Unit Testing Framework of Team System. It offers, for example, new flavors of Assert methods, which you can use to pinpoint the results that differ from what you expect. The Assert class should be familiar to anyone who has used NUnit.

Unfortunately, there are currently no special features in Visual Studio 2005 that improve database unit testing, so you'll probably still end up performing some of the tasks discussed in this article. Figure 5 shows a simple test class that uses exactly the same rollback mechanism as employed in the previous demos, using SWC. The syntax is only slightly different. As with all new features in Visual Studio 2005, changes could occur before the final release.

Figure 5 Rolling Back with SWC (Close Figure 5)
[TestClass]
public class TransactionalTests2005
{
    [TestInitialize]
    public void Initialize()
    {
        ServiceConfig config = new ServiceConfig();
        config.Transaction = TransactionOption.RequiresNew;
        ServiceDomain.Enter(config);
    }

    [TestMethod]
    public void UpdateCategoriesTest()
    {
        // Insert something into the database
    }

    [TestCleanup]
    public void Cleanup()
    {
        if (ContextUtil.IsInTransaction)
        {
            ContextUtil.SetAbort();
        }
        ServiceDomain.Leave();
    }
}

Conclusion

Database testing can be a challenge unless you use the right tools and techniques. Database unit testing is an integral part of the testing process. Using COM+ 1.5 makes it very easy to make the code work as you envisioned it. Many of the techniques I've described here will probably remain unchanged through the next release of Visual Studio. And while Visual Studio 2005 won't necessarily change the code you write in your tests, it will help you quickly get up and running with the unit testing process for your apps.

Roy Osherove is the Principal of Team Agile, a consultancy devoted to agile software development and .NET architecture. He also maintains a blog with related information at www.iserializable.com. You can reach Roy at Roy@TeamAgile.com.


Say What?: Make Sense of Your Web Feedback using SQL Server 2005

Say What?

Make Sense of Your Web Feedback using SQL Server 2005

Peter Pyungchul Kim
This article is based on a prerelease version of SQL Server 2005. All information contained herein is subject to change.
This article discusses:
  • How to automate classification of text-based data
  • How to use SQL Server 2005 components to perform manual and automatic text classification
  • Performing term extraction and data preparation
  • Mining model development and refinement
This article uses the following technologies:
SQL Server 2005
Contents
SQL Server 2005 Data Mining
Preparing Data for Manual Classification
Mining Models Development
Using and Refining the Mining Models
Getting the Best Result
Defining the Model
Testing and Further Refinement
Deploying the Model
Conclusion

T
he interactive nature of the Web has made it a great medium for data collection. As Web sites have evolved beyond click and buy, the solicitation of feedback has held the promise of greater insight into the customer and her needs, necessary product improvements, and future directions. But collecting reams of text-based data is not the same as gleaning useful information from it. In a typical customer feedback Web page, users enter comments about your product (see Figure 1). Some fields present preset answers users can choose from. Other fields let users type in any comments they like. This sort of free-form comment is especially difficult to put to good use because it must be divided into classifications if it is to be analyzed, quantified, and used for decision making.

Figure 1 Customer Feedback Web Page
Figure 1 Customer Feedback Web Page

The task of dividing text into categories is called text classification—a process that involves identifying the relationship between the categories and the text (words and phrases). Text classification can be manual or automatic. In the manual approach, a group of domain experts analyze the feedback and assign it to categories manually. Various software tools and technologies can help with this task. In the automatic approach, patterns between the feedback and its category are extracted from previously categorized results and used to classify new feedback without human intervention.

The SQL Server™ 2005 data mining functionality can be used for both manual and automatic classification. In this article, I'll cover both and show you how similar practices can be applied to any type of text classification problems, including document classification and blog grouping.

A general Web feedback system typically involves more than just text classification. It consists of a Web feedback front end that collects user feedback and stores it in a database, a GUI-driven classification component that lets a reviewer manage a hierarchy of categories and manually or automatically classify the new feedback, and a reporting component that generates various trend analysis reports. In this article, I'll focus specifically on how the SQL Server 2005 data mining features can be used to develop the core functionality of the text classification component.

SQL Server 2005 Data Mining

Data mining is a technology that enables you to uncover patterns in your data. Detecting relationships between the categories and the text in the Web feedback is a basic data mining task. For instance, if a piece of feedback contains the words "virus" and "Windows Update," a data mining model can classify the feedback in the security category with a high degree of confidence.

Microsoft introduced its first data mining features in SQL Server 2000 as part of Analysis Services. There has since been a huge investment made to equip SQL Server 2005 with a full range of data mining functionality for enterprise customers. Just looking at the surface, you can see that five new algorithms and more than a dozen visualizations have been added, along with additions to the Data Mining Extensions to SQL (DMX) language. You'll also find other great improvements such as programmability models and integration with other components such as online analytical processing (OLAP), Integration Services (formerly known as Data Transformation Services, or DTS), and Reporting Services.

In particular, it should be noted that Integration Services has come of age and is ready for data mining. The new data flow environment is equipped with dozens of building blocks (called transforms) that can help prepare data for data mining. These transforms perform such tasks as sampling, fuzzy grouping/lookup, split/merge/join, adding a derived column, sorting, as well as extraction and loading between data sources. Among the transforms, term extraction and term lookup are the two that I'll be using extensively in this article to prepare data for mining based on text such as Web feedback.

Preparing Data for Manual Classification

To perform manual classification, a group of domain experts must analyze feedback (data) and assign it to categories manually. SQL Server 2005 data mining techniques can then be applied to the feedback to obtain deeper insights from all those comments. In order to perform SQL Server 2005 data mining on the text, you first need to transform the text into term vectors. Producing term vectors from feedback involves two steps: building a dictionary and generating term vectors. In a term vector, text is represented as a set of terms and their frequencies. The term vector preparation can best be performed through an Integration Services project inside Business Intelligence Development Studio.

In order to identify terms from the text so that you can analyze the feedback, you need a dictionary of meaningful terms (words and phrases) for the problem domain. You can build a dictionary based on existing Web feedback by using the Term Extraction transform. This transform parses and understands language-specific grammar, and extracts words and phrases that are potentially representative. The user can control the behavior of the transform by specifying minimum frequency, maximum length of a term (in characters) and optionally supplying user-defined stop words to ignore certain terms.

Although the same dictionary can be reused throughout the classification process, it is certainly possible that the dictionary will lose some relevance over time as new issues are introduced in the Web feedback. Therefore the dictionary must be rebuilt accordingly. This rebuild process can be as simple as rerunning the Integration Services package against a new set of Web feedback.

The term vector table generated from the dictionary terms contains tuples of <CaseKey, Term, Frequency>. CaseKey is the unique identifier of the feedback, and it will be used to pivot terms for each feedback item. The Frequency column contains the number of occurrences of each term in the customer feedback.

The term vector table can be populated by using the Term Lookup transform. An Integration Services package can feed the Description fields from the WebFeedback table into the Term Lookup transform to get back the aforementioned tuples of <CaseKey, Term, Frequency> and then save them to the TermVectors table. The Term Lookup transform is configured to use the dictionary created in the previous step to identify terms.

Mining Models Development

Mining models can best be developed through an Analysis Services project inside Business Intelligence Development Studio. The process involves creating a data source, a data source view from the data source, and a mining structure that may contain several mining models (for information on creating mining models programmatically, see Liu Tang and Paul Bradley's article on the subject in this issue of MSDN®Magazine).

In order to build data mining models, you need to identify and model the case. A case is an entity from which you want to derive patterns. In this example it's some Web feedack. In SQL Server 2005 data mining, a case is represented by a set of columns similar to those in any relational table. Unlike a relational table, however, a case can also have a nested table column to model one-to-many relationships. For instance, a shopping basket can be a case with a nested table column containing a list of products purchased. Text from Web feedback, such as the case in this example, can be viewed as a set of terms. That is, you can use a nested table column to model the text of the feedback. Developing mining models in SQL Server 2005 typically takes at least three steps: creating a mining structure, defining multiple mining models inside the mining structure, and processing the models. A mining structure not only provides a structural definition of cases being modeled, but is also a container of multiple models that share the same cases. A mining structure can be created based on tables defined in a data source view.

Figure 2 Data Source View
Figure 2 Data Source View

A data source view in an Analysis Services project provides a high-level view of tables and views from a data source. Typically, a data source view is used to explore relationships among the tables and views and perform simple exploratory analysis such as obtaining global distribution, pivoting, and so on. A high-level object such as a cube or a mining structure can be created on top of tables and views from a data source view. Figure 2 shows a data source view created for the data mining of manual classification of Web feedback. It contains only the TermVectors table prepared in the previous step because you'll be using it as both a case-level and nested-level table. Note that this is possible only when there is no case-level column other than the key. You will see a data source view that contains separate tables for case level and nested level in the automatic classification task, which I'll explore later.

The first screen in Figure 3 shows a dialog from the Data Mining Wizard where you select TermVectors as both the case table and the nested table. In the next dialog, the Wizard lets you define a mining structure by selecting the usage type of each column. As shown in the second screen in Figure 3, CaseKey of the TermVectors table is being used as the key of the case and Term as the key of a nested column (Term is unique in the rows pivoted by CaseKey from the TermVectors table). For the sake of simplicity, the mining structure doesn't include the Frequency column .

Figure 3 Data Mining Wizard Dialogs
Figure 3 Data Mining Wizard Dialogs

After you're finished with the Wizard, you can continue to add more candidate mining models to the mining structure. The SQL Server 2005 data mining component provides two unsupervised algorithms to help with manual text classification: Microsoft_Clustering and Microsoft_Association_Rules. Figure 4 shows the two models created inside the mining structure. Note that the first model (WFeedback_CL) was created as part of the Data Mining Wizard, and the second model (WFeedback_AR) was added to the mining structure later.

Figure 4 Data Mining Models for Manual Classification of Web Feedback
Figure 4 Data Mining Models for Manual Classification of Web Feedback

Deploying the project will create and process the mining structure in the specified Analysis Services server. Note that the processing time is highly dependent on the complexity of the models, as defined by the number of pieces of feedback, the number of distinct terms, and the set of parameters used in the models.

Using and Refining the Mining Models

The Microsoft_Clustering model, WFeedback_CL, helps the reviewer who is performing the manual classification to automatically identify clusters (groups) of cases that are similar to each other based on the terms. The clusters are often good candidates for categories. Using the cluster viewer, the reviewer can analyze how each cluster is different from the others based on the terms used. For instance, Figure 5 shows discrimination information for Cluster 2 and other clusters. While terms like SCSI, motherboard, monitor, controller, and hard drive favor Cluster 2, terms like image and expo favor other clusters. You can apply similar analysis to other clusters. Using this technique, you or the reviewer may decide that Cluster 2 is a category under hardware issues (or create a new one if necessary) and then assign the cases in the cluster to that category.

Figure 5 Cluster Viewer for WFeedback_CL
Figure 5 Cluster Viewer for WFeedback_CL

Assigning the cases in a cluster to a category can be accomplished in three steps. First enable the DRILLTHROUGH option of the model using the property editor, and reprocess the model. Second, obtain CaseKeys of the feedback in the cluster using the following DMX (the SQL Server data mining language):

SELECT CaseKey FROM WFeedback_CL.CASES WHERE IsInNode('Cluster 2')

Third, update the category column of the feedback selected using the SQL statement shown in the following code:

   UPDATE WebFeedback SET Category =        'hardware issues'
   WHERE CaseKey IN (<list of CaseKeys selected in
       the previous step>)

The last two steps can be performed in a single SQL statement by using OPENQUERY after setting a linked server to the Analysis Services server. Refer to SQL Server Books Online for detailed information on how to set up a linked server.

Figure 6 WFeedback_AR in the Dependency Network Viewer
Figure 6 WFeedback_AR in the Dependency Network Viewer

The Microsoft_Association_Rules model, WFeedback_AR, can be used to analyze association among the terms. Strong association among the terms often indicates a distinctive subject in the feedback. For instance, Figure 6 is a portion of the dependency network of the model, WFeedback_AR. It shows groups of terms that appear together in the same feedback with a certain degree of confidence. From the groups, the reviewer may learn that there are cases about security issues, hardware issues, and My MSN access issues. The reviewer may decide to make each group a separate category. He may also want to scope down to only the cases that contain the terms in a group in the dependency network. You can easily identify the cases with certain terms using the SQL command shown in the following code snippet:

SELECT * FROM WebFeedback wf, TermVectors tv
WHERE wf.CaseKey = tv.CaseKey AND tv.Term IN
   (<list of terms>)

Getting the Best Result

Often the initial models don't yield the best result. This is because the default parameters for the models may not be the best for the particular data set. A repetitive refinement process is almost always required to produce the best model. The refinement process is a loop of multiple steps such as browsing the model content, adjusting model parameters, and reprocessing the model until you're able to obtain the most accurate model. Since the two models I have built in this section are unsupervised (rather than accuracy-driven), deciding which one is best is a subjective call. As shown in the previous section, a model that gives good insights on the feedback should be considered a good model. Each algorithm provides a different set of parameters to adjust. Figure 7 shows a partial list of parameters for Microsoft_Clustering and the Microsoft_Association_Rules algorithm.

Figure 7 Some Algorithm Parameters (Close Figure 7)

Microsoft_Clustering
Parameter | Definition
CLUSTER_COUNT | Specifies the maximum number of clusters to be built by the algorithm. If set to 0, the algorithm automatically determines the best number of clusters to build for predictive analysis; however, it may result in too many clusters or a cluster size that is too large for analysis.
MAXIMUM_INPUT_ATTRIBUTES | Limits the number of input attributes (such as, distinct terms) being considered to model. The default is 255, which may be too small for Web feedback classification. It may have to be adjusted to a larger number or to 0 to disable the limitation. Note, however, that increasing or removing this limitation will introduce a longer time to process the model.
Microsoft_Association_Rules
Parameter | Definition
MINIMUM_ITEM_SIZE and MAXIMUM_ITEM_SIZE | Specifies the minimum and maximum number of items allowed in an itemset. Be sure to set these parameters to include itemsets with size 1 and size 2 since a dependency network is generated from such itemsets.
MINIMUM_SUPPORT and MAXIMUM_SUPPORT | Specifies the minimum and maximum number of cases that the itemset must contain before generating a rule. Setting this window too high will result in too few rules while setting it too low will result in too many rules, longer processing time, and more memory consumption.
MINIMUM_PROBABILITY and MAXIMUM_PROBABILITY | Specifies the minimum and maximum probability that a particular rule is true. Too high a probability often indicates obvious rules, while too low a probability means there are too many trivial rules.

Data mining technology can also be used to automate the text classification process completely. To start you need a set of Web feedback that has already been categorized manually. A data model can be built on top of such feedback to be used to find any patterns between texts as well as demographic information of feedback and the categories. The patterns can then be applied in order to classify new feedback automatically.

Data preparation for automatic classification is similar to that for manual classification except that it requires another step—sample preparation. That is, in addition to the preparation of dictionary and term vectors, you also need to prepare multiple samples for different purposes in supervised modeling (such as prediction-driven modeling). Note that you do not need the entire feedback to train a model. A relatively small random sample is good enough as long as it contains all significant patterns that could be found in the feedback as a whole. Although there is no strict rule for determining the right size of the sample, a sample size of 5,000 cases is sufficient in many text classification applications. To figure out the right size of the sample, you can also begin increasing the sample size until the accuracy ceases to improve.

As in the manual classification, you can stage the Web feedback to a SQL Server database using the Integration Services Import/Export wizards, or a manually created Integration Services package that loads the whole feedback, performs sampling, and saves the sample to a staged database.

You'll need to have multiple samples of cases to use for different purposes throughout the data mining process. Typically, they should include train and test sample cases. Train sample cases are used to train and refine models. Test sample cases are typically set aside to be used to test and validate models. It is strongly recommended that test sample cases don't overlap with train sample cases. Otherwise, the model tends to learn too much detail that may be specific only to the train sample cases. This is called overfitting.

Figure 8 Integration Services Package
Figure 8 Integration Services Package

Note that sampling should be applied on cases rather than terms. Sampling on cases wouldn't change patterns as long as the same or similar patterns appear in other cases sampled. Sampling on terms, however, effectively changes the content (behavior) of the feedback. Figure 8 shows an Integration Services package used to create the three samples. Seventy percent of total cases collected at the initial data staging step are used for training; the remaining thirty percent are for testing.

Figure 9 Mining Data Source View
Figure 9 Mining Data Source View

Figure 9 shows a data source view created for data mining for automatic classification of Web feedback. The TermVectors table is the same as the one shown in the manual classification section except that it is now generated from the selected samples. Each sample consists of a key (CaseKey), a text (Description), and a category (Category). The data source view contains the two samples (TrainSamples and TestSamples, as you see in the figure).

Defining the Model

Unlike the mining structure in the manual classification section where the TermVectors table was used for both case and nested tables, I use TrainSamples as the case table and TermVectors as the nested table (see the first screen in Figure 10) because now you will use a non-key column (Category) from the case table. In the next step, the Data Mining Wizard presents a dialog where you define a mining model inside the structure by selecting the usage type for each column. As shown in the second screen in Figure 10, you are using CaseKey as the key of the case, TermVectors as input, and Category as output (here it's predictable—the column that you will predict from the model).

Figure 10 Data Mining Wizard Dialogs
Figure 10 Data Mining Wizard Dialogs

After you are finished with the Wizard, you can continue to add more candidate mining models to the mining structure. The SQL Server 2005 data mining component provides multiple algorithms to address automatic text classification problems. The algorithm that yields the best accuracy depends on the data set. In general, the following four algorithms should be considered as candidates for the best model: Microsoft_Neural_Network, Microsoft_Naive_Bayes, Microsoft_Decision_Trees, and Microsoft_Clustering.

Figure 11 shows the candidate models created inside the mining structure. Deploying the project will create and process the mining structure on the specified Analysis Services server. Note that the processing time is highly dependent on the complexity of the models, which is determined by the number of pieces of feedback and the number of distinct terms.

Figure 11 Candidate Models for Automatic Classification (Close Figure 11)

Testing and Further Refinement

As I noted earlier, to get the most accurate model, you may have to continually refine the results. Figure 12 presents a partial list of parameters for each algorithm that may need to be adjusted for the Web feedback classification task.

Figure 12 Parameters to Tweak for Better Accuracy (Close Figure 12)

Microsoft_Neural_Network
Parameter | Definition
HIDDEN_NODE_RATIO | Controls the number of nodes in the hidden layer of the network. Setting this parameter to 0 makes the model behave as a pure logistic regression algorithm, which tends to produce good accuracy for the text classification task.
MAXIMUM_INPUT_ATTRIBUTES | Limits the number of input attributes (such as distinct terms) being considered to model. The default is 255, which may be too small for Web feedback classification. It may have to be adjusted to a larger number or to 0 to disable the limitation. Increasing or removing this limitation, however, will introduce a longer time to process the model.
Microsoft_Naive_Bayes
Parameter | Definition
MAXIMUM_INPUT_ATTRIBUTES | Is the same as in Microsoft_Neural_Network.
Microsoft_Decision_Trees
Parameter | Definition
COMPLEXITY_PENALTY | Controls the complexity (depth) of the tree produced by the model by specifying a number between 0.0 and 1.0. This parameter may need to be adjusted to a lower penalty (in other words, to learn more fine-grained patterns) as patterns between the category and the terms in Web feedback tend to be more fine-grained than those in other non-text data mining applications.
MINIMUM_LEAF_CASES | Indicates the minimum number of cases that a leaf node in the tree has to contain. For the same reason described above, it may need to be changed to a smaller number than the default.
MAXIMUM_INPUT_ATTRIBUTES | Is the same as in Microsoft_Neural_Network.
Microsoft_Clustering
Parameter | Definition
MAXIMUM_INPUT_ATTRIBUTES | Is the same as in Microsoft_Neural_Network.

Measuring the accuracy of models can be achieved by using the test samples of Web feedback that were set aside in the data preparation step. SQL Server 2005 provides a lift chart for accuracy measurement. Figure 13 shows a dialog where the TestSamples table (for case) and the TermVectors table (for nested column) are used to produce a lift chart. (Lift is the measure of the accuracy of a predictive model.) Figure 14 shows the lift chart for all the four models developed in the previous step by using the test samples. The lift chart is created as outlined in the following paragraph.

Figure 13 Text Samples and Their Term Vectors
Figure 13 Text Samples and Their Term Vectors

The data mining engine feeds each test sample to each mining model as input and obtains a prediction value. The model not only predicts the most likely Category, but also returns a probability to indicate how confident that prediction is. The higher the probability the more likely the prediction is correct (that is, the prediction result is the same as the one in the test sample).

Figure 14 Lift Chart Showing Accuracy of Models
Figure 14 Lift Chart Showing Accuracy of Models

Once all prediction results and their probabilities are obtained, they are sorted in order of decreasing probability. The graph is created to show the percentage of the correct predictions as the population increases towards 100 percent of the prediction results.

An ideal model would predict everything correctly. That is, the percentage of correct predictions is the same as the percentage of predictions covered at any point. On the other hand, a random guess model would always show 1/N of the population, where N is the number of distinct categories because the chance of a random prediction being correct is 1/N. Obviously, a model with a graph closer to the ideal model is better. Figure 14 shows that WFeedback_NN outperforms the other three models.

Deploying the Model

SQL Server 2005 data mining employs a client-server architecture in which all data mining models are hosted on the server. Deploying a model is as simple as placing the model on the server. It could even be done by developing the model on the deployment server directly. The model can also be migrated easily to a different server simply by using the DMX IMPORT and EXPORT commands.

Applications that perform Web feedback classification can be developed in two different ways. An operational classification system can set up a DTS package that produces a term vector for a new piece of Web feedback using a Term Lookup transform. The term vector can be saved in a temporary file and used in a DMX query that performs prediction for the category. Alternatively, a batch classification system can be implemented using the DTS package, where a Term Lookup transform produces term vectors for the new Web feedback and a Data Mining Prediction transform produces categories for each piece of feedback.

Conclusion

Organizations can analyze Web feedback to help make business decisions and reach business goals such as increasing customer satisfaction and enhancing the quality of their products and services. The use of the techniques I've described is not limited to the text classification problems I've outlined; they can be applied to everything from document classification and patent categorization to memo grouping. The SQL Server 2005 data mining component is ready for prime time. In particular, the two Integration Services transforms (Term Extraction, Term Lookup) are very useful for producing term vectors from text. Multiple data mining algorithms combined with the nested table column functionality and lift chart from Analysis Services are the tools you need to develop the best models for making sense of your text-based data.

Peter Pyungchul Kim, Ph.D. is a developer in SQL Server Data Mining team at Microsoft. He is a primary developer of the data mining language, DMX, and has architected several data-mining algorithms such as decision trees, Naïve Bayesian, and neural network in SQL Server 2000 and SQL Server 2005.


Mine Your Business: AMO Lets You Dig Deeper into Your Data from Your Own Applications

Mine Your Business

AMO Lets You Dig Deeper into Your Data from Your Own Applications

Liu Tang and Paul Bradley
This article is based on a prerelease version of SQL Server 2005. All information contained herein is subject to change.
This article discusses:
  • Business Intelligence Development Studio
  • Creating and processing mining models programmatically with AMO
  • Using AMO to manage Analysis Services objects
This article uses the following technologies:
SQL Server 2005
Contents
Benefits of AMO
What's in it for Developers?
Starting the Project
Create an Analysis Database
Define a Data Source
Define a Data Source View and Create a View Object
Build a Mining Structure
Create a Mining Model
Train the Mining Model
Secure What You Have Built
Replicate a Mining Model by Cloning
Replicate a Mining Model by Clicking
View Results
Conclusion

R
emember the day you got your first big box of crayons—that colossal, multitiered package with an extraordinary number of colors? You needed every color to express your ideas more fully, down to every nuanced detail. For client-side developers, Analysis Management Objects (AMO) for SQL Server™ 2005 is a lot like that big box of crayons. Until now, lack of basic tools has left many developers unable put their own ideas into action and to answer fundamental data mining questions. Developers have found it challenging to derive and draw the vivid pictures that data mining delivers (such as market trends, key prospect profiles, and how customers define satisfaction) without bringing in extra resources.

Benefits of AMO

AMO lets developers put their own programs into action. It facilitates client-side custom programming as Decision Support Objects (DSO), the object model in SQL Server 2000 Analysis Services. With AMO, a whole range of data mining questions can be answered at the operational level. This means that sales and marketing departments get answers more quickly and thus can make informed decisions. Specialized resources, like the IT team and analysts, can be brought in when they're needed most.

AMO puts data to work on basic business problems. Are retail sales eating into catalog sales? Do inventory patterns validate the new merchandising strategy? With AMO, business questions can be answered easily and directly.

Suppose you are a marketing manager at a bicycle manufacturing firm in the upper Midwest, looking for new income streams in the winter months. You have a hunch that your customers might be good prospects for local businesses looking for leads. You want to see if the data makes a case for your idea and get the numbers you need to forecast how much income you could generate by renting the customer list to real estate agents, one of three target segments you've identified. We'll use this example later on in the article to show just how AMO works.

What's in it for Developers?

AMO provides an object model to program against and a varied object set to work with, including servers, databases, mining structures, and models, as well as online analytical processing (OLAP) objects. When you use AMO, you can directly manipulate a whole range of data mining objects in order to seamlessly embed data mining functionality into your own applications.

AMO provides developers with access to seven high-powered mining algorithms, the same algorithms that can be accessed from two GUI applications included with SQL Server 2005: the Business Intelligence Development Studio and the SQL Server Management Studio. Both development workspaces install automatically when you choose either a full setup or the client component of SQL Server. These algorithms are named as follows: Microsoft® Association Rules, Microsoft Clustering, Microsoft Decision Trees, Microsoft Naïve Bayes, Microsoft Neural Network, Microsoft Sequence Clustering, and Microsoft Time Series.

Each algorithm is designed to detect telltale patterns hidden in your data. Using them independently or in combination, you can answer a wide range of practical questions. For example, using the algorithms, you can perform a market share analysis to help evaluate the performance of a specific trading area. You can determine product affinity across a market basket of products for guidance in structuring a cross-sell campaign or to develop list selection criteria for targeted mailings.

Figure 1 Data Mining Using AMO
Figure 1 Data Mining Using AMO

The Business Intelligence Development Studio is a working platform that lets you visualize the data mining process in a way that's comfortable whether you're a developer or database administrator. As you can see in Figure 1, you can go deep into the data with full access to the entire data mining process. You can manipulate the data directly to create mining models, view mining results, and predict missed values for new data. Building a mining model from the user interface can be achieved programmatically using AMO, and follows exactly the same procedure as in the user interface.

SQL Server Management Studio is an administrative platform that lets you leverage existing mining models and manage databases. Here, you can browse mining models and make predictions based on them, or back up and restore databases.

SQL Server 2005 comes with a practice database called AdventureWorksDW to help you find your way around AMO. AdventureWorksDW gives you access to sales data from the fictional bicycle manufacturer referred to earlier in this article.

Let's go through a step-by-step sample exercise using AdventureWorksDW. We're going to find out how many customers are homeowners and where those customers are from. To recap our example, you are a marketing manager for a bicycle manufacturer exploring the possibility of renting the customer list to local real estate agents and other businesses. You want to profile the customer base for home ownership.

Starting the Project

Using AdventureWorksDW, you'll work with data found in the DimCustomer table that is shown in Figure 2. This table summarizes demographic information, including home ownership, car ownership, marital status, number of cars owned, and annual income for more than 10,000 customers.

Figure 2 DimCustomer Table
Figure 2 DimCustomer Table

To start writing code with AMO, all you need are references to the System.Data.dll and Microsoft.AnalysisServices.dll assemblies. By referencing the assembly Microsoft.AnalysisServices (installed as part of client component or full installation setup), you import all object definitions required by the Analysis Server. System.Data (installed as part of the Microsoft .NET Framework 2.0) is needed for data manipulations for OLE DB, data sets, and other data-related operations you may perform.

Next, so you won't need to use the namespace-qualified typename each time it's used in the code, reference a few core namespaces from these assemblies in imports directives, as shown in the following code snippet:

Imports System.Data
Imports System.Data.OleDb
Imports Microsoft.AnalysisServices

Create an Analysis Database

The analysis database contains all other objects. To create one, make sure the server is already started. Then define an analysis Server object and establish a connection to the server:

Dim srv As New Server

srv.Connect("servername")
This code tells the database object which server the database will reside on. Multiple connection sessions are supported by the server to maintain AMO functionality. To specify a session ID, pass an ID string to the Connect method as the second parameter (omitted in this example for simplicity).

Next, a database object can be initialized and added to a collection of databases, which are connected to the server. During the initialization, the system will assign a unique database ID through the Utils class, an AMO class that manages the binding of object IDs and object names and frees developers from having to manage object IDs. The mapping of database names and IDs is automatic. Figure 3 shows how to initialize a Database object and add it to the Server object. The Utils class can also be used to serialize and deserialize the major server objects (database, data source, mining structure, mining model and others), though only a few are included in this exercise.

Figure 3 Creating a Database Object (Close Figure 3)
Function CreateDatabase( _
        ByVal srv As Server, ByVal databaseName As String) As Database

    ' Check to see if the database already exists
    If srv.Databases.ContainsName(databaseName) Then
       Throw New ArgumentOutOfRangeException("databaseName", _
           "Database already exists in the server.")
    End If

    ' Create the new database object
    Dim dbNew As Database = New Database(databaseName, _
        Utils.GetSyntacticallyValidID(databaseName, GetType(Database)))

     ' Add the new database object to the server's collection
    srv.Databases.Add(dbNew)
    dbNew.Update()

    ' Return a reference to the database
    Return dbNew
End Function

As elsewhere, unique names are important. Before adding a new database object to the server, make sure no existing database has the same name, or the old one will be overwritten.

The newly created database object must be added to the database collection on the server. The server objects are organized in the form of a hierarchical tree, as shown in Figure 4. To add a new object, link the isolated (child) node to its corresponding parent node, calling the Add method. Child nodes are referred to as a collection of their node type or server object type in AMO.

Figure 4 Server Object Hierarchy
Figure 4 Server Object Hierarchy

Multiple database objects are stored on the server object and each database can contain multiple data source objects, as well as other objects, at different hierarchical levels. There are many object collections in AMO, only some of which are used in our examples (identified in the example code by an Add method). For major server objects, changes made through AMO won't materialize on the server until you call the appropriate Update method.

Define a Data Source

A data source tells the program where the data comes from (AdventureWorksDW, in this case) and establishes a connection to a source database. Figure 5 builds a data source object.

Figure 5 Building and Naming a Data Source Object (Close Figure 5)
Function CreateDatasource( _
        ByVal db As Database, ByVal datasourceName As String, _
        ByVal connectionString As String) As DataSource

    ' Create a new data source
    Dim dsNew As RelationalDataSource = _
        New RelationalDataSource(datasourceName, _
            Utils.GetSyntacticallyValidID(datasourceName, _
                GetType(RelationalDataSource)))

    ' Add it to the database's data sources
    db.DataSources.Add(dsNew)
    dsNew.ConnectionString = "SQLOLEDB.1;Data Source=sourcename;" & _
        "Integrated Security=SSPI;Initial Catalog=AdventureWorksDW"
    dsNew.Update()

    ' Return the new data source
    Return dsNew
End Function

Two types of data source objects, RelationalDataSource and OlapDataSource, are used to connect to databases. We create a RelationalDataSource object when connecting to a relational database. As was the case for the database object in Figure 3, the mapping of data source names and IDs is provided by AMO.

Define a Data Source View and Create a View Object

Data source view functions map between the database data and your view of the data. In a data source view, you can select only those database tables needed for the model, or create new tables by joining or cutting, which is performed by a named query in a GUI. In the data source view you can further refine data before initiating the actual mining process. (Note that the sample code in this article shows only the basic functionality for defining a data source view. There are many more possibilities in AMO.)

Now that you have both a database object and a data source object, you can create a DataSourceView object—a slightly more complex task than those completed so far. As before, a data source ID must be established. In addition, the data source view must be populated with the schema information from the table.

As shown in Figure 6, you first create an instance of DataSourceView and add it to the database, setting its DataSourceID to match the ID of the relevant DataSource. Next, you load the schema information from the selected table into the data source view. The schema information is then inserted into the DataSourceView.

Figure 6 Creating a Data Source View (Close Figure 6)
Function CreateDatasourceView( _
        ByVal db As Database, ByVal ds As DataSource, _
        ByVal datasourceviewName As String, ByVal tableName As String) _
        As DataSourceView

    Dim rdsv As DataSourceView =
        db.DataSourceViews.Add(datasourceviewName, _
            Utils.GetSyntacticallyValidID(datasourceviewName, _
                GetType(DataSourceView)))
    rdsv.DataSourceID = ds.ID

    Dim cn As OleDbConnection = New OleDbConnection(ds.ConnectionString)
    Dim cmd As OleDbCommand = New OleDbCommand(tableName, cn)
    cmd.CommandType = CommandType.TableDirect;

    Dim dss As New DataSet
    Dim ad As OleDbDataAdapter = New OleDbDataAdapter(cmd)
    ad.FillSchema(dss, SchemaType.Source)

    ' Make sure we have the correct name
    dss.Tables(0).TableName = tableName

    ' We need to clone here because the original DataTable already
    ' belongs to a DataSet
    rdsv.Schema = New DataSet
    rdsv.Schema.Tables.Add(dss.Tables(tableName).Clone())
    rdsv.Update()

    Return rdsv
End Function

Build a Mining Structure

A mining structure organizes mining data, defining the data columns that one or more mining models will draw from. Because algorithms uncover specific patterns or relationships, multiple models are often required to achieve a complete solution to any one problem. This allows you to solve the problem in different ways and, by comparing results, find the best solution.

It makes sense, then, to think of a mining structure as an entity that contains multiple mining models (all of which share the same column structure and column content type), detailed information about the nature of the data the model will use, and information about the relationships between or among columns.

Remember, we want to look at home ownership patterns for the customer base as a whole and for segments of the whole. This means we need to build a mining structure with NumberCarsOwned, MaritalStatus, YearlyIncome, and HouseOwnerFlag columns, plus a CustomerKey column.An Overview of ADOMD

To facilitate programming against SQL Server 2005 data mining, Analysis Services supports two objects models in addition to the AMO object model: ADOMD Server and ADOMD .NET. Like AMO, they are object models for both data mining and OLAP.

ADOMD Server is a server-side object model residing inside the Analysis Services server. With it, you can send queries to the server without having to set up a connection with the server, which makes it easy to navigate the server objects and retrieve information from them directly. You get all the data mining objects you need, from mining model column attributes to mining model contents. Plus, you don't need to flush big mining model contents into the client side before doing complicated queries and computations. ADOMD Server isolates server tasks from client tasks gracefully, letting the server do its job and reserving the complex manipulation of queries for the client-side programmer.

ADOMD .NET is a client-side query tool that complements ADOMD Server. It has nearly the same set of objects and object hierarchy as ADOMD Server, the main difference being that programming with ADOMD .NET requires a connection to the server before sending queries or navigating the server objects. All the query results are dumped to the client when the server responds.

The code that follows shows how to initialize a mining structure object, bind it to its data source view, and set its case table as DimCustomer. For the purpose of binding, the CaseTableName has to be the same as the table name in data source viewer.

Dim ms As MiningStructure = db.MiningStructures.Add(miningstructureName, _
    Utils.GetSyntacticallyValidID(miningstructureName, _
        GetType(MiningStructure)))
ms.Source = New DataSourceViewBinding(dsv.ID)
ms.CaseTableName = "DimCustomer"   'binding to data source table name

When you create a data column, you must set IsKey and Content properties. They determine how the data will be interpreted by the data mining engine during the model training. IsKey specifies that the column will be treated as a key column. Content specifies the content type: DISCRETE (of distinct categorical value applicable to all data) or CONTINUOUS (of continuous value applicable only to numeric data). Numeric data can be either DISCRETE or CONTINUOUS. For example, you could have an Age column where 1 means "less than 25" or where 1 means the value 1. After setting IsKey and Content, the data column is added to a collection of mining structure columns.

The code in Figure 7 shows how columns are created and added to the mining structure called ms. There are two kinds of mining structure columns, ScalarMiningStructureColumn and TableMiningStructureColumn. ScalarMiningStructureColumn is used for all regular, non-table data columns, and TableMiningStructureColumn is used for creating nested tables. In this example, all the columns are non-table, so all are initialized as ScalarMiningStructureColumn objects and added to the mining structure in the same way. Finally, calling the Update method will ensure that the columns are actually created in the mining structure on the server.

Figure 7 Creating and Adding Columns (Close Figure 7)
Function CreateMiningStructure( _
        ByVal db As Database, ByVal dsv As DataSourceView, _
        ByVal miningstructureName As String) As MiningStructure

    ' Create the mining structure
    Dim ms As MiningStructure = db.MiningStructures.Add( _
        miningstructureName, _
        Utils.GetSyntacticallyValidID(miningstructureName, _
            GetType(MiningStructure)))
    ms.Source = New DataSourceViewBinding(dsv.ID)
    ms.CaseTableName = "DimCustomer" ' binding to data source table name

    ' CustomerKey
    Dim smsc As ScalarMiningStructureColumn = _
        New ScalarMiningStructureColumn("CustomerKey", _
            Utils.GetSyntacticallyValidID("CustomerKey", _
                GetType(ScalarMiningStructureColumn)))
    smsc.IsKey = True
    smsc.Content = "Key"
    smsc.KeyColumns.Add("DimCustomer", "CustomerKey", OleDbType.Integer)
    ms.Columns.Add(smsc)

    ' HouseOwnerFlag
    smsc = New ScalarMiningStructureColumn("HouseOwnerFlag", _
        Utils.GetSyntacticallyValidID("HouseOwnerFlag", _
            GetType(ScalarMiningStructureColumn)))
    smsc.IsKey = False
    smsc.Content = "Discrete"
    smsc.KeyColumns.Add("DimCustomer", "HouseOwnerFlag", OleDbType.WChar)
    ms.Columns.Add(smsc)

    ... 'for MaritalStatus and NumberCarsOwned

    ' YearlyIncome
    smsc = New ScalarMiningStructureColumn("YearlyIncome", _
        Utils.GetSyntacticallyValidID("YearlyIncome", _
            GetType(ScalarMiningStructureColumn)))
    smsc.IsKey = False
    smsc.Content = "Continuous"
    smsc.KeyColumns.Add("DimCustomer", "YearlyIncome", OleDbType.Integer)
    ms.Columns.Add(smsc)

    ms.Update()

    Return ms
End Function

Create a Mining Model

As noted, a mining model is contained inside a mining structure, which may have more than one model, each using different algorithms and parameter settings. We will first initialize a mining model object and add it to a mining structure, then specify the algorithm and parameter. Because each algorithm can have multiple algorithm parameters, AlgorithmParameters is a hash table collection that maps each parameter name with its corresponding value. Parameter settings are algorithm-specific. That is, each algorithm has its own set of parameters, which affects how patterns are generated by the algorithm. Details of parameter settings can be found in SQL Server Books Online. The following code is what we have so far:

Dim mm As MiningModel = ms.MiningModels.Add(miningmodelName, _
    Utils.GetSyntacticallyValidID(miningmodelName, GetType(MiningModel)))
mm.Algorithm = "Microsoft_Decision_Trees"
mm.AlgorithmParameters.Add("COMPLEXITY_PENALTY", 0.3)
Each model column needs to bind with its corresponding mining structure column, by column ID. The binding is shown in Figure 8, in the second line of each model column definition.

Figure 8 Creating the Mining Model Columns (Close Figure 8)
Function CreateMiningModel( _
        ByVal ms As MiningStructure, ByVal miningmodelName As String) _
        As MiningModel

    Dim mm As MiningModel = ms.MiningModels.Add(miningmodelName, _
       Utils.GetSyntacticallyValidID(miningmodelName,
           GetType(MiningModel)))
   mm.Algorithm = "Microsoft_Decision_Trees"
    mm.AlgorithmParameters.Add("COMPLEXITY_PENALTY", 0.3)

    Dim mc As MiningModelColumn
    mc = New MiningModelColumn("CustomerKey", _
        Utils.GetSyntacticallyValidID("CustomerKey", _
            GetType(MiningModelColumn)))
    mc.SourceColumnID = ms.Columns("CustomerKey").ID
    mc.Usage = "Key"
    mm.Columns.Add(mc)

    mc = New MiningModelColumn("HouseOwnerFlag", _
        Utils.GetSyntacticallyValidID("HouseOwnerFlag", _
            GetType(MiningModelColumn)))
    mc.SourceColumnID = ms.Columns("HouseOwnerFlag").ID
    mc.Usage = "PredictOnly"
    mm.Columns.Add(mc)

    mc = New MiningModelColumn("MaritalStatus", _
        Utils.GetSyntacticallyValidID("MaritalStatus", _
            GetType(MiningModelColumn)))
    mc.SourceColumnID = ms.Columns("MaritalStatus").ID
    mc.Usage = "Input"
    mm.Columns.Add(mc)

    mc = New MiningModelColumn("NumberCarsOwned", _
        Utils.GetSyntacticallyValidID("NumberCarsOwned", _
            GetType(MiningModelColumn)))
    mc.SourceColumnID = ms.Columns("NumberCarsOwned").ID
    mc.Usage = "Input"
    mm.Columns.Add(mc)

    mc = New MiningModelColumn("YearlyIncome", _
        Utils.GetSyntacticallyValidID("YearlyIncome", _
            GetType(MiningModelColumn)))
    mc.SourceColumnID = ms.Columns("YearlyIncome").ID
    mc.Usage = "Input"
    mm.Columns.Add(mc)

    mm.Update()

    Return mm
End Function

Data mining columns have several property types, the most important of which are data type, content type, and usage. Look back to Figure 7 and you'll see that you've already completed the first level of defining the data and content type of the columns. Now, you need to further define the usage of the columns.

In data mining with Analysis Services, there are four types of column usage: input, predict, predict only, or key. Only one key column can be set; it's a unique identifier for the rows feeding into the processing. The last update statement shown in Figure 8 sends the column metadata to the server.

Train the Mining Model

All necessary definitions for the mining model are complete. Before it can be of any use, though, you have to train the model (or process it) to find useful patterns in the data. Processing happens inside the server. By way of a complex computation that is dictated and run by the algorithm, data is pulled from the data source provided. Once processing is triggered, you simply wait for the results. The time needed for processing depends on the amount of data involved and the complexity of the problem. Before processing, the model has only metadata defined. After processing, the patterns that are found are stored in the model.

There are a number of process options, including ProcessFull, ProcessStructure, and ProcessDefault. Suppose you have a mining structure with two mining models. You can process both models in one batch by fully processing the mining structure and mining models together. Or you can first process just the mining structure, then the models individually. ProcessStructure forms metadata by collecting data from the source, then prepares the data according to its content type for future model use. The following code shows the second choice, but with only one model:

'Process mining structure first
ms.Process(ProcessType.ProcessStructure)

'Process the mining model only
mm.Process(ProcessType.ProcessDefault)

The ProcessDefault option guarantees that after running, any object will be fully processed. So we could have written ProcessDefault without first processing the structure.

AMO can also clear the data from the mining structure and mining model through ProcessClearStructureOnly and ProcessClear. After calling ProcessClearStructureOnly, the metadata stored in the mining structure is destroyed so that no new models can be added to the mining structure. However, existing trained mining models won't be affected: the patterns are preserved and you can still make predictions based on them. The ProcessClear option will clear everything from mining structure to mining model.

Secure What You Have Built

AMO offers three levels of access privilege to protect mining structures and mining models from being viewed or modified by unauthorized persons. They include None (lowest), Read, and ReadWrite (highest).

The security in SQL Server 2005 data mining is organized by roles. Role objects can be defined at either the server or database level. Access privilege at the server level applies to all server objects. At the database level, only objects confined to the database are restricted. All access privileges are bound to roles and can be applied to database objects and to many OLAP objects, not only to mining structures and mining models.

Figure 9 shows how to set access privileges for the mining structure and mining model with the role defined at the database level. Here, users of the myrole group cannot change the mining structure, but may modify the node caption of the mining model and make predictions based on the model.

Figure 9 Setting Access Privileges (Close Figure 9)
' Create new role;
Dim role As Role = db.Roles.Add("myrole");
role.Members.Add("someone");
role.Update();

Dim msp As MiningStructurePermission
Dim mmp As MiningModelPermission

msp = ms.MiningStructurePermissions.Add(role.ID);
msp.Access = Access.Read;
msp.Update();

mmp = mm.MiningModelPermissions.Add(role.ID);
mmp.Access = Access.ReadWrite;
mmp.Update();

Replicate a Mining Model by Cloning

In the Business Intelligence Development Studio, only a few mouse clicks are required to create a new mining model based on an existing one. This can also be achieved programmatically by cloning, since the new model shares the same mining structure as the old. Figure 10 shows how.

Figure 10 Cloning (Close Figure 10)
Function AddNewMiningModelBasedOnExistingModel( _
        ByVal existingModel As MiningModel, _
        ByVal newMiningModelName As String, _
        ByVal newAlgorithmName As String) As MiningModel

    Dim derivedModel As MiningModel

    If existingModel Is Nothing Then
        Throw New ArgmentNullException("existingModel")
    Else
        derivedModel = existingModel.Clone()
        If existingModel.Algorithm <> newAlgorithmName Then
            derivedModel.AlgorithmParameters.Clear()
        End If
    End If

    derivedModel.Name = newMiningModelName
    derivedModel.ID = _
        existingModel.Parent.MiningModels.GetNewID(newMiningModelName)
    derivedModel.Algorithm = newAlgorithmName

    existingModel.Parent.MiningModels.Add(derivedModel)

    derivedModel.Update()

    Return derivedModel
End Function

In Figure 10, the MiningModel.Clone method is called to generate a derived model that is an exact copy of the existingModel (note that the derivedModel will not reside on the mining structure until Update is called). If the derivedModel uses a different algorithm, calling AlgorithmParameters.Clear is required in order to avoid incompatibility between the derivedModel algorithm and the parameter settings stored in existingModel.

To add derivedModel to the mining structure, an ID has to be assigned over and above the model and algorithm names. This is done using the MiningModels.GetNewID method. Be aware that the Parent attribute for existingModel is the mining structure shared by both existingModel and derivedModel. The derivedModel is registered with the mining structure once existingModel.MiningModels.Add(derivedModel) is executed

The Parent attribute shows the hierarchy of server objects. Each object corresponds to a node in the server object's tree. The attribute Parent gets the direct parent node of the current object. For programming convenience, cross-level parent attributes are available. A mining model might include an attribute pointing to its mining structure, as well as Database and Server attributes, in a format like ParentMiningStructure, ParentDatabase, and so on.

Replicate a Mining Model by Clicking

If you've tried out SQL Server 2005 Business Intelligence Development Studio, you know you can create a new mining model directly from an existing mining structure. Figure 11 shows what the AMO code supporting this function looks like. When you replicate this way as opposed to the method shown in Figure 10, the key difference is that MiningStructure.CreateMiningModel is called from the mining structure existingStructure, which creates a new mining model composed of all the data columns from existingStructure. Passing in the argument false indicates that the model shouldn't be added to the model collection.

Figure 11 Deriving from an Existing Mining Structure (Close Figure 11)
Function AddNewMiningModelBasedOnExistingStructure( _
        ByVal existingStructure As MiningStructure, _
        ByVal newMiningModelName As String, _
        ByVal newAlgorithmName As String) As MiningModel

    Dim derivedModel As MiningModel

    If existingStructure Is Nothing Then
        Throw New ArgumentNullException("existingStructure")
    Else
        derivedModel = existingStructure.CreateMiningModel(False)
    End If

    derivedModel.Name = newMiningModelName
    derivedModel.ID = _
        existingStructure.MiningModels.GetNewID(newMiningModelName)
    derivedModel.Algorithm = newAlgorithmName

    existingStructure.MiningModels.Add(derivedModel)

    derivedModel.Update()

    Return derivedModel
End Function

There is a difference between deriving models from existing mining structures and deriving models from existing mining models. The model derived from an existing structure has the same number of data columns as the structure; the one derived from the existing model has the same number of data columns as the model, which could be less than the structure.

View Results

After processing the mining model, you can view the outcome manually by going to SQL Server Management Studio. Right-click on the model in question and choose Browse from the context menu. You will see a decision tree of HouseOwnerFlag in Figure 12. The complete tree has six levels, though Figure 12 shows only three levels for the sake of brevity. Yellow represents customers who own a home (HouseOwnerFlag = 1), while red represents customers who rent or live with family (HouseOwnerFlag = 0).

Figure 12 Decision Tree (Close Figure 12)

Trained model result viewers, like the one shown in Figure 12, are redistributable, meaning you can view results by embedding them in your own custom client applications. In order to do so, simply write a program in ADOMD (either ADOMD Server or ADOMD .NET, depending on your access), where you can browse the results and perform predictions.

ADOMD is a query tool, perfect for manipulating the trained model results. In addition to directly sending queries to the server, ADOMD exposes the details of model attributes and content information piece by piece. It gives you an in-depth view of a whole range of data, including model column type, model content node, content distribution (nested in each row of content node), mining services, algorithm parameters, and much more. Without ADOMD, getting this information requires a fair bit of choreography, including querying the server through the OLE DB command, retrieving each data element one by one by splitting the data set, as well as some more complicated steps. For more information, see the sidebar "An Overview of ADOMD" and go to www.sqlserverdatamining.com, where you'll find a downloadable sample to integrate the viewer into client applications.

Whichever way you view the results, the marketing team will be happy to know that most customers on record are homeowners, including roughly half of the unmarried customers and half of the customers at nearly all income levels. The small number of homeowners with incomes above $122,200 is an unexpected finding worth looking into down the road. In the meantime, it looks like AdventureWorksDW has a sizeable list of customers that local real estate agents would be willing to rent.

Conclusion

AMO provides a complete set of objects for data mining and a great toolkit for developers who want to build, process, and manage data mining models. Fully compatible with the .NET environment, AMO lets developers embed data mining into their own applications easily and smoothly.

With SQL Server 2005 and AMO you can integrate business intelligence into applications directly and manage Analysis Services objects programmatically. Whatever your industry, AMO for data mining can add more definition to research capabilities.

Liu Tang is a Software Design Engineer/Test for the Data Mining team in SQL Server at Microsoft, where she has worked on the designing and testing of SQL Server Data Mining features for the past three years. She has a Master's degree from Simon Fraser University, Canada, majoring in data mining and databases.

Paul Bradley, Ph.D., is a Principal in Data Mining Technology at Apollo Data Technologies, LLC.


Make It Snappy: Juice Up Your App with the Power of Hyper-Threading

Make It Snappy

Juice Up Your App with the Power of Hyper-Threading

Yaniv Pessach
This article discusses:
  • The behavior of hyper-threaded CPUs
  • Performance on hyper-threaded computers
  • Optimizing your application for hyper-threading
  • CPU caches and affinity
  • Concurrency patterns suitable for hyper-threading
This article uses the following technologies:
C#, .NET Framework, Win32
Contents
The Importance of Hyper-Threading
The Window View
Client Versus Server
Being Hyper
Two Heads Are Faster Than One
Hit and Miss
Locking
False Sharing and Cache Aliasing
Ah, the Competition
To Serve and Hyper-Thread
CPU Affinity
Producer/Consumer
Detecting Hyper-Threading
Conclusion

H
yper-threading technology improves CPU efficiency by allowing the processor to execute two instruction streams concurrently. This feature, found on newer Intel Pentium 4 processors, can typically improve the performance of apps by 20 to 30 percent, boosting some apps by up to 40 percent.

Unfortunately, other applications see no performance benefit, and a few can experience significant performance degradation (I've seen application degradation of as much as 20 percent), especially when violating recommended performance best practices such as the ones discussed in this article. Moreover, given the concurrency this feature introduces to single-processor computers, some multithreading bugs that may only manifest themselves on multiprocessor machines are likely to show themselves on a hyper-threaded single-CPU machine, as well.

In this article, I'll explore the hyper-threading technology and demonstrate how adding parallelism to your code can improve performance on hyper-threaded machines. I'll cover advanced optimizations for hyper-threading and will show you a few useful patterns. My code samples are in C#, but the same principles apply to both managed and unmanaged applications.

The Importance of Hyper-Threading

Hyper-threading technology is available on newer Pentium 4 CPUs, often on computers with clock speeds of 2.4GHz and up, as well as on all Xeon CPUs with speeds of 2.2 GHz and up. This feature is enabled by default in most, but not all, machines that have the capability.

Before hyper-threading, the Pentium 4 had to execute one instruction stream at a time. The CPU maintained a single instruction pointer (IP), and all multitasking on a single-CPU machine relied on the operating system's ability to allocate timeslices of the CPU to threads. Hyper-threading enables the Pentium 4 to execute two instruction streams simultaneously; the CPU maintains two instruction pointers and machine states and invisibly switches execution between them.

When a machine instruction from a program is executed, one or more execution slots will be needed. An execution slot corresponds to an internal CPU resource, such as the floating point unit (FPU), and represents the CPU's ability to perform an operation, such as a floating point calculation.

The Pentium 4 CPU always knows which execution slots are available. Features such as Out-Of-Order Execution (where the CPU can try to execute instructions in a different order than they appear in the instruction stream) enhance performance by keeping more of the execution slots utilized, as opposed to waiting (known as stalling) until a slot becomes available.

By giving the CPU the freedom to switch between the two execution streams, more of the CPU's execution slots can be used. When one instruction stream is stalled on access to the main memory or needs an execution slot that is in use, the CPU can execute instructions from the second instruction stream while the memory for the first stream is being fetched or until the execution slot becomes available.

In a hyper-threaded CPU, two instruction streams are executed at the same time. In reality, the CPU executes a few instructions from the first instruction stream, then a few from the second instruction stream, switching based on an algorithm that tries to optimize execution slot availability and to minimize the impact of stalls while trying to distribute resources fairly between execution streams. Execution context (such as register values) is kept with the instruction stream. The net result of all of this is that a hyper-threaded CPU emulates two physical CPUs, each of which is referred to as a logical CPU.

Hyper-threading can result in better utilization of CPU resources and therefore better performance. The flip side is that those CPU resources are indeed shared, so when both streams of instructions require the same shared resource, performance can suffer.

In addition to shared resources such as the FPU and the arithmetic logic unit (ALU), space inside the CPU cache (divided into level 1 and level 2 caches, known as L1 and L2), which caches recently accessed RAM areas, is shared. The L2 cache space is not allocated to a logical CPU. Sharing the cache space can either help or hinder performance, but I'll discuss that a little later.

If there is no second execution stream, all cache space and other resources are made available to the one execution stream. As the multiple instruction stream mode carries a slight performance cost, the CPU would only use it if instructions are scheduled to execute on both logical CPUs. Also, on a computer with a single processor, enabling hyper-threading entails using the multiprocessor kernel, which handles synchronization better, but comes at a small performance cost.

In the ideal case, each instruction stream would execute almost as fast as it would if the CPU were running only one instruction stream. However, in reality, it's rare that two instructions streams could operate without frequently requiring access to shared resources, and as a result each instruction stream is noticeably slower than that ideal. If an application's work is split in two in an intelligent manner, however, hyper-threading can still result in improved overall application performance.

The Window View

Under recent versions of Windows®, each execution stream is exposed as a logical CPU, so the operating system schedules thread execution on each logical CPU. This means that on a hyper-threaded machine with one physical CPU, and thus two logical CPUs, when two threads are executing they execute at the same time, each on a separate logical CPU.

A user will see that the machine has two logical CPUs, and when two threads are executing, they would execute on both logical CPUs at the same time. I will use the terms "logical1" and "logical2" as a shorthand for the first and second logical CPU exposed by a specific physical CPU.

On Windows NT and later, the scheduler is hyper-threading-aware. When multiple threads are executing concurrently, they will be scheduled on both logical CPUs, rather than with alternating timeslices as is the case with a non-hyper-threaded single CPU. This can lead to noticeable performance improvements and increased responsiveness for hyper-threaded systems.

A dual CPU with hyper-threading is more complex; the operating system first schedules threads on separate physical CPUs before scheduling them on logical CPUs belonging to the same physical CPU. This means that with only two threads running, each thread would be assigned to its own processor rather than having them hyper-threaded onto the same CPU. That said, from this point forward I will focus the rest of the article on discussing single-CPU machines with hyper-threading enabled.

Client Versus Server

The key to optimizing applications for hyper-threading is ensuring that both logical CPUs are performing useful operations. To improve performance further, I'll discuss ways to optimize the division of labor between threads.

The discussion here is relevant both to client and server applications. In a server, sometimes the straightforward division of labor makes the most sense. For example, if you are developing an ASP.NET application or a Web service expecting many requests per second, the usual multithreaded design would ensure that multiple threads are executing at the same time. For this reason, existing server applications tend to show more of a performance increase when hyper-threading is enabled.

Applications that need speedy I/O (network and disk) access benefit from splitting operations into multiple threads or from using asynchronous programming models. For client or server applications, where each request is CPU-intensive, additional gains can be made by splitting work into multiple threads (this is contrary to applications running on a non-hyper-threaded single CPU computer, where splitting a CPU-intensive task up into multiple threads will yield little benefit and may actually hurt performance due to the overhead of context switching). Client applications where work is done in a separate thread from the UI thread (almost always a good idea for usability reasons) show even better responsiveness on hyper-threaded machines.

Being Hyper

Multithreaded applications are likely to show increased performance and responsiveness once hyper-threading is turned on. I'll discuss several approaches you can take to maximize performance. Each approach has downsides, however, so measuring your performance is necessary when evaluating whether an approach helps.

A good place to start understanding the potential performance gains of your code is by measuring execution time on both hyper-threaded and non-hyper-threaded machines (or you can test on the same machine provided that its BIOS allows you to turn off hyper-threading technology). A nice side effect of programming for hyper-threading is that most of it will improve application performance when run on a dual-processor machine, as well. I'll point out which approaches are specific to hyper-threading and will not work as well on a dual- or multi-processor machine. In general, applications that scale well to multiple processors would do well in a hyper-threaded environment.

Good places to start looking for CPU-intensive operations are pieces of code that render and draw graphics, encode or decode music or video, compress or encrypt data, or execute in tight loops. You can start by profiling your application in order to identify those CPU-intensive operations.

A useful tool is the Task Manager (TaskMgr.exe), shown in Figure 1. Under the performance tab, you can see each logical CPU utilization shown separately. You should ensure that Task Manager is configured to show one graph per CPU (under View | Select Columns | CPU Usage).

Figure 1 Dual-Processor with Hyper-Threading Enabled
Figure 1 Dual-Processor with Hyper-Threading Enabled

Similar information is available by checking the processor's "% Processor Time" performance counter either for all CPUs or for each logical CPU separately. PerfMon.exe is a handy tool to capture this information.

Sometimes, measuring only CPU utilization does not identify all CPU-intensive operations, since it may not show hot spots or short durations of CPU activity, but a CPU profiler can help identify those areas (TaskMgr and PerfMon tend to average out the CPU utilization over a period of a second, so they miss hot spots of shorter duration). The CLR profiler is useful for identifying methods that use excessive CPU, as are other profilers such as VTune, and the profiling tools that are available with the VisualStudio® 2005 Team System.

It's important to note that, for both Task Manager and Perfmon, CPU utilization percentages can be misleading. A CPU-bound operation executing in one thread will show a utilization of 50 percent. Due to the performance interdependencies between the logical CPUs, however, it is unlikely that utilizing both CPUs will result in double the performance.

For the rest of the article, I will assume two threads (T1 and T2) running on two logical CPUs (LC1 and LC2) on one physical CPU.

Two Heads Are Faster Than One

Suppose your client application performs some CPU-intensive operation, such as making complex calculations over a set of data. You notice that when the user initiates the operation, the calculation is running for three seconds, and then the results are available. Obviously, you want the wait to be as short as possible.

Checking the CPU utilization while the application is running is a good start, but profiling tools (such as the CLR profiler) can help you identify which part of the application is consuming the CPU and needs optimizing.

If the user is running on a hyper-threaded machine, one solution would be to split the work into two threads executing at the same time. This would cause both logical CPUs to be used. You will also need to wait until both threads are completed. Note that this approach assumes that the processing of each segment takes the same amount of time; if this assumption isn't fulfilled, the CPU hyper-threading capabilities will not be fully utilized in the latter part of the execution.

To be more generic, you can split CPU-intensive operations into multiple tasks and threads of execution, where the number of tasks and threads is the same as the number of logical processors (more threads are needed for operations involving I/O). For example, on a dual-processor computer with hyper-threading enabled, you might split a set of data to be processed into quarters, with a separate thread processing each part. Splitting up the work in this fashion decreases the need for locking, assuming that each task can be processed independently of the others. In general, lock contention degrades performance considerably on multiple CPU machines as well as on hyper-threaded machines, and as a result should be avoided whenever possible.

Figure 2 shows a code sample that doesn't make use of threading and that does some numerical processing on an array. One simple way to split this workload is to split the array in half and then process each half in separate threads running in the .NET ThreadPool, as shown in the code sample in Figure 3. The threads signal their respective ManualResetEvent when they complete their work, while the main thread waits for both events to be signaled (in the WaitHandle.WaitAll call).

Figure 3 Using the ThreadPool (Close Figure 3)
public class ThreadWorkItem
{
    public int Result;
    public int FirstElement;
    public int LastElement;
    public ManualResetEvent ManualEvent;
}

void DoWork2(int firstElement, int lastElement, ref int result)
{
    ThreadWorkItem wi1 = new ThreadWorkItem();
    wi1.FirstElement = firstElement;
    wi1.LastElement = (firstElement + lastElement)/2;
    wi1.ManualEvent = new ManualResetEvent(false);

    ThreadWorkItem wi2 = new ThreadWorkItem();
    wi2.FirstElement = wi1.LastElement+1;
    wi2.LastElement = lastElement;
    wi2.ManualEvent = new ManualResetEvent(false);

    ThreadPool.QueueUserWorkItem(new WaitCallback(DoWorkCallBack), wi1);
    ThreadPool.QueueUserWorkItem(new WaitCallback(DoWorkCallBack), wi2);

    WaitHandle.WaitAll(new ManualResetEvent[]{
        wi1.ManualEvent, wi2.ManualEvent});
    result = wi1.Result + wi2.Result;
}

void DoWorkCallback(object state)
{
    ThreadWorkItem wi = (ThreadWorkItem)state;
    DoWork(wi.FirstElement, wi.LastElement, out wi.Result);
    Wi.ManualEvent.Set();
}
Figure 2 CPU-Intensive Code on One Thread (Close Figure 2)
public int[] records;

void DoWork(int firstElement, int lastElement, out int result)
{
    result = 0;
    for (int i= firstElement;i<=lastElement;i++)
    {
        int tempvaluei = records[i];
        for (int j=0; j<1000; j++)
        {
            tempvaluei = (tempvaluei * 3 / 2)  +
                ((tempvaluei * tempvaluei) % 70) + 1;
        }
        result += tempvaluei;
    }
    return result;
}

The DoWork2 method will only be faster when the cost of the calculation is higher than the overhead of creating the WorkItems, waiting on the events, and all of the other processing required to execute these tasks asynchronously.

Of course, other ways to split the work are possible. For this example, both threads could iterate over all the elements, each performing part of the calculation, rather than each thread performing the whole calculation for only half of the elements. Which solution proves faster often depends on the use of the CPU cache.

Splitting the work into two threads involves overhead: data management overhead and overhead associated with thread creation, management, and switching. Not every operation requires the execution (and coding) overhead that this approach requires. Before splitting a task into multiple threads, you should consider the cost of threads and synchronization.

Hit and Miss

One key element when optimizing your code for hyper-threading is making the best use of the shared CPU cache. Compared to the speed of today's CPUs, RAM is slow. For data-intensive operations, often the time it takes to access memory is responsible for much of the CPU time used. This is because most CPU instructions (such as an integer addition) take one cycle, while main memory access can take as many as a few hundred cycles.

To address this, the Pentium 4 has a two-level memory caching system. An L1 (level 1) data cache will typically hold 8KB, while an L2 cache (level 2) might hold 512KB, but cache sizes increase with CPU releases, and some newer CPUs include an L1 of 16KB and an L2 of 2MB. When the processor requires information in memory, if that data is already in the L1 cache (known as an L1 cache hit), access to it takes on the order of 4 cycles. If the information required is in the L2 cache, access takes on the order of 10 cycles. If the request must go all the way to RAM, this cache miss may result in the operation requiring 100 cycles or more. And remember, if the information is not in physical memory and has to be brought in (paged) from disk, the wait time is equivalent to millions of CPU cycles.

In the Pentium 4, caching of RAM takes place in chunks called cache lines. The cache line size on all hyper-threading-enabled CPUs is 64 bytes, and during memory reads, up to two cache lines are fetched. Placing items that are often referenced together nearby in memory (generally referred to as locality) usually improves performance because there's a better chance that the latter memory accesses will be satisfied by the cache.

As you probably know, the common language runtime (CLR) already improves locality of reference. If you allocate objects consecutively, they will be allocated in consecutive addresses on the managed heap, which makes accessing them in sequence faster. And when a garbage collection is performed, the garbage collector compacts its heaps in order to keep objects closer together. (The CLR is also hyper-threading and cache-size aware, and performs additional optimizations to take advantage of the specific hardware it is running on.) The locality of your data becomes even more important the faster CPUs get. It's a good habit to allocate data together that will be used together, especially if those objects have similar lifetimes.

Along these lines, performance of multithreaded applications running on hyper-threaded processors is often improved when multiple threads read from a common area of memory. If one thread loads information from the main memory to the L2 cache, later requests by a second thread for that same information may be satisfied by the L2 cache. The information would not be stored twice in the L2 cache, freeing up valuable cache space (this usually doesn't happen with symmetrical multiple physical processors, since caches are typically not shared between CPUs), so relying on it in multiprocessor machines is discouraged unless you ensure (as I'll show later) that both threads run on the same physical CPU. Figure 4 shows an example that might benefit from those cache efficiencies when multiple threads are executing the DoWork method. The array used can be a common memory table, such as a translation table in this case.

Figure 4 Sharing Read-Only Data (Close Figure 4)
public int[] records = ...;
public int[] arr = ...;

void DoWork(int firstElement, int lastElement, out int result)
{
    result = 0;
    for (int i=firstElement; i<=lastElement; i++)
    {
        int tempvaluei = records[i];
        for (int j=0; j<1000; j++)
        {
            tempvaluei = arr[(tempvaluei * 3 / 2 + 1) % arr.Length];
        }
        result += tempvaluei;
    }
}

Elements from the array are more likely to already be in the CPU cache when another thread has recently accessed it. However, caching easily calculated values in tables (such as when arr[i] = Math.Pow(r,i)) may actually be slower than just recalculating the value, so as always make sure to profile your applications.

Sharing writable data between threads is a different story. To maintain information consistency, access to the shared data needs to be protected with locks. If you don't use locks, you'll get synchronization bugs. Many synchronization bugs that manifest only rarely on a single logical CPU machine can happen much more frequently on a hyper-threaded machine or in a multiple processor machine. Consider the code sample in Figure 5, where two threads update a counter at the same time.

Figure 5 Synchronization Bugs (Close Figure 5)
public class ThreadWork
{
    public static int GlobalCounter;

    private ManualResetEvent manualEvent;
    private int countTo;

    public ThreadWork(ManualResetEvent manualEvent, int countTo)
    {
        this.manualEvent = manualEvent;
        this.countTo = countTo;
    }

    public void DoWork()
    {
        for(int i=0; i<countTo; i++)
        {
            ThreadWork.GlobalCounter++;
        }
        manualEvent.Set();
    }
}

class TestCounters
{
    public static void TestCounter()
    {
        int numThreads = 10;
        int countTo = 100000;
        ManualResetEvent[] events = new ManualResetEvent[numThreads];
        Thread[] threads = new Thread[numThreads];

        ThreadWork.GlobalCounter = 0;
        for (int j=0; j<numThreads; j++)
        {
            events[j] = new ManualResetEvent(false);
            ThreadWork tw = new ThreadWork(events[j], countTo);
            threads[j] = new Thread(new ThreadStart(tw.DoWork));
        }
        for (int j=0; j<numThreads; j++)
        {
            threads[j].Start();
        }

        WaitHandle.WaitAll(events);

        Console.WriteLine("Expected:{0} Actual:{1}",
            numThreads * countTo, ThreadWork.GlobalCounter);
    }
}

The operation ThreadWork.GlobalCounter++ is actually composed of three operations: fetching the value stored in GlobalCounter, incrementing it by one, and storing it back. If the second thread fetches GlobalCounter in the middle of the update operation, it would see the old value, which might result in GlobalCounter being incremented only once.

In a single-processor computer, threads do not actually execute concurrently; each gets a timeslice and continues execution until it issues a blocking operation (such as file I/O) or its allocated timeslice ends. It is less likely, then, that a thread switch would occur in the middle of the update operations. In a hyper-threaded machine (or in one with multiple physical CPUs), the operations can execute at the same time, so the likelihood of this happening is higher. For example, the code in Figure 5 produced the right results on my non-hyper-threaded machine, but got the values off by 20 percent on my hyper-threaded machine. That's not good.

An additional source for hyper-threading and multiprocessor hard-to-detect bugs is relying on thread priority for locking. Some programs using threads with different priorities rely on the lower-priority thread not executing while the higher-priority thread is running. Assuming this is unsafe even on a single CPU system, as the higher-priority thread can block (when performing I/O, and periodically by the starvation-prevention scheduling inherent in Windows scheduler) but on a hyper-threaded system, the CPU executes the top two ready threads, not just one (in a dual processor system, the top two threads each execute on a separate CPU). Also, on a hyper-threaded system, a lower-priority thread can compete for CPU resources with the higher-priority thread (since both are executed as separate instruction streams, but the CPU is not aware of thread priority), causing background threads that perform polling, lower-value operations, or "busy waiting" to have negative performance impact.

Locking

Pay special attention to how you implement locking when your code will be run on a hyper-threaded or multiple-CPU machine. A simple fix for the code in Figure 5 would be to use Interlocked.Increment(ref ThreadWork.globalCounter) rather than ThreadWork.globalCounter++. More complex structures would need the use of more complex synchronization primitives such as Monitor, Mutex, and Semaphore.

Locks slow down execution in two ways: lock entry and exit takes time, and if a lock results in a thread blocking, a thread context switch normally occurs. What's more, in a hyper-threaded system, while one of the threads running on the processor is blocked, the L2 cache may be underutilized. Minimizing the number of locks as well as the time that locks are held can improve performance. You can minimize locking by having each thread update a separate instance (as I did in Figure 2), keeping wi1.Result and wi2.Result separate, and only generating the result after the calculations were complete. Additionally, you can try to make it less likely that threads would block by minimizing the amount of time the lock is held.

If your code can be restructured with a minimum of shared writable data, or updating shared data can be deferred, performance may benefit. Despite the use of the more expensive lock method in Figure 6 (which is transformed by the C# compiler into using Monitor), by removing the common memory reference and the interlock operation from inside the loop, you can speed up the code. Even though Interlock.Increment is considerably faster than Monitor.Enter, it is considerably slower than incrementing a variable, and is slower on hyper-threaded and multiprocessor systems than on a single processor.

Figure 6 Implementing Locking (Close Figure 6)
private object syncObj;

public void DoWork()
{
    int counter = 0
    for(int i = 0; i<countTo; i++)
    {
        counter++;
    }
    lock(syncObj)
    {
        ThreadWork.GlobalCounter += counter;
    }
    manualEvent.Set();
}

False Sharing and Cache Aliasing

Avoid modifying data shared between threads. Maintaining a local copy of the results per thread and merging the result when thread execution is complete can help. This is especially helpful if the modified data is limited in size, as increasing the application working size also impacts performance. Earlier I mentioned cache lines and locality. Assume the existence of the following class:

class OClass
{
    public int Field1;
    public int Field2;
}
If one thread is reading data from an instance of OClass by referring to Field1, and another thread on the same CPU is writing to nearby memory by assigning to Field2, it is possible (if Field1 and Field2 are close enough) that Field1 and Field2 will share a cache line. This will degrade performance. In machines with multiple physical CPUs, this problem is even worse.

Another best practice is to avoid writing to memory addresses sharing a cache line from two threads at the same time. A common workaround is padding. If OClass had additional fields between Field1 and Field2, the cache sharing might not occur. This problem is often seen when accessing two instances that were allocated sequentially, because the CLR will place them in adjacent memory locations. This generally improves performance since objects allocated together are generally used together, and an adjacent object is likely to be in the cache after the first object is accessed. But when multiple threads try to access fields from those objects, they may share a cache line, which would slow performance—this is called "false sharing". Padding the class (by adding fields) can help, or you can ensure that those elements are not allocated one after the other.

That said, don't just go adding unused fields to your classes in hopes that it'll improve performance. In general, it won't, and it'll make for sloppy code that's hard to maintain. I mention these issues because it's important to understand what's happening in the system under the covers, and to know where to look when problems arise. If you choose, after measuring and profiling your application, to add unused fields to a specific class, consider using an explicit layout to ensure that the fields are not reordered by the compiler. Also consider the effect of garbage collection—if the objects O1, O2, and O3 were allocated in sequence, O1 and O3 started out not sharing a cache line. But if no references to O2 exist, O2 can be garbage collected and O1 will now be adjacent to O3.

Another consideration is a set of effects known as the aliasing problem. A cache capacity aliasing problem occurs due to the CPU cache architecture, known as N-Way associative cache. The data in separate memory areas (separated by multiples of 2KB in the case of an 8KB L1 cache, and multiples of 64KB in the case of L2 cache of size 512KB, 128KB for a 1MB L2 cache, and so on) share a cache line location, and compete for n cache spots (n is usually 2, 4, or 8). For example, data in the addresses 0x00010, 0x08010, 0x38010 and 0x90010 (addresses are in base 16) are separated by a multiple of 32KB (0x8000), and would share an L2 cache location. This means that if more than n of those addresses are repeatedly accessed (by one or more threads), some of them would have to be read from the main memory, reducing performance. Additional aliasing problems exist in all but the newest Pentium 4 processors when memory addresses are separated by multiples of 64KB.

This isn't just an issue for unmanaged code, but affects managed code as well. If you have an array of C# integers, each of which occupy 4 bytes, and the flow of your program has thread A accessing element [i] when thread B accesses element [i+16KB], the memory areas accessed will be 64KB apart. Avoid placing objects referenced by concurrent threads 16KB (or multiples) apart in memory. There are some less common CPU resource issues as well; in general, for CPU-intensive operations try limiting the number of concurrent threads sharing any data to 4 to 8. More threads can be used if the thread operations utilize a significant amount of I/O or perform blocking operations.

Since CPU cache entries are not tagged to one logical CPU, it is possible for a thread running on logical CPU 1 to access enough memory to kick the data needed for a second thread running on logical CPU 2 off the cache. If the threads in your application access large chunks of memory and perform little additional work (such as copying large blocks of memory or summing up the values in an array of integers) running multiple threads on the same physical CPU might not improve performance, and can even hurt performance. For such applications, measure the performance with and without hyper-threading enabled.

Ah, the Competition

With hyper-threading, not all CPU operations are created equal. As mentioned earlier, to optimize performance under hyper-threading you should minimize competition for shared CPU resources between the threads. The most commonly contended resource besides the CPU cache is the FPU, responsible for floating point calculations. It turns out that executing heavy floating point operations simultaneously on both logical CPUs reduces performance significantly.

If you modify the calculation in the code sample from Figure 2 to use floating point operations (shown in Figure 7), the gains from splitting the task over multiple threads will diminish, and sometimes can result in a slowdown of up to 40 percent, if the operating threads do nothing but floating point operations due to inefficient CPU resource use.

Figure 7 Floating Point Calculations (Close Figure 7)
public float[] records;

void DoWork (int firstElement, int lastElement, out int result)
{
    result = 0;
    for (int i=firstElement; i<=lastElement; i++)
    {
        float tempvalue = records[i];
        for (int j=0; j<1000; j++)
        {
            tempvalue = (Math.Sin(tempvalue) * 3 /2)  +
                ((tempvaluei * tempvaluei) % 70) + 1;
        }
        result += tempvalue;
    }
}

Note that the Pentium 4 has only one FPU, so there is little to gain in trying to schedule execution of floating point operations on both logical CPUs. Performance might be better if the floating point operations are kept to one thread per physical CPU.

To Serve and Hyper-Thread

Server applications that process numerous simultaneous requests will often perform each operation in a separate thread, utilizing all logical CPUs, often by queuing work items to a thread pool. As a result, many server applications (such as ASP.NET applications) may benefit from hyper-threading without any code changes. One area to consider when optimizing server apps to hyper-threaded machines are the shared data problems I discussed earlier.

With ASP.NET, consider the Web garden configuration. In the default configuration for Web gardens, a process that has affinity to a logical CPU maintains its own copy of the data and services a separate set of requests. Since the ASP.NET Web garden configuration is based on logical CPUs and not physical CPUs, it is likely that all logical CPUs will be utilized, but no shared caching between the logical CPUs on one physical CPU will be realized.

CPU Affinity

A thread is normally scheduled to execute on any available logical CPU in the system. Sometimes, though, you want more precise control over which CPU will execute a thread. You can get this through affinity.

The CPU affinity is a bitmask representing which CPUs a thread or a process is allowed to execute on. This information can be retrieved by P/Invoking to the Win32® API GetProcessAffinityMask:

[DllImport("kernel32.dll", SetLastError=true)]
static extern int GetProcessAffinityMask (int hProcess,
    ref int lpProcessAffinityMask, ref int systemAffinityMask);
The returned value of the third parameter, systemAffinityMask, is a bitmask representing which logical CPUs are present on the machine. In all current version of Windows, the logical CPUs are numbered 0, 1, 2, and so on.

On a single physical CPU machine without hyper-threading, only the lower bit in systemAffinityMask will be set, resulting in the value 1. On a single physical CPU machine with hyper-threading, or a dual physical CPU machine without hyper-threading, both lower bits will be set, and systemAffinityMask will be 1+2=3. A dual physical CPU machine with hyper-threading would show systemAffinityMask as 1+2+4+8=15.

When hyper-threading is enabled, the operating system will assign a number to each logical CPU. In a system with n physical CPUs, the first physical CPU will be exposed as logical CPU 0 and logical CPU n. The second physical CPU will be exposed as logical CPU 1 and logical CPU n+1. Physical CPUs [A, B] would then be exposed as logical CPUs [0(A), 1(B), 2(A), 3(B)]. Using the returned systemAffinityMask is a simple way to count the number of logical CPUs, simply by counting the number of set bits. (This information is also available from several other Windows API calls. However, using the process affinity mask would tell you how many logical CPUs are available to your process, and is preferable.)

The sample shown in Figure 8 shows how to set a process affinity mask. By setting the process affinity mask to only one of the logical CPUs for each physical CPU, the application will behave as if hyper-threading was disabled.

Figure 8 Process Affinity Mask (Close Figure 8)
public void SetProcessAffinityToPhysicalCPUForHyperthreadOnly(
    int processid)
{
    int res;
    int hProcess;
    int ProcAffinityMask = 0, SysAffinityMask = 0;
    hProcess  = OpenProcess(PROCESS_ALL_ACCESS, 0, processid);
    res = GetProcessAffinityMask(
        hProcess, ref ProcAffinityMask, ref SysAffinityMask);
    if (SysAffinityMask == 3) // 1 proc, 2 logical CPUs
        res = SetProcessAffinityMask(hProcess, 1);
    else if (SysAffinityMask == 15)  //dual proc, 4 virtual CPUs
        res = SetProcessAffinityMask(hProcess, 3);
    res = CloseHandle(hProcess);
}

There is one caveat, however. The code in Figure 8 assumes that hyper-threading is enabled. The process referred to, and all processes spawned from it, will run only on the lower-numbered logical CPUs. This approach can be used to get an approximation of the performance of an application with hyper-threading disabled without having to reconfigure hyper-threading on the machine. It can also be useful for trying different performance solutions, but testing your application on both hyper-threaded and non-hyper-threaded machines should still be considered a requirement.

If the process affinity mask (returned in processAffinityMask) differs from the system affinity mask, threads in the process can only run on select logical CPUs. This can be done from the Task Manager user interface or by calling the SetProcessAffinityMask method. To change the process affinity mask in Task Manager, right-click a process from the process tab and select "set Affinity."

Similarly, calling SetThreadAffinityMask can limit a thread to run only on certain CPUs. Never change the affinity of a ThreadPool thread since the thread will be reused by the ThreadPool. This approach would be inadvisable in hosted situations. Also, this assumes that the managed thread you're running on is tied to an underlying OS thread, which may not always be the case. For example, if your code is running as a stored procedure in SQL Server™ 2005, it's possible for a managed thread to be mapped to a fiber, and you should avoid changing or relying on anything to do with a specific OS thread.

If for some reason you do need to control which operation runs on which logical CPU, you could create as many threads as there are logical CPUs, so that the first thread is limited to the first logical CPU, the second thread is limited to the second logical CPU, and so on. When you set a thread affinity mask, it will not execute if no logical CPU in the mask is available. In order to prevent performance and behavior bugs in this situation, it is advisable to use a "soft" affinity by using P/Invoke to access the SetThreadIdealProcessor method instead.

When setting a thread ideal processor, the operating system schedules the thread on the ideal processor as soon as it is available, but will execute it on any other available CPU when the ideal processor is not available.

Producer/Consumer

A very effective and highly scalable way to use caching is a variation on the producer/consumer threads pattern. In this pattern, a "producer" thread performs a transformation on some piece of data, stores the transformed data in a buffer, and then processes the next piece of data. A second thread (the consumer) processes data in the buffer and creates a final output.

Using this variation, the memory that the consumer thread is processing is likely to be in the CPU cache already if the consumer and producer run on the same physical CPU. The producer and consumer threads can run concurrently on separate logical CPUs. To scale well on multiprocessor machines and when either producer or consumer uses I/O, multiple producer and consumer threads can be used; performance would benefit if producers and consumers of the same data reside on the same physical CPU. Using this variation also helps when the workload cannot be accurately divided in advance (for example, when some data elements take more processing, and you do not know which elements would do so before assigning them to threads).

This pattern is also useful for processing streams, and it can be used in situations where running multiple threads on separate segments of the data is inefficient, as when the consumers' (or the producers') calculations rely heavily on floating point operations. Implementing the producer/consumer pattern requires more code than can be covered in this article.

Detecting Hyper-Threading

It is generally not necessary to detect hyper-threading since scaling an application based on the number of logical processors usually works. In the cases where detection is needed, doing so from managed code is difficult.

On Windows Server™ 2003, this information is exposed through P/Invoking GetLogicalProcessorInformation; however, this API is not available on Windows XP. You can get the information in Windows XP by using the CPUID assembler instruction directly, but I won't go into the details of this approach here.

Conclusion

Hyper-threading is the first step in introducing multiple, interdependent logical CPUs to desktop machines. I've presented various ways to optimize code for hyper-threaded machines by splitting execution into multiple threads. Future trends in CPU design will make multithreading an even more useful tool in improving application performance for both server and desktop applications (see Herb Sutter's article on concurrency at The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software). Every developer would benefit from adding multithreading to their toolbox.

Yaniv Pessach is a 13-year industry veteran and a software design engineer at Microsoft for Indigo Web services, where he focuses on network programming and code performance. Reach Yaniv at his personal Web site, www.yanivpessach.com.


Get A Raise: Discover a Series of Fortunate Event Handlers in Visual Basic

Get A Raise

Discover a Series of Fortunate Event Handlers in Visual Basic

Ken Getz
This article discusses:
  • How events work in Visual Basic 6.0, Visual Basic .NET, and Visual Basic 2005
  • How Visual Basic .NET events are compiled
  • Exceptions and manual invocation of event handlers
  • Writing custom events
This article uses the following technologies:
Visual Basic, .NET Framework
Code download available at: EventHandling.exe (158KB)
Contents
Events
Doing Things the Visual Basic 6.0 Way
Using ILDASM to Investigate Events
Multiple Event Handlers
Exceptions and Multiple Event Handlers
Investigating the Exception Behavior
Manually Invoking Each Listener
Using the .NET Event Design Pattern
Adding and Removing Handlers Dynamically
Custom Events in Visual Basic 2005
What's Next?

E
vents are an integral part of your coding arsenal, whether you're using Visual Basic® 6.0, Visual Basic .NET 2002, Visual Basic .NET 2003, or Visual Basic 2005. Forms and controls raise events, and your code handles those events. The first application that you wrote in Visual Basic most likely involved placing a button on a form, handling the click event, and displaying some text in an alert box when you clicked the button at run time. What could be easier?

But how much do you really know about events? What's going on when you add an event handler to a class? In this article, which I've based on courseware I wrote for AppDev, I'll demonstrate various ways to interact with events and event handlers, and I'll show how they can solve common problems. Some of this information may not be new to you—but if you've taken events at face value, there are sure to be some surprises. In either case, download the two sample applications (one for Visual Basic .NET .2002 and 2003 and one for Visual Basic 2005) and follow along. All of the content applies to Visual Basic .NET 2002 and 2003 and Visual Basic 2005, with the exception of the final section covering custom events, which only work in Visual Basic 2005.

I'll assume you have some basic knowledge of delegates and multicast delegates. If you haven't investigated these important Microsoft® .NET Framework features, it's time that you do. For more information, check out Ted Pattison's two-part overview on delegates (Basic Instincts: Implementing Callback Notifications Using Delegates and Basic Instincts: Implementing Callbacks with a Multicast Delegate).

Events

Visual Basic (before Visual Basic .NET) provided simple mechanisms for you to create and handle events, and Visual Basic .NET 2002 and 2003 offered several different ways to do both. Visual Basic 2005 allows you even greater control over event handlers, as you'll see in this article.

Events provide a loosely coupled mechanism that allows classes to register for notifications that may or may not happen sometime in the future. If the "listeners" get notified that the event they're waiting for has occurred, they handle the situation. If not, they just keep listening. A button click event handler registers itself with the class providing the button's functionality; when a user clicks the button, the button's class raises the Click event, all the listeners (there can be multiple Click event handlers for the button) run their code, and execution of the code continues.

My sample application includes a series of classes (FileSearch1 through FileSearch5 for Visual Basic .NET 2002 and 2003, and FileSearch1 through FileSearch6 for Visual Basic 2005) that search for files in a specified location and raise an event when one is found. The FileSearch classes simply want to let any class that happens to be listening know when something interesting happens. In this case, the something interesting occurs each time the FileSearch class finds another file. Any number of listener classes might want to react to this event.

In .NET terms, a class can raise an event at any point during its code execution. Other classes can subscribe to the event, and can be notified by the .NET Framework when the event occurs. The class that raises the event generally has no knowledge of how many (if any) listeners exist, although it can gather this information with some effort, as you'll see later in the article. In addition, multiple listeners can register to be notified, and each can be notified without the knowledge of any other listener.

Doing Things the Visual Basic 6.0 Way

The designers of the .NET Framework, and the Visual Basic .NET language, went to great lengths to ensure that you can treat events in .NET just as you did in Visual Basic 6.0. That is, you can:

  • Declare an event using the Event keyword
  • Raise the event using the RaiseEvent statement
  • Handle the event using a WithEvents variable

The real difference with .NET is the underlying mechanism. As opposed to using some hidden plumbing in Visual Basic 6.0, Visual Basic .NET uses a visible, extensible, common plumbing—delegates—to manage event handling.

Another difference between Visual Basic 6.0 and the versions of Visual Basic .NET is that in .NET, you can use the Handles clause to indicate that a particular procedure should run in response to a particular event. The Handles clause allows any procedure that matches the parameter signature of the event to react to the event. This sounds a lot like a delegate, and under the covers, it is. At compile time, the .NET Framework creates a Delegate class with the name of your event, adding the text "EventHandler" at the end. For example, when you declare an event named FileFound, the .NET Framework creates a delegate type named FileFoundEventHandler for you. Every procedure that handles this event must have a signature that matches that delegate type. You'll see an example of this concept later in the article.

Clicking RaiseEvent on the sample form demonstrates how Visual Basic .NET supports the standard event handling found in Visual Basic 6.0. The sample project includes the FileSearch1 class, which uses the following code to set up the event:

' From FileSearch1.vb
Public Class FileSearch1
    ' Search for files in specified locations.
    ' Once found, this class raises the FileFound event
    ' for each found file.

    Public Event FileFound(ByVal fi As FileInfo)

    ... ' Code removed here...
End Class
As it finds files, the FileSearch1 class raises the FileFound event:
' From FileSearch1.vb
Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

For Each fi As FileInfo In afi
    RaiseEvent FileFound(fi)
Next
alFiles.AddRange(afi)

In frmMain.vb, you'll find the following declaration, which allows the code to use the variable fs1 to react to events raised by the FileSearch1 instance:

Private WithEvents fs1 As FileSearch1
Clicking RaiseEvent runs the following code:
' From frmMain.vb
fs1 = New FileSearch1( _
    Me.txtSearchPath.Text, Me.txtFileSpec.Text, _
    Me.chkSearchSubfolders.Checked)

Try
    fs1.Execute()
Catch
End Try
Finally, frmMain.vb contains an event handler that handles the FileSearch1.FileFound event:
' From frmMain.vb
Private Sub EventHandler1( _
  ByVal NewFile As System.IO.FileInfo) _
    Handles fs1.FileFound

    AddText("EventHandler1: " & NewFile.FullName)
End Sub

Although frmMain.vb only includes a single procedure that handles the FileSearch1.FileFound event, it's quite possible (and likely) that you would have more than one procedure that handles a particular event. That is, when the FileSearch1 class raises its FileFound event, it's possible that multiple procedures might handle it. This should sound a lot like the concept of multicast delegates, because it is. Under the covers, .NET converts events into delegate classes. Investigating the IL using ILDASM.exe will make this clear.

Using ILDASM to Investigate Events

If you use ILDASM (the .NET Framework IL Disassembler tool included with the .NET Framework SDK) to open the sample executable, you'll get information like you see in Figure 1. Even though FileSearch1 didn't explicitly contain a multicast delegate named FileFoundEventHandler, the Visual Basic .NET compiler created one, corresponding to the type defined by the FileFound event that the code declares. The compiler also creates an instance named FileFoundEvent of the delegate type FileFoundEventHandler representing the event listener(s).

Figure 1 Sample Code in ILDASM
Figure 1 Sample Code in ILDASM

Using this technique, the compiler can enforce strict adherence to the parameter signature of your event declaration without your having to worry about creating your own delegate type and subscribing to that type in your event declaration. As you'll see later, you're perfectly welcome to create your own delegate representing your event and use that delegate as the type for your event.

Multiple Event Handlers

The next class I'll discuss, FileSearch2, is an identical copy of the FileSearch1 class. The only difference in the use of FileSearch2 is that the sample form includes multiple listeners for the FileFound event of the FileSearch2 class. That is, frmMain.vb includes the following declaration:

' From frmMain.vb
Private WithEvents fs2 As FileSearch2
The sample form also includes the event handlers shown in Figure 2. These also listen for the FileFound event raised by the FileSearch3 and FileSearch4 classes, which I will cover later.

Figure 2 Multiple Event Handlers (Close Figure 2)
' From frmMain.vb
Private Sub EventHandler2( _
  ByVal NewFile As System.IO.FileInfo) _
    Handles fs2.FileFound, fs3.FileFound, fs4.FileFound

    AddText("EventHandler2: " & NewFile.FullName)
End Sub

Private Sub EventHandler3( _
  ByVal NewFile As System.IO.FileInfo) _
    Handles fs2.FileFound, fs3.FileFound, fs4.FileFound

    AddText("EventHandler3: " & NewFile.FullName)
End Sub

Clicking Multi-Listener on the main form creates an instance of the FileSearch2 class, calls the instance's execute method, and displays the output shown in Figure 3.

Figure 3 Multiple Listeners Allow Multiple Procedures to Run
Figure 3 Multiple Listeners Allow Multiple Procedures to Run

But watch out. When you use the multiple Handles clauses reacting to the same event, you have no control over the order in which the event handlers run. The .NET Framework provides two alternatives, discussed later in the article, which allow you to take greater control over multiple listeners.

Exceptions and Multiple Event Handlers

As you've seen, everything is working as expected. If you have multiple handlers for an event, when the event is raised, each of the handlers gets called in turn by the .NET Framework. So far, so good. What happens if one of the event handlers raises an exception? Then things don't go so well.

To demonstrate this, click RaiseEvent Error on the sample form. This example creates a new instance of the FileSearch3 class (there's nothing new in the class itself). The sample form in Figure 4 provides several procedures that handle the FileSearch3.FileFound event, but one throws an exception.

Figure 4 One Event Handler Raises an Error (Close Figure 4)
' From frmMain.vb
Private Sub EventHandler2( _
  ByVal NewFile As System.IO.FileInfo) _
    Handles fs2.FileFound, fs3.FileFound, fs4.FileFound

    AddText("EventHandler2: " & NewFile.FullName)
End Sub

Private Sub EventHandler3( _
  ByVal NewFile As System.IO.FileInfo) _
    Handles fs2.FileFound, fs3.FileFound, fs4.FileFound

    AddText("EventHandler3: " & NewFile.FullName)
End Sub

Private Sub EventHandler4( _
  ByVal NewFile As System.IO.FileInfo) _
    Handles fs3.FileFound, fs4.FileFound

    AddText("EventHandler4: Throwing exception!")
    Throw New ArgumentException
End Sub
Figure 5 One Event Listener Throws an Error
Figure 5 One Event Listener Throws an Error

What happens when you run this code? You'll get the results shown in Figure 5. If any event listener raises an exception, the whole event-handling chain stops. If you pause to consider what's going on, this behavior makes sense.

Investigating the Exception Behavior

In a moment I'll show you code from the FileSearch3 class. Each time an instance of the class finds a file, the instance raises its FileFound event. This one statement causes the .NET Framework to run each event-handling procedure, one after the other. Take a look at the following code:

' From FileSearch3.vb
' Search for matching file names.
Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

For Each fi As FileInfo In afi
    ' This is equivalent to:
    ' FileFoundEventHandler.Invoke(fi)
    RaiseEvent FileFound(fi)
Next
alFiles.AddRange(afi)
The problem with this technique (as you've seen) is that if an exception occurs in any listener, the exception bubbles back to the event-raising code, and the .NET Framework calls no more of the event listeners, and event handling grinds to a halt.

If you look at the IL generated for the sample code, this situation becomes clearer. Figure 6 shows the disassembly of the FileSearch3.Search method in ILDASM. The RaiseEvent statement you saw in the class compiles down to a FileFoundEventHandler.Invoke method call. Internally, once this method call has been made, your code transfers control to the delegate's implementation, and if an unhandled exception occurs anywhere in the invocation list, the exception bubbles back to the caller (this code), and no more listeners get called.

Figure 6 RaiseEvent Compiled to MSIL (Close Figure 6)

There is an alternative. Rather than simply calling the RaiseEvent statement over and over again, it is possible for you to call each individual listener explicitly. You can take advantage of the members of the Delegate class to solve the problem of unhandled exceptions in multiple listeners.

Manually Invoking Each Listener

Although the RaiseEvent mechanism is convenient (and comfortable if you're used to Visual Basic 6.0), it has its drawbacks, as you've seen. Rather than relying on RaiseEvent to invoke the event's listeners, you can do this work yourself. And you gain a level of flexibility that you relinquish when you use RaiseEvent.

If you want to take complete control over the invocation of the event listeners instead of just hoping things work out the way you want, you'll need to take advantage of the implicit FileFoundEventHandler delegate type. You can retrieve an array containing all of the event's listeners by calling the GetInvocationList method of the event delegate instance itself. Once you have the list, you can call the Invoke method of each listener individually, and trap any exception raised by the event handler. That way, if any listener raises an exception, you can handle it and move on.

The FileSearch4 class contains the code shown in Figure 7 in its Search method. Run the code by clicking GetInvocationList on the sample form. As you'll see, this sample still calls the event listener that raises an error, but in this case, the code doesn't stop looking for files the first time the listener raises the error. Because the FileSearch4.Search method includes code to invoke each listener individually, it can handle exceptions for each call as well.

Figure 7 Catch Errors for Each Listener (Close Figure 7)
' From FileSearch4.vb
Dim ListenerList() As System.Delegate
Dim Listener As FileFoundEventHandler
ListenerList = FileFoundEvent.GetInvocationList()

' Search for matching file names.
Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

For Each fi As FileInfo In afi
    For Each Listener In ListenerList
        Try
            Listener.Invoke(fi)
        Catch
            ' Something goes wrong? Just move on to
            ' the next event handler.
        End Try
    Next
    alFiles.AddRange(afi)
Next

The new code in FileSearch4.Search takes the following actions:

  • Declares an array of System.Delegate types so that the code can track all of the listeners for the event:
    Dim ListenerList() As System.Delegate
  • Declares an instance of the delegate type that describes the event in order to loop through the array of listeners (you should remember that all of the listener procedures must be of this particular type, or the code wouldn't have compiled—that's the way delegates work):
    Dim Listener As FileFoundEventHandler
  • Retrieves the list of listeners, calling the built-in GetInvocationList method of the FileFoundEvent delegate:
    ListenerList = FileFoundEvent.GetInvocationList()
  • Finds the files and loops through the array containing the cor-responding FileInfos , as you've seen previously:
    Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

For Each fi As FileInfo In afi
    ' Code removed here...
Next
  • For each file found, FileSearch4.Search loops through the list of event listeners and calls each delegate's Invoke method individually. This allows the code to trap (and in this case, dis-regard) any exceptions raised by each individual listener:
    For Each Listener In ListenerList
    Try
        Listener.Invoke(fi)
    Catch
        ' Something goes wrong? Just move on to
        ' the next event handler.
    End Try
Next

The sample project declared neither the FileFoundEventHandler type nor the FileFoundEvent variable. The Visual Basic .NET compiler creates these items when it finds the event declaration in your code. Although you can clear up this ambiguity by declaring these objects yourself, you needn't—the Visual Basic .NET compiler will do the work for you.

In Visual Basic .NET 2002 and 2003 you can't modify the manner in which .NET-based applications raise their own built-in events. (You can modify this behavior in Visual Basic 2005, as you'll see later.) Using versions prior to Visual Basic 2005, there is a way to ensure that you don't cause trouble in your event listeners (remember that an unhandled exception in any event handler will cause the .NET Framework to stop calling listeners for the current event). To do so, make sure your own event handlers don't allow exceptions to bubble up. Handle all exceptions in your event procedures so that you won't break the chain of event handlers if you want to have a single event handled by multiple event procedures.

Using the .NET Event Design Pattern

Although there's nothing wrong with raising events as you might have done in Visual Basic 6.0 and in my preceding examples, the .NET Framework has adopted a particular design pattern for events which you should adopt for your applications. In this pattern, all events provide two arguments: an Object, providing a reference to the object that raised the event (normally named sender), and an EventArgs object (or an object that inherits from EventArgs), providing information pertinent to the event (normally named e).

The standard .NET Framework design pattern for events adds three more recommendations. First, if your event needs to pass any information to its listeners, you should create a class that inherits from EventArgs and that contains the additional information. You can use your class's constructor to accept and store the information. In the sample project, the FileFoundEventArgs class looks like Figure 8.

Figure 8 FileFoundEventArgs Class (Close Figure 8)
' From FileFoundEventsArgs.vb
Public Class FileFoundEventArgs
    Inherits EventArgs

    Private mfi As FileInfo

    Public ReadOnly Property FileFound() As FileInfo
        Get
            Return mfi
        End Get
    End Property

    Public Sub New(ByVal fi As FileInfo)
        ' Store the FileInfo object for later use.
        mfi = fi
    End Sub
End Class

Second, provide a procedure that raises the event. Most .NET Framework classes raise an event from an overridable protected procedure, normally named OnEventName (in the case of the FileFound event, the procedure would be named OnFileFound). Code that needs to raise the event calls the OnEventName procedure, which in turn raises the event. Making this a protected method means it's available to the objects of the current type and to any object based on a class that inherits from the current class. Making it overridable means that inheriting classes can change the behavior of the event—an inheriting class could add code that runs before or after calling the base class's OnEventName procedure, or could bypass it altogether. In the sample project, the FileSearch5 class provides the following protected procedure:

' From FileSearch9.vb
Protected Overridable Sub OnFileFound(ByVal fi As FileInfo)
    RaiseEvent FileFound(Me, New FileFoundEventArgs(fi))
End Sub
This procedure passes the keyword Me in the RaiseEvent statement's first parameter. This keyword refers to the object in which code is currently running, which is most certainly the object that's raising the event.

Third, you may find it useful to create your own event delegate type. Although you can get by without defining an explicit event delegate, you gain some flexibility in creating your own. When you create a delegate, you're defining a "type" for procedures. If you have more than one event that requires the same set of parameters, it can be useful to create a delegate that defines the type. That way, should you need to modify the parameters, you can simply modify the delegate, leaving the event declarations alone.

For example, you could declare the FileFound event, without using an event delegate, like so:

Public Event FileFound( _
    ByVal sender As Object, ByVal e As FileFoundEventArgs)
If you then wanted to declare another event, using the same parameters, you would need to repeat the whole declaration:
Public Event FileFoundSomeOtherEvent( _
    ByVal sender As Object, ByVal e As FileFoundEventArgs)

Alternatively, you could declare a new delegate type that described the parameter signature of your events:

Public Delegate Sub FileFoundEventHandler( _
    ByVal sender As Object, ByVal e As FileFoundEventArgs)
Then you could declare events of this type:
Public Event FileFound As FileFoundEventHandler
Public Event FileFoundSomeOtherEvent As FileFoundEventHandler
If you don't take this extra step, the Visual Basic .NET compiler will do the work for you, adding the new delegate type to the class's metadata. The FileSearch5.Search method takes advantage of this mechanism, calling the OnFileFound method for each file it finds:
' From FileSearch5.vb
Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

For Each fi As FileInfo In afi
    OnFileFound(fi)
Next

Clicking Event Design Pattern on the sample form creates an instance of the FileSearch5 class and calls its Execute method (just like all the previous examples). In this case, the event handler for the FileSearch5.FileFound event is a bit different—rather than accepting simply a FileInfo object, this event handler looks like a standard .NET event handler; it accepts two parameters and uses the FileFound property of its FileFoundEventArgs parameter to display the found file name:

' From frmMain.vb
Private Sub fs5_FileFound( _
  ByVal sender As Object, ByVal e As FileFoundEventArgs) _
    Handles fs5.FileFound

    AddText(e.FileFound.FullName)
End Sub

Although you aren't required to use the standard .NET event-handling design pattern, it's always best to have your own events match those raised by built-in .NET objects. You gain the "familiarity" benefit in your event listeners, and they look like other events. Creating your own event delegate is optional, but if you have multiple events that pass the same parameters, using the event delegate can simplify your code. Plus, it will be easier to modify your code anytime you need to because changes to the event parameters only have to be made in one place.

Adding and Removing Handlers Dynamically

Every event listener that you've seen so far requires you to hook up the event handler at design time. The Handles clause is a convenient and simple way to hook up events raised by objects declared using the WithEvents keyword, but it doesn't provide any flexibility at run time. In addition, when multiple procedures handle the same event, the Handles clause gives you no control over the order in which event handlers execute. (Of course, to get around this you can use the trick you saw earlier, looping through the items returned by calling GetInvocationList. This technique requires additional code in the class that raises the event, not in the code that hooks up event listeners.)

To gain complete flexibility over when and in what order event handlers are called, you can use the AddHandler (and RemoveHandler) statements rather than the Handles clause. The AddHandler and RemoveHandler statements allow you to supply a specific event and the address of a procedure to be called in response to that event. Each call to AddHandler associates the procedure with the event so that the .NET Framework calls the procedure when the event occurs. In addition, AddHandler always adds the event handler to end of the invocation list for the event. This means that you control the order in which events are handled.

Of course, if you think about it for a minute, you realize that when you have multiple procedures, each with a Handles clause for the same event, the Visual Basic compiler creates a multicast delegate instance for the event handler without allowing you to control the order in which they're added. Raising the event calls the Invoke method of that delegate instance, which then calls each of the event listeners in the order in which they were added (and you had no control over that order). When you use the AddHandler and RemoveHandler statements rather than the Handles clause, you're simply taking control over the order in which items are added to the multicast delegate. Each time your application calls the AddHandler statement for the same event, you're adding a new listener for the event to the end of the list. When you raise the event, the .NET runtime calls each of the listeners, in turn.

If you click Add/RemoveHandler on the sample form, a new instance of the FileSearch5 class is created, and multiple event handlers for the instance's FileFound event are hooked up. Then, when the code calls the Execute method of the instance, the listbox on the sample form displays the results:

' From frmMain.vb
Dim fs5 As New FileSearch5( _
    Me.txtSearchPath.Text, Me.txtFileSpec.Text, _
    Me.chkSearchSubfolders.Checked)
AddHandler fs5.FileFound, AddressOf EventHandler7
AddHandler fs5.FileFound, AddressOf EventHandler6
AddHandler fs5.FileFound, AddressOf EventHandler5
AddText("Note the order of invocation:")
fs5.Execute()
Next, the code removes EventHandler7 from the invocation list and calls the execute method again:
RemoveHandler fs5.FileFound, AddressOf EventHandler7
AddText(String.Empty)
AddText("And then there were two:")
fs5.Execute()
Finally, the code removes the remaining event handlers:
RemoveHandler fs5.FileFound, AddressOf EventHandler6
RemoveHandler fs5.FileFound, AddressOf EventHandler5

Remember, the procedures whose address you're supplying when you call the AddHandler and RemoveHandler statements must be of the correct delegate type. Therefore, unless the parameter signature of the procedures whose addresses you supply to AddHandler and RemoveHandler match the event's parameters (that is, unless they're of the correct delegate type), your code won't compile.

Figure 9 Control the Order of Event Handler Invocation
Figure 9 Control the Order of Event Handler Invocation

Figure 9 shows the output from clicking Add/RemoveHandler. As you can see, the event procedures are called in the order in which you added them to the invocation list.

Custom Events in Visual Basic 2005

Remember the problem that occurred when an event had multiple listeners, and one event listener threw an exception? This problem has a reasonably simple solution, using the invocation list of the multicast delegate instance associated with the event. In Visual Basic .NET 2002 and 2003, however, there were some other event challenges that simply couldn't be worked around without complex or inefficient code. Prior to Visual Basic 2005, the instance of the event delegate type was always created for you by the Visual Basic compiler, and the compiler provided no way for you to modify the behavior of this delegate instance.

Visual Basic 2005 adds the new Custom keyword for event declarations. This keyword allows you to supply code for the AddHandler, RemoveHandler, and RaiseEvent behaviors of the event. It's up to you to create the appropriate delegate type and to create the instance of the type that holds information about the event listeners. Beyond that, however, you have complete control over how you handle the event.

To create a custom event in Visual Studio 2005, place your cursor within a class, and enter a declaration for your event, like this:

Public Custom Event <YourEventName> As <EventDelegateType>
When you end the line of code, the editor will insert the associated AddHandler, RemoveHandler, and RaiseEvent sections. For example, imagine that you want to solve the exception issue by creating a custom event handler. The FileSearch6 class in the Visual Basic 2005 version of the sample project contains a custom FileFound event that does this. The code includes a declaration for the appropriate event delegate, as shown earlier:
Public Delegate Sub FileFoundEventHandler( _
    ByVal sender As Object, ByVal e As FileFoundEventArgs)
Typing the event declaration adds an empty custom event, like the one that is shown in Figure 10.

Figure 10 Generated Empty Event (Close Figure 10)
Public Custom Event FileFound As FileFoundEventHandler

    AddHandler(ByVal value As FileFoundEventHandler)

    End AddHandler

    RemoveHandler(ByVal value As FileFoundEventHandler)

    End RemoveHandler

    RaiseEvent(ByVal sender As Object, ByVal e As FileFoundEventArgs)

    End RaiseEvent
End Event

It's up to you to provide the storage for the event delegate instance, and to provide the code that stores, removes, and invokes the event listeners. For this example, because the code needs to be able to invoke each listener individually and trap and handle any exceptions, the FileSearch6 class includes a generic List collection in which to store the FileFoundEventHandler instances. Each time any class calls AddHandler for this event, or uses a Handles clause trapping this event, the Visual Basic runtime engine calls the AddHandler portion of the FileFound custom event. The code must add the FileFoundEventHandler passed into the generic List. The RemoveHandler section removes the specified delegate instance from the internal collection. The RaiseEvent section calls the Invoke method of each delegate instance, trapping and dealing with exceptions as they occur. The complete custom event looks like the code shown in Figure 11.

Figure 11 The Custom Event (Close Figure 11)
Private listeners As New List(Of FileFoundEventHandler)

Public Custom Event FileFound As FileFoundEventHandler
    AddHandler(ByVal value As FileFoundEventHandler)
        listeners.Add(value)
    End AddHandler

    RemoveHandler(ByVal value As FileFoundEventHandler)
        If listeners.Contains(value) Then
            listeners.Remove(value)
        End If
    End RemoveHandler

    RaiseEvent(ByVal sender As Object, ByVal e As FileFoundEventArgs)
        For Each listener As FileFoundEventHandler In listeners
            Try
                listener.Invoke(sender, e)
            Catch ex As Exception
                ' Something goes wrong? Just move on to the next handler.
            End Try
        Next
    End RaiseEvent
End Event

By pushing this code into the event declaration itself, the code that raises the event needn't worry about handling exceptions. That is, rather than using the code you saw previously to raise the event, code consuming the FileSearch6 class in Visual Basic 2005 can simply call the OnFileFound method (which raises the event) or call RaiseEvent directly. The onus of worrying about exceptions is now in the right place—in the event code itself. This technique wasn't available until Visual Basic 2005. Note that Visual Studio 2005 is still in beta. As such, details are subject to change until the final release.

How else might you use custom events? Rocky Lhotka, a Visual Basic MVP, includes another detailed example on his blog (.NET 2.0 solution to serialization of objects that raise events). He demonstrates how you might use this technique to work around problems involving serialization of classes that raise events, in which the listeners aren't serializable. (Amazingly, this comes up quite often, because forms aren't serializable, but are often listeners for user-created events.) Paul Vick, a member of the Visual Basic development team at Microsoft, includes an example on his blog that shows how you might use a custom event to reduce overhead for classes that expose a large number of events in which only a few might be consumed. This is the case with forms, for example—the Form class exposes a ton of events, but most of the time, you only handle one or two of them. Without some tricks, the compiler would generate a delegate instance for each of those events, even though you're not using them. See Paul's blog at Custom events.

What's Next?

All the code you've seen in this article runs synchronously. That is, whether the FileSearch class calls the delegate method itself or raises an event, the search cannot continue until the listener (or listeners) completes its work. If the event-handling procedure stalls, displays an alert, or otherwise neglects to return, the search loop would stall as well. In general, that can't be a good thing. Fortunately, the .NET Framework provides several mechanisms for asynchronous event handling—see my column in the March 2005 issue of MSDN®Magazine (Advanced Basics: Doing Async the Easy Way) for one such technique you can follow to put the finishing touches on your event-handling code, and you'll be that much closer to event-handling Nirvana!

Ken Getz is a senior consultant with MCW Technologies. He is coauthor of ASP.NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook, and VBA Developer's Handbook, 2nd Edition (both Sybex, 2001). Reach him at keng@mcwtech.com.


Editor's Note: Reader E-mail

Editor's Note
Reader E-mail

Here at MSDN Magazine, we get a lot of mail from readers. As you can imagine, much of it involves fulsome praise for our content, our eye-catching covers, or the handsome photo that now graces this page. However, some of the mail we receive goes well beyond these basics and talks about, well, the basics. Especially prevalent around this time of year, our favorite mail takes this form:

Dear Columnist:
Provide the anwers to the following questions:
1. How large is a pointer?
2. How do you write strcpy?
3. How do you implement an infinite loop? —A. Reader

Here's the thing. Our mission in creating MSDN Magazine every month is to be the best source of information for developers who use Microsoft tools and technologies. We have a lot of pages of content because there's so much to talk about out there—we assume that readers can find introductory "how to program" information in alternate sources. Besides, since you found your way to our Web site, there's no doubt that you are one of the world's most sophisticated developers. We don't have time to do your school homework for you! Not only that, but we just might tell your teacher that you're asking us to do your work.

Okay, we admit it. MSN Search and Google weren't around when we were in school. But they exist now! We're a bit jealous. Of course, once this Editor's Note is on the Web, you'll search for "implement an infinite loop" and it'll bring you right back to this page. And after reading it, you'll follow our instructions and search for the term again, and it'll bring you back here. So now the question is: how long will you continue to do this before realizing that you've just answered your own question about infinite loops?

Another old favorite piece of mail looks like this:

Waiting for your reply (if you have not): Final Notice.
Client ID #: 627914
You have been approved for an extension/refinance of your home, saving you thousands per year.

Hey, how did you find out about our secret magazine hideaway? True, interest rates have been rising lately, but when we got that $600,000 jumbo mortgage on the conference room down the hall, we specifically checked the "do not contact us by e-mail" box.

In our editorial inbox, we tend to get all sorts of general assistance letters. People who can't find answers in a search engine often find their way to our contacts page. Mail like this always arrives with purple letters on a pink background, with stationery of kittens or seashells repeated down the left margin. Most of the time we can't make heads or tails out of what they're supposed to mean:

I'M TRYING TO DOWNLOAD THE GRIP TOOLBAR ON MY INTERNET EXPLORER, BUT I CAN'T SEEM TO LOCATE IT. CAN YOU PLEASE SEND ME THE LOCATION???

Grip toolbar? Is this some sort of specialized add-in for the motion picture industry? Maybe it comes packaged with the best boy toolbar and the dolly shell extension.

Mail like this aside, we do welcome all comments and questions about the magazine. If you'd like to tell us what you think about an article, drop us a line at mmeditor@microsoft.com. If you have a new product that you think we just have to hear about, send it to our New Stuff desk at newstuff@microsoft.com.

You can also reach us through our Web site. Every article on the site has a ratings and comments area on the bottom of the page. If you have thoughts about a piece we've run, you can let us know online. We look at every comment entered on the site, and we use aggregate ratings to help plan future coverage. We can even see which authors get the best and worst grades from our readers. (Which is why this editor no longer writes features for the magazine. Sigh.)

We always want to hear from you, so let us know what you're thinking. We'll use your feedback to improve the magazine, which will, in turn, lead to more feedback from you... and we're in an infinite feedback loop. Not a bad thing at all.

—J.T.

Thanks to the following Microsoft and Intel technical experts for their help with this issue: Raymond Balint, Joe Binder, Richard Burte, Simon Calvert, Rob Caron, Girish Chander, Raymond Chen, Bogdan Crivat, Chris Guzak, Philip Kerly, Ronald Laeremans, Martyn Lovell, Jamie MacLennan, Mark Matusiefsky, Bradley Millington, Jim Newkirk, Matt Odhner, Shankar Pal, Polita Paulus, Eric Perlin, Riyaz Pishori, Matt Powell, Raphael Renous, Juan Rodriguez, Michael Rys, Paul Schafer, Sam Spencer, and Maura Van Der Linden.

ClearType, IntelliSense, JScript, Microsoft, MSDN, Microsoft Press, Visual Basic, Visual C++, Visual Studio, Windows, Windows NT, Win32, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Marnie Hutcheson

Contents
Understand Your Database Schemas
Fast Data Integration
Build Complete SQL Scripts
Add Tools to Your SQL Editor
A Data Replication Solution
Manage Data from Multiple Sources
A CD-Based Training Suite

A few months ago I had a large SQL Server™ asset management database into which I had to import data from text files in daily batches. I needed to tweak the tables, add some queries, and generate new reports. To be sure that I had identified all dependencies, I asked the vendor for the schema. They sent a PDF that measured 6' x 8'. I couldn't print it in any usable fashion, and trying to follow the connections on the screen was a frustrating exercise to say the least, so it wasn't much help. I ended up doing everything by trial and error using the tools that come with SQL Server. I really wish I had known about this bunch of tools then! I could have saved weeks of debugging.

Understand Your Database Schemas

EasyStruct 4.5 from Vishwak Solutions gives you a complete picture of the structure of any database that supports OLE DB. You can use it to document the structure of a database or to answer specific questions, such as "What are the dependencies for this query?" This makes it very handy for modifying or migrating a database. It displays dependencies, stored procedures, and triggers. You can also view statistics and object properties like privileges and creation dates.

You can use the "Execute Query" feature to run a stored procedure within EasyStruct, so you can view, insert, update, and delete data. You can view the output in grid format or save it in HTML format.

EasyStruct 4.5 will let you generate a complete SQL script for recreating an existing database. It also supports XML and XML Schema natively. It allows you to define rules governing the relations between elements and attributes using a standard XML syntax that can be parsed and managed using other XML applications.

You select the objects you want to work with, and EasyStruct 4.5 lets you control the number of database objects in a single page. The reports include Entity Relationship diagrams using a new .NET Framework-based Diagram Application.

Price: $299 per single-user license.

www.easytools.com

Fast Data Integration

Extract, transform, and load (ETL) processes allow companies to move data from multiple sources into their own databases, transforming it (reformatting and cleaning it) along the way. Visual Importer Professional 7.0 by DBSoftware Laboratory combines ETL and design capabilities in one environment, simplifying the creation and deployment of complex data warehouses and marts.

Visual Importer Professional is a data integration tool for the high-speed transfer of data from text files and ODBC source databases into SQL Server, Oracle, or ODBC target databases. Its capabilities range from simple record transfer within a single system to supporting cross-system data extraction and loading that can be transformed by calculations including data formatting, concatenations, and loading cross or pivot tables.

There is a comprehensive error log, rejected records file, and a mapping editor to help you keep track of events. Visual Importer Professional 7.0 runs on Windows® 98 through Windows Server™ 2003.

Price: $150 per single-user license.

www.dbsoftlab.com

Build Complete SQL Scripts

Superior SQL Builder 1.5 by Red Earth Technologies is a SQL script builder that lets you build complete SQL scripts (not just single queries) without typing any code. Instead of building scripts using tokens that you replace with text values, Superior SQL Builder 1.5 uses soft programmed Script Templates that support multiple data types. Each data type can be configured at design time and run time, and exposes functionality that can be used to build highly customizable scripts.

The data you provide to the Script Templates via Inputs is exposed via Outputs. The data outputs of a Script Template can be used as Inputs into another Script Template, so the changes that you make in one Script Template "flow" downstream into other Script Templates.

You can build any type of SQL statement. Superior SQL Builder 1.5 virtually eliminates syntax, logic, and typographical errors when developing SQL, because instead of typing or cutting and pasting SQL, you drag and drop Superior SQL Builder Script Templates onto the work area and then connect them in order to define your data flow.

There are lots of templates to choose from. You fill in forms to configure the elements of the Script Templates to suit your solution. You select your database server type and when you are ready, you just click a button and your SQL script is generated. The Script Templates can work together to convert the entire project to the appropriate database dialect for SQL Server, Microsoft® Access, MySQL, and Oracle.

Superior SQL Builder runs on Windows 98 and later.

Price: $349 per license.

www.superiorsql.com

Add Tools to Your SQL Editor

DTM SQL editor by DTM soft is an integrated SQL environment that starts with a SQL editor and lets you add on more tools as you need them. The editor uses a step-by-step approach to let you edit SQL scripts and build SQL statements.

You don't have to dig into the actual SQL, but the editor does feature syntax highlighting, multilevel undo and redo features, and integrated search and replace tools. You can also use macros to automate entering frequently used fragments.

The add-on tools include a Connections Manager, database Schema Viewer, a View Execution History option, and a lot of other handy tools that are all available from the customizable user interface. The clipboard and data export tools export to plain text, HTML, XML, and Microsoft Office formats. In addition, all of your result queries can be saved as SQL statements.

DTM SQL comes in several options that range from the Standard version that gives you the SQL editor and connection to a single database, all the way to the Enterprise edition that lets you add on all the tools and work with large projects having multiple data connections.

Price: DTM SQL Enterprise is $220 per single-user license. Additional modules are available for purchase separately.

www.sqledit.com

A Data Replication Solution

ClusterReplica MSSQL Edition 2.0 by XLink TechnologyInc. is a turnkey replication and failover solution that is specifically designed for small businesses running SQL Server. It binds two servers running Windows 2000, Windows XP, or Windows Server 2003 into one cluster system with automatic failover capability. This is a critical feature if you are trying to support 24?7 availability and disaster recovery.

ClusterReplica MSSQL Edition version 2.0 adds enhancements to network communications that improve connections and client service on the clustered server. The Snapshot technology has been improved so that ClusterReplica can handle real-time, open file data replication for SQL database files and Windows registry files. Other improvements help reduce the incidence of corrupted files during replication and insure successful failover for SQL Server instances.

XLink has taken pains to make this product as easy as possible to set up and use. They claim that the cluster replication will be automatically configured when ClusterReplica is installed with both primary and secondary stations up and running—no IT experts required. The manuals and instructions support this goal with lots of clear instruction presented in simple steps.

Price: $799 per single-cluster license.

www.xlink.com

Manage Data from Multiple Sources

Aqua Data Studio 4.0 by AquaFold Inc gives you an integrated database environment with a single consistent interface to all major relational databases. Aqua Data Studio lets you administer and manage data from multiple data sources simultaneously, so you can tackle multiple tasks simultaneously from one application.

The Query Analyzer lets you create database scripts with specific relational database management system (RDBMS) syntax highlighting, auto-completion data retrieval, and custom results formatting. The new export tool gives you a wizard for extracting and exporting data from any relational database into different data sources, including SQL script with INSERT statements so that you can transfer data directly to other databases. You can visually manage the security of any relational database's users, roles, and object permissions.

Aqua Data Studio supports Windows, Macintosh OS X, Solaris, and the Linux operating system.

Price: $149 per single commercial user license. Free for educational institutions and personal use.

www.aquafold.com

A CD-Based Training Suite

SQL Server 2000 Grand Slam Multimedia Training Suite by SMI Corp. consists of 30 CDs that teach you techniques for administering and programming SQL Server 2000.

Here is the breakdown of the Grand Slam CD set: Practical SQL Server 2000 Administration (21 CDs), Data Warehouse Architecture with OLAP Cubes (5 CDs), Crash Course in SQL Server 2000 (3 CDs), and 120 Minutes to SQL Server Stardom (1 CD).

The video portion of the training lets you view the computer screen user interfaces used to administer, design, and program SQL Server as though you were looking over the instructor's shoulder. All you have to do is follow the mouse and the cursor as the instructor talks you through the short and practical lessons.

The material is aimed at beginner and intermediate students. Many of these courses are available individually.

Price: $895.

www.sqlusa.com

All prices were confirmed at press time and are subject to change.

Send your New Stuff to newstuff@microsoft.com.

Marnie Hutcheson is president of Internet Development Associates (Ideva), a firm in Ocala, Florida, that specializes in Internet and intranet Web application design and development. She has published technical papers and books on various computing topics. You can reach her at marnie@ideva.com.


Web Q&amp;A: Locking Pop-Up Blocker, Mixed Authentication, and More

Web Q&A
Locking Pop-Up Blocker, Mixed Authentication, and More
Edited by Nancy Michell

Q How can I prevent users from disabling the Windows® XP SP2 pop-up blocker in Microsoft Internet Explorer?

A Setting the permissions to "deny" on this registry key will pre-vent the user from changing the pop-up blocker settings:

HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\New Windows
Internet Explorer will have the pop-up blocker on by default.

Q I'd like to have one ASP.NET application that uses Windows authentication. When the user logs onto that app, he could click a link on the page that points to a second ASP.NET app that uses Basic authentication. Is there any way to pass the logged-on user's credentials to the second app using Basic authentication, so that the user doesn't get a second prompt for credentials?

A If you know the user name and password, it is possible to add the Basic credentials to the CredentialCache before making the WebRequest to the second application. However, it is not possible to get the user's password when they have logged in with Integrated Windows authentication.

If your second application will also accept Windows authentication, then you can add those to the CredentialCache instead, as long as delegation is supported. If both use Basic authentication, you can successfully call to the second Web application with the supplied credentials. Figure 1 shows some code that will allow you to pull the credentials from the first request and pass them on to the next. The sample shows how to add all three types of IIS authentication credentials. All can be used to make subsequent requests to another application on the same IIS server, but only the Basic credentials can be used to make a request to a different server. This is an added security feature that is built into IIS so that account could not be misused to access anything on other machines. See the "Authentication and impersonation types" and "Token types" sections of the article "Accessing Network Files from IIS Applications" for more details.

Figure 1 Retrieving Basic Credentials (Close Figure 1)
// Get the type of authentication
string authType = Request.ServerVariables["AUTH_TYPE"];

// If the request is not anonymous, get incoming creds
if (authType != null && authType.Length > 0)
{
    if (authType.Equals("Basic")) // add Basic credentials
    {
        // Get "Basic username:password" from the WWW-Authenticate header
        // and remove "Basic ".  "username:password" is base64-encoded.
        string auth = Request.Headers["Authorization"].Substring(6);
        string userinfo = Encoding.ASCII.GetString(
            Convert.FromBase64String(Auth));

        // Parse out the username and password
        int colonPos = userinfo.IndexOf(":");
        string username = userinfo.Substring(0, colonPos);
        string password = userinfo.Substring(colonPos+1);

        NetworkCredential basicCreds =
            new NetworkCredential(username,password);
        CredentialCache credCache = new CredentialCache();
        MyCredentialCache.Add(theUrl, "Basic", basicCreds);
        myProxy.Credentials = credCache;
    }
    else // Set default credentials, often Windows integrated
    {
        myProxy.Credentials = CredentialCache.DefaultCredentials;
    }
}
// make outbound request

Q Is there any way I can get the application pools feature found in IIS 6.0 or some functionality like it if I'm using IIS 5.0? I believe the only way to get multiple instances of the ASP.NET worker process under IIS 5.0 is to have applications running under different versions of the runtime.

A IIS 6.0 allows you to isolate applications based on namespace so, for example, all requests to http://server/foo can go to one worker process, while all the requests to http://server/bar can go to another worker process. The decision about which worker process to route to is made in kernel mode (HTTP.SYS).

IIS 5.0 doesn't follow this model. Every request has to go through INETINFO.EXE, which hosts the W3SVC service. INETINFO.EXE looks at the configured script maps and figures out which ISAPI extension to route the request to. For instance, ASPX requests go to ASPNET_ISAPI.DLL. ASPNET_ISAPI.DLL implements a process model that forwards requests to ASPNET_WP.EXE processes. This is also based on namespace, but only requests mapped to ASPNET_ISAPI.DLL can take advantage of the ASP.NET process model. In addition, routing from INETINFO.EXE to ASPNET_WP.EXE and back is relatively expensive (it involves a process hop).

If you want multiple instances of ASP.NET worker processes, you can follow the advice from Knowledge Base article 812833, "FIX: You cannot run multiple processes of ASP.NET at the same time". While the article alludes to being a fix on Windows Server™ 2003, it is really an answer to users unable to load multiple ASP.NET worker processes, which is what you're trying to do on IIS 5.0.

Q I need to spawn a process in IIS, run some batch files, and then return the batch files results to the user. The Knowledge Base article at How to spawn a process that runs under the context of the impersonated user in Microsoft ASP.NET pages discusses how to spawn a process that runs under the context of the impersonated user in ASP.NET pages, but it's not helping me. What should I do?

A An example of how to spawn a process and read standard out- put (and so forth) is shown in the code in Figure 2. This code does not perform any impersonation, which may be a requirement for your scenario. If it is, you're going to have to follow the example in the Knowledge Base article you mentioned.

Figure 2 Spawning a New Process (Close Figure 2)
private bool RunProcess(
    string command, string arguments, string workingDir)
{
    const int timeout = 30000; // 30 seconds

    ProcessStartInfo psi = new ProcessStartInfo(command, arguments);
    psi.RedirectStandardOutput = true;
    psi.RedirectStandardError = true;
    psi.UseShellExecute = false;
    psi.CreateNoWindow = true;
    if (workingDir != null) psi.WorkingDirectory = workingDir;

    bool retValue = false;
    using(Process p = System.Diagnostics.Process.Start(psi))
    {
        using(StreamReader stdOut = p.StandardOutput)
        {
            using(StreamReader stdErr = p.StandardError)
            {
                if (!p.WaitForExit(timeout))
                {
                    p.Kill(); // kill the process after 30 seconds
                }
                else if (p.ExitCode != 0)
                {
                    string output = stdOut.ReadToEnd();
                    string error = stdErr.ReadToEnd();
                    ... // use output and error info here for logging
                }
                else retValue = true;
            }
        }
    }
    return retValue;
}

Q Is it possible to have ASP.NET service two Web requests for the same ASP.NET session in two separate worker processes? For instance, when the existing worker process is getting recycled while servicing a request for one session and at the same time another request for that same session arrives, will the second request be processed by the old worker process or will it be processed on the new one?

Also, does ASP.NET provide any guarantee that session state for the same session is not loaded into two worker processes simultaneously? Otherwise, this could leave the session in an inconsistent state.

A There is a locking mechanism, so the problem you envision should not occur (moreover, if the process is being recycled, it's unlikely that it will be tinkering with session variables at that time). For out-of-process session state, the locking mechanism allows concurrent access to the same session data only in the event that all concurrent requests manipulate session state in a read-only manner (such as <@ Page EnableSessionState="ReadOnly" ... > ).

However, if read-only session state is not specified, then in the scenario described, any new inbound requests for session state would actually be blocked by the ASP.NET runtime until the old, pending request finished executing the page and subsequently released its lock on the session data. Each time a request successfully gets an exclusive lock on a piece of session data, the lock is valid for the execution timeout for a request.

If a second request starts up and asks for the same piece of session data, it will encounter the already existing exclusive lock and do one of two things. If the lock is older than the execution timeout, the second request will force the lock to be released and the second request will take exclusive ownership of the session state. If the lock is still valid though, the second request will spin on a timer, attempting to acquire the exclusive session state lock roughly every 500 milliseconds.

The second request will continue spinning until the first request completes, or until the lock becomes older than the execution timeout, or until the second request itself exceeds the execution timeout. Normally the first request completes execution at some point in time before the request execution timeout—and then when the second request hits the next polling interval it detects that the lock has been released and the second request then exclusively locks and takes control of the session state data.

Q Using a few client-side lines of JScript®, I can add (or remove) rows and cells to an HTML table, but when submitting the page all those changes are lost (newly added rows for instance). I guess this happens because they are not part of the ViewState content.

How can I persist those changes and save them between page postbacks, apart from moving all of that logic server side? I'm thinking of reading the value of the hidden __VIEWSTATE variable in JScript, decoding it, appending data to the string, re-encoding the ViewState, and so forth.

A Tampering with ViewState is something you do not want to do. Even though the encoding at this point is Base64, there is no guarantee it will stay that way in the future. For instance in the Microsoft® .NET Framework 2.0 the persistence format is very different from what the .NET Framework 1.1 uses.

Also, if you have EnableViewStateMac turned on (everyone should), you will get Invalid ViewState exceptions on the server, since there is a hash added at the end that needs to match the right ViewState. (If you have this turned off, you are taking a big security risk, since someone could hijack the ViewState and attempt a script injection attack.)

So having said all that, it is better and safer to use an approach where you register your own hidden field and set its value to your own persistence format (maybe a comma-separated value). Then on the server side, check for that form field and, if it's there, add the HTML back in on the server side.

Send your questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Greg Bybee, Carlo Cardella, David Chen, Dino Chiesa, Thomas Deml, Nick Duane, Chris Gariepy, Jimmy Leyow, Carlos Aguilar Mares, Mark Michelet, Dan Nemesek, John Nisi, Shahar Prish, Karthik Raman, Stefan Schackow, Adam Semel, Karl Tietze, Stephen Toub, and Radomir Zaric.


Cutting Edge: Data Repeater Controls in ASP.NET

Cutting Edge
Data Repeater Controls in ASP.NET
Dino Esposito

Code download available at: CuttingEdge0506.exe (144KB)
Contents
The Repeater Control
Template Properties
User Controls
Implementing the ITemplate Interface
List Controls
ItemCreated Event
When to Use Which

The ASP.NET Repeater is a basic container control that allows you to create custom lists from any data available to the page. It's a handy control, especially since most ASP.NET pages that display data need to repeat the same kinds of data over and over. In this month's installment, I'll take a look at the Repeater control and discuss when it's a good idea to use it and when other tools may be a better option. Let's get started.

The Repeater Control

Imagine you're told to create a page that pulls a list of order records from a database and displays highlights. I define highlights as the most pertinent information for a record, a handful of a table's columns formatted any way that suits you, your client, and your boss. For example, it could be the order ID in bold on a row, then a horizontal ruler followed by order date, followed by a word that indicates if the order has been shipped or not. How would you create a page like this? And what kind of repeater tool would you need? Let's consider the most obvious solution first—the ASP.NET Repeater control.

The Repeater control does exactly what its name suggests—it repeats the specified data-bound templates without applying any sort of additional formatting. For example, the Repeater doesn't create a table or a vertical or horizontal list of data by default. You determine the final layout of the generated markup by creating ASP.NET templates and explicitly adding markup that separates one row from the next.

The canonical example of a separator element in the context of a free-form repeater control is the <BR> tag. However, the markup for the separator is strictly page-specific and is up to you. The most clear and elegant way to specify a separator is through its own template, named SeparatorTemplate.

When the page runs, the control loops through the records in the data source and renders the specified template for each record. In addition, before and after processing the data items, the Repeater emits some markup for the header and the footer of the resulting structure. Because the Repeater control has no default appearance, it is useful for creating virtually any kind of list with any layout and style.

The following is some typical Repeater-based code to produce the aforementioned orders list:

<asp:Repeater runat="server">
  <ItemTemplate>
     <%# DataBinder.Eval(Container.DataItem, "orderid") %>
     <hr>
     <%# DataBinder.Eval(Container.DataItem, "orderdate") %>
  </ItemTemplate>
  <SeparatorTemplate>
    <br>
  </SeparatorTemplate>
</asp:Repeater>

Technically speaking, the Repeater is an iterative control because it iterates over a data source and produces its output. Most data-bound controls follow this pattern, especially list controls. ASP.NET list controls derive from the ListControl base class and include controls such as CheckBoxList, ListBox, DropDownList, RadioButtonList (and BulletedList in ASP.NET 2.0). All of these create a particular type of list based on the records in the data source. Another control with "list" in its name, the DataList, is another story. Despite its name, the DataList is an iterative control that's a big brother to the Repeater. The two are roughly the same, except that the DataList has a richer set of built-in capabilities as far as templates and layout are concerned.

What do list controls like CheckBoxList and DropDownList have in common? And how do they differ from the Repeater? Both list and iterative controls output markup for every row of data bound to the control. Whereas list controls have a built-in rendering, iterative controls such as Repeater and DataList apply a custom and user-defined template that can be changed dynamically. This information will come in handy later.

So what kind of code would you use to create the page that takes a bunch of orders and renders them in a list? Figure 1 shows the simplest, and perhaps most natural, way of doing this.

Figure 1 Using a Repeater Control to Create a List (Close Figure 1)
<%@ Page Language="C#" %>
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>

<SCRIPT runat="server">
private void Page_Load(object sender, System.EventArgs e)
{
    if (!IsPostBack) LoadData();
}

public void LoadData()
{
    SqlDataAdapter adapter = new SqlDataAdapter("SELECT * FROM orders",
        "server=(local);database=northwind;Integrated Security=SSPI");
    DataTable table = new DataTable();
    adapter.Fill(table);
    Repeater1.DataSource = table;
    Repeater1.DataBind();
}
</SCRIPT>

<HTML>
  <HEAD>
    <title>ListControls #1</title>
  </HEAD>
  <body>
     <form id="Form1" method="post" runat="server">
        <asp:Repeater runat="server" id="Repeater1">
           <itemtemplate>
              <b><%# DataBinder.Eval(Container.DataItem, "orderid") %></b>
              <br>Due by: <%# DataBinder.Eval(Container.DataItem,
                      "orderdate", "{0:d}") %>
                  &nbsp;&nbsp;&nbsp;
                  Shipped: <%# DataBinder.Eval(Container.DataItem,
                      "shippeddate", "{0:d}") %>
           </itemtemplate>
           <separatortemplate>
              <hr>
           </separatortemplate>
        </asp:Repeater>
     </form>
   </body>
</HTML>

The page from Figure 1 works nicely but has at least two inherent drawbacks. One is that all the required template markup is incorporated in the ASPX source and is tightly coupled to the page. This is not an issue per se, but it will be when you want to reuse the code, and the solution based on it, elsewhere. In other words, using a handcrafted Repeater is hardly a repeatable operation without cut-and-paste techniques; therefore any contents based on the Repeater are not very reusable or extensible.

When you use the Repeater control to fill a page with templates, you actually build a solution based on two logical levels: the data repeater and the templates. The logic that repeats the templates and binds them to data is incorporated in the Repeater control; the template content is usually inline with the ASPX source. To reuse this solution on another page or in another application, the only viable approach is cutting and pasting the template content from page to page.

Before we look into practical solutions, an overview of the template properties in ASP.NET might be helpful.

Template Properties

Consider the ItemTemplate property of the Repeater control. According to the MSDN® documentation, the property gets or sets the template that defines how items in the control are displayed. The property is declared as follows:

public virtual ITemplate ItemTemplate {get; set;}
It is a read/write overridable property of type ITemplate. When processing a page, if the ASP.NET page parser finds an inline template, it creates an instance of a template builder and makes it return an ITemplate object. Figure 2 shows a sample of the code that the page code generator creates for the header of a Repeater.

Figure 2 Header of a Repeater Control (Close Figure 2)
// Instantiate the Repeater object
Repeater __ctrl = new Repeater();
Repeater1 = __ctrl;
__ctrl.ID = "Repeater1";

// Build the template for the header
BuildTemplateMethod _builder;
_builder = new BuildTemplateMethod(__BuildControl__control3);
CompiledTemplateBuilder _template = new CompiledTemplateBuilder(_builder);
__ctrl.HeaderTemplate = _template;

... //  Later in the dynamically generated class file control

void __BuildControl__control3(Control __ctrl)
{
    IParserAccessor __parser = (IParserAccessor) __ctrl;
    LiteralControl _header = new LiteralControl(
        "text of <headertemplate>");
    __parser.AddParsedSubObject(_header);
}

The CompiledTemplateBuilder class is listed in the ASP.NET documentation, but is described as supporting the Microsoft® .NET Framework and should not be used from your code directly. It inherits from the supported class TemplateBuilder, though. The same holds true for the BuildTemplateMethod delegate:

public delegate void BuildTemplateMethod(Control control);

The constructor of the CompiledTemplateBuilder class takes an instance of the delegate and calls it back to build a template class based on the source of the specified control. What's the _control3 in the code of Figure 2? It's simply the literal control that contains all the source code of the <HeaderTemplate> tag.

Okay, so there should be a way to create templates programmatically, but what's the fastest and easiest way? The simplest way is resorting to user controls.

User Controls

A user control is primarily a form made of any combination of server and client controls sewn together with server and client script code. A user control has a rich user interface and can expose an object model built on top of contained controls. In ASP.NET user controls derive from the UserControl class and do not implement ITemplate. So how can user controls provide a template?

The Page class inherits from its base TemplateControl class a method named LoadTemplate, which just obtains an instance of the ITemplate interface from an external ASCX user control file. You specify the virtual path to the ASCX file and the method does the rest. In particular, LoadTemplate first gets an absolute path to the ASCX control and then compiles the user control to a dynamic class. The obtained class is then wrapped by a built-in simple template class that doesn't do much more than implement ITemplate. Modify the code in Figure 1 to move the template definition to a separate ASCX file. Next, try the following code:

Repeater1.ItemTemplate = LoadTemplate("ucitemtemplate.ascx");
Repeater1.DataSource = table;
Repeater1.DataBind();

You shouldn't notice anything different in the page's markup, but this time the item template is being generated from an external compiled resource. A little change is required to make the item template of Figure 1 work unchanged as a distinct ASCX component. The object of the change is the DataBinder expression:

DataBinder.Eval(Container.DataItem, "OrderId");
Container.DataItem returns the data item object bound to the Repeater's item being rendered. From within a Repeater instance, Container evaluates to a RepeaterItem object where DataItem is a valid property. When used from within a user control, though, Container is just a Control and the DataItem property doesn't get defined. There are two ways to solve this. One is casting Container to RepeaterItem; the other is moving the DataItem name up into the navigation path of the data binding expression:
DataBinder.Eval(Container, "DataItem.OrderId");

In this way, the same object is retrieved as expected, but DataItem is not treated like a string and is accessed through reflection. Both approaches lead you to the same result; the second provides a more reusable solution since you don't need to know what kind of iterative control is calling the user control.

Now add the following instruction immediately after assigning the ItemTemplate property:

Response.Write(Repeater1.ItemTemplate.ToString());

What do you expect the real type of ItemTemplate to be? UserControl? Or is it more likely to be a dynamic class name generated after the URL of the ASCX? The name of the template class, actually, is the following:

System.Web.UI.TemplateControl+SimpleTemplate
SimpleTemplate is an internal nested type:
public class TemplateControl {
   ...
   internal class SimpleTemplate : ITemplate {
      ...
   }
}

SimpleTemplate is supplied with the Type of the user control. When instantiated within a host control, it simply creates an instance of the user control and adds it to the host site.

The solution based on user controls is sadly not problem-free. You need a new user control for each different template you want to generate dynamically. In many cases, you have a proliferation of relatively small and simple user controls that must be deployed individually and duplicated to reuse them in other projects.

Implementing the ITemplate Interface

As mentioned, the ITemplate interface is the only requirement for a template property. Both the classic inline template and the user control trick are mere workarounds that create an underlying object that implements ITemplate. If reusability is not an issue, line templates are a pretty good solution. If you want a bit of reusability and flexibility but don't see deployment as an issue, you can opt for user controls. In all other cases, you should read on and consider writing your own class that implements ITemplate.

The main benefit to you when implementing ITemplate is simplified deployment and full reusability of the Repeater-based solution. Let's see how to rewrite the previous example with ITemplate classes. But first, a brief overview of the interface is in order:

public interface ITemplate
{
   void InstantiateIn(Control container);
}

The interface has only one method—InstantiateIn—which returns void and takes a Control parameter. The control passed to InstantiateIn defines the Control object to which child controls in the template will be parented. These child controls can be defined within an inline template or a user control, or created programmatically within a new class.

The goal of InstantiateIn is quite simple. It has to create and add any needed control to the specified container control. If data binding is required on any of these controls, the DataBinding event must be hooked up too. The following class creates a header template that simply displays the word "Orders" in H1 style:

public class OrderHeaderTemplate : ITemplate
{
   public void InstantiateIn(Control container)
   {
      string txt = "<h1>Orders</h1>";
      LiteralControl h1 = new
          LiteralControl(txt);
      container.Controls.Add(h1);
   }
}
It does so by using a literal control. Note that the header template is not data-bound, meaning that no data from the data source is visible at this point for binding. The following line shows you how to connect the previous type with the HeaderTemplate property of a Repeater:
Repeater1.HeaderTemplate = new
    OrderHeaderTemplate();

Defining the item template is a little bit more complicated because it is data-bound. Let's take a look at the example in Figure 3. The template to reproduce determines the order ID on a row, followed on the next row by the order and the shipment dates. A few static literals complete the template. The order ID is added as a Label. There's no reason to use a Label in lieu of a literal as far as pure text is concerned. Here I opted for a Label simply because it provides a Font property which makes it easy to display the order ID in bold type.

Figure 3 Sample Repeater Item Template Using a Custom Class (Close Figure 3)
public class OrderItemTemplate : ITemplate
{
   public void InstantiateIn(Control container)
   {
      Label header = new Label();
      header.Font.Bold = true;
      header.DataBinding += new EventHandler(OrderHeader_DataBinding);
      container.Controls.Add(header);

      container.Controls.Add(new LiteralControl("<br>Due by: &nbsp;"));

      LiteralControl orderDate = new LiteralControl();
      orderDate.DataBinding += new EventHandler(OrderDate_DataBinding);
      container.Controls.Add(orderDate);

      container.Controls.Add(
          new LiteralControl("&nbsp;&nbsp;&nbsp;Shipped:&nbsp;"));

      LiteralControl orderShipped = new LiteralControl();
      orderShipped.DataBinding +=
          new EventHandler(OrderShipped_DataBinding);
      container.Controls.Add(orderShipped);
   }

   private void OrderHeader_DataBinding(object sender, EventArgs e)
   {
      Label header = (Label) sender;
      RepeaterItem container = (RepeaterItem) header.NamingContainer;
      header.Text = DataBinder.GetPropertyValue(
          container.DataItem, "orderid").ToString();
   }

   private void OrderDate_DataBinding(object sender, EventArgs e)
   {
      LiteralControl orderDate = (LiteralControl) sender;
      RepeaterItem container = (RepeaterItem) orderDate.NamingContainer;
      orderDate.Text = (string) DataBinder.GetPropertyValue(
          container.DataItem, "orderdate", "{0:d}");
   }

   private void OrderShipped_DataBinding(object sender, EventArgs e)
   {
      LiteralControl orderShipped = (LiteralControl) sender;
      RepeaterItem container =
          (RepeaterItem) orderShipped.NamingContainer;
      orderShipped.Text = (string) DataBinder.GetPropertyValue(
          container.DataItem, "shippeddate", "{0:d}");
   }
}

In addition to the insertion of the control in the existing container's context, the key thing that is going on here is the DataBinding event handler. A DataBinding handler is needed for each control that would require DataBinder.Eval in the inline template. First, you save a reference to the sender through the sender parameter of the event handler. The sender is the control on behalf of which you're handling the DataBinding event. The NamingContainer property of the sender indicates the nearest control that implements INamingContainer. The nearest container for a child control generated within a Repeater's row is just the RepeaterItem object of the corresponding row. Getting this reference is essential, as it is the only way to access the bound data.

The RepeaterItem type has a property named DataItem. This property is DataRowView in the case of a bound DataView; it is String in the case of a bound string collection. To get the bound value for a particular field, use either DataBinder.Eval or DataBinder.GetPropertyValue. In general, Eval is more flexible as it can retrieve both individual and indexed properties. The Eval method has two public overloads, as shown here:

public static string Eval(object container, string expression)

public static string Eval(
    object container, string expression, string format)

Figure 4 ASP.NET Repeater
Figure 4 ASP.NET Repeater

While building a custom class for each template, you can define as many constructors as you need, add to them as many parameters as required, and in general customize the class as you like with extra methods, properties, and events. Figure 4 shows a sample page in action powered by custom classes that implement the ITemplate interface.

List Controls

The final option is list controls. List controls inherit from the abstract class ListControl and provide a data-bindable, fixed rendering mechanism. Using a list control is considerably easier than setting up an iterative control. Defining templates requires quite a bit of declarative code or, if accomplished programmatically, requires that you write a class that implements ITemplate.

A list control takes its data from the DataSource property; other properties are used to select particular columns of data for personalized rendering. For example, consider the DropDownList control. It renders out to HTML as follows:

<select>
   <option value="...">...</option>
    ...
</select>

The value attribute is expected to be a unique value in the collection of items; the inner text of the <option> tag contains the displayed text for the item. These two elements can be bound to data columns through DataValueField and DataTextField, respectively. The list controls CheckBoxList and RadioButtonList provide more formatting options, such as the maximum number of columns or rows to show and the repeat direction. To write a custom list control, you simply need to extend the base ListControl class (to take advantage of the RepeatInfo class's default rendering, you can also implement IRepeatInfoUser). Let's take a closer look.

A list control is based on the idea of a repeated control. The repeatable control doesn't have to be a single, individual control; it can also be a tree of controls with a common root. Typically, you implement this control through a private property, ControlToRepeat. Figure 5 shows an example for a button list control.

Figure 5 Button List Control (Close Figure 5)
private Button ControlToRepeat
{
   get
   {
      if (_controlToRepeat == null)
      {
         _controlToRepeat = new Button();
         _controlToRepeat.EnableViewState = false;
         Controls.Add(_controlToRepeat);
      }
      return _controlToRepeat;
   }
}

In addition, you expose a few public properties to let users control the layout of the control: RepeatDirection, RepeatColumns, and RepeatLayout. These properties have the same meaning as they do in DataList or CheckBoxList controls.

The Render method of the control takes into account the control's repeating capabilities, as shown in Figure 6. The Render method builds an instance of the RepeatInfo class and passes it a reference to the control itself. The method RenderRepeater of the RepeatInfo structure ends up calling the members of the IRepeatInfoUser interface of the control and finally provides the expected behavior. Figure 7 shows the sample ButtonList control in action in the Visual Studio® .NET designer.

Figure 7 ButtonList in the Visual Studio .NET Designer (Close Figure 7)

Figure 6 IRepeatInfoUser (Close Figure 6)
public bool HasHeader { get { return false; } }

public bool HasSeparators { get { return false; } }

public bool HasFooter { get { return false; } }

public Style GetItemStyle(System.Web.UI.WebControls.ListItemType itemType,
    int repeatIndex) {
    return null;
}

public void RenderItem(System.Web.UI.WebControls.ListItemType itemType,
    int repeatIndex, RepeatInfo repeatInfo, HtmlTextWriter writer) {
    ButtonItem item = Items[repeatIndex];
    Button btn = ControlToRepeat;
    btn.Text = item.Text;
    btn.CommandName = item.CommandName;
    btn.RenderControl(writer);
}

public int RepeatedItemCount { get { return this.Items.Count; } }

public virtual RepeatDirection RepeatDirection {
    get
    {
        object o = ViewState["RepeatDirection"];
        if (o != null)
            return (RepeatDirection)o;
        return RepeatDirection.Vertical;
    }
    set { ViewState["RepeatDirection"] = value; }
}

public virtual RepeatLayout RepeatLayout {
    get
    {
        object o = ViewState["RepeatLayout"];
        if (o != null)
            return (RepeatLayout)o;
        return RepeatLayout.Table;
    }
    set { ViewState["RepeatLayout"] = value; }
}

public virtual int RepeatColumns {
    get
    {
        object o = ViewState["RepeatColumns"];
        if (o != null)
            return (int)o;
        return 0;
    }
    set { ViewState["RepeatColumns"] = value; }
}

protected override void Render(HtmlTextWriter writer) {
    if (Items.Count >0) {
        RepeatInfo ri = new RepeatInfo();
        Style controlStyle = (base.ControlStyleCreated ?
            base.ControlStyle : null);
        ri.RepeatColumns = RepeatColumns;
        ri.RepeatDirection = RepeatDirection;
        ri.RepeatLayout = RepeatLayout;
        ri.RenderRepeater(writer, this, controlStyle, this);
    }
}

protected override Style CreateControlStyle() {
    return new TableStyle(this.ViewState);
}

ItemCreated Event

When a RepeatItem is created in the Repeater control, the ItemCreated event fires. The event simply signals the creation of the element; it says nothing about the data associated with the element. Data binding the item, in fact, happens through the ItemDataBound event. ItemCreated and ItemDataBound are extremely useful for modifying the control style dynamically to, say, alert users that values are at a critical threshold. Default list controls, though, have no such events. If you ever tried to make one particular checkbox in a CheckBoxList collection bold or italic, you probably know what I mean.

To finish with a flourish, let's see how to add a ButtonCreated event to the ButtonList control. First, declare a custom delegate:

public delegate void ButtonCreatedHandler(
    object sender, ButtonCreatedEventArgs e);
Next, create the data structure to hold event data:
public class ButtonCreatedEventArgs : EventArgs {
   public Button ControlToRepeat;
   public int ButtonIndex;
}
You pass the event handler the index of the button being created and a reference to the button itself. This way, users can make last-minute changes quite easily. The event is declared as follows:
public event ButtonCreatedHandler ButtonCreated;

Where do you fire the event? If you look at the behavior of the Repeater control, the ideal launch time is when each button is created. Unlike Repeater and other iterative controls, though, which create a hierarchy of child controls before rendering, list controls create their own children directly at rendering time. In other words, list controls don't really have distinct moments for the creation, data binding, and rendering of the child controls. Suppose you catch the ButtonCreated event in an ASP.NET page and then write the following code:

void ButtonList1_ButtonCreated(
    object sender, Samples.ButtonCreatedEventArgs e)
{
   if (e.ControlToRepeat.CommandName == "Test")
      e.ControlToRepeat.BackColor = Color.Yellow;
}
All buttons have a yellow background, not just the one with the given command name. What's up? The trick is the ControlToRepeat private property that I talked about a bit earlier. That property represents the main piece of the list control's user interface. For better performance, a single instance of the control is used—the one stored in the ControlToRepeat property. When it comes to implementing the ButtonCreated event, though, that apparently positive feature turns into a negative. Because all buttons are rendered through the same Button object, any changes made at a certain point in the flow propagate to all subsequent buttons in the list. To avoid that, you need to handle buttons separately (see Figure 8). To see the difference, compare the implementations of the RenderItem method in Figure 5 and Figure 8.

Figure 8 Firing the ItemCreated Event from a List Control (Close Figure 8)
public void RenderItem(
    ListItemType itemType, int repeatIndex, RepeatInfo repeatInfo,
    HtmlTextWriter writer)
{
    // Prepare the button to render—new button
    ButtonItem item = Items[repeatIndex];
    Button btn = new Button();
    btn.Text = item.Text;
    btn.CommandName = item.CommandName;

    // Fire the ItemCreated event
    if (ItemCreated != null)
    {
        ButtonCreatedEventArgs bcea = new ButtonCreatedEventArgs();
        bcea.ItemIndex = repeatIndex;
        bcea.Item = Items[repeatIndex];
        bcea.ControlToRepeat = btn;

        ItemCreated(this, bcea);
    }

    // Render the control
    btn.RenderControl(writer);
}

When to Use Which

There are many ways to repeat controls and templates in ASP.NET pages. If you want to be able to switch templates at run time or deploy using different views of the same content, then user controls are just perfect. For a quick and page-specific solution, nothing is better than inline code. You should use ITemplate classes or list controls when you have a custom but fixed pattern to repeat for each bound record and want this to be reusable in the full sense of the word—as if it were a true custom control. Whether an ITemplate class or a custom list control will work better for you depends on other factors. For example, how important are a control's layout capabilities? Would you need to let users choose a horizontal or vertical view? Or a maximum number of rows or columns? If you answered yes to any of these quick questions, a custom list control should be your first choice.

The RepeatInfo class provides layout capabilities to IRepeatInfoUser implementations, but the Repeater doesn't use it. On the other hand, implementing the interface is much easier in a list control where you control the whole binding mechanism. (This could also be done in a control that inherits from the Repeater class, but is a task for brave hearts only.) List controls are powerful, but often overlooked. In ASP.NET 2.0, building list controls is even easier than in ASP.NET 1.x.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the new book Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Advanced Basics: Scaling Up: The Very Busy Background Compiler

Advanced Basics
Scaling Up: The Very Busy Background Compiler
Matthew Gertz

Contents
Rich Information Leads to Rich Features
Scaling Up the Background Compiler
Performance Tips for Earlier Versions
Going Forward

One of the features that distinguishes Visual Basic® from the other languages in Visual Studio® is its use of a background compiler (BC). The BC runs from the moment you start up a Visual Basic project until the time you close the last one down. Whenever you type in a line of code and commit it, the BC picks up the change and compiles it, adding the resulting information to the internal representation of the application you're building. The BC reacts to external changes as well, keeping its compiled state up-to-date as you add or remove project references, change project-level settings (such as Option Strict or Option Explicit), or check out a more recent copy of a project file from source control.

The most obvious benefits of the BC are that you can launch into debugging or build an executable for distribution quite quickly. Because the application is already "built," all that remains is for you to load it into the debugger or drop it to disk. But the BC also has a much broader impact on your development process in Visual Studio that is not so obvious—nearly all of the Visual Basic IDE features in Visual Studio .NET 2002 and Visual Studio .NET 2003 rely on the BC for information. Without the BC, these features would either not function at all or would be seriously diminished.

This added functionality does not come without costs, of course. Since the Visual Basic UI features generally need information from the compiler in order to function, the UI will occasionally block on background compilation in order to wait for a complete representation of the symbols. In general this delay is not noticeable, but for large solutions the delay can be quite apparent, especially when working with large blocks of code all at once. This trade-off between functionality and performance has been a focus of the Visual Studio development team. Several of the improvements resulting from their efforts are found in Visual Studio 2005.

In this column, I'll discuss how the BC is used by the various Visual Basic Editor features, what Microsoft has done to improve their performance, and steps you can take to minimize the impact of the BC in your development projects.

Rich Information Leads to Rich Features

The reason that the Visual Basic Editor features can provide you with such rich information is that they all have access to the internal compiler symbols generated by the BC. For example, IntelliSense® can distinguish between two similarly named variables in different or nested scopes, regardless of whether or not they exist in the current project or in some other part of the solution, because the compiler makes it very clear which one is which. Without that information, IntelliSense would have to parse text and make guesses about the type to which a particular variable binds.

Different editor features require different levels of information from the background compiler. Because of this, the BC exposes methods by which a given feature can wait for a specific state to be reached before proceeding, without having to wait for a complete compilation. Figure 1 lists the different states of the compiler.

Figure 1 Compiler States (Close Figure 1)

State | Description
No State | Essentially, no compilation information is available at this time.
Declared | Everything is parsed and it is clear which code is a keyword and which code is a variable. The boundaries between blocks of code are also clearly delineated. Although it is clear that something is a variable, its type is not understood yet, nor is it known whether it's the same as some similarly named variable.
Bound | In this state it is clear what the variables mean in the context of other variables and references.
Types Emitted | A more esoteric state, but the upshot is that now a variety of metadata about the to-be-compiled app can be made available. (From a UI point of view, this is generally not an important state.)
Compiled | All of the intermediate language is generated and ready to go.

For example, pretty-listing, auto-complete, and smart indentation simply require the parsing to be done in order to fully understand what the code looks like, and so they only require information that is generated during the transition to the Declared state (and don't even need to be fully at Declared to operate correctly). IntelliSense always requires "bound" information because it needs to fully understand what the various types mean. However, IntelliSense works on whatever information was generated during the last "commit" (carriage return, arrow down, paste, focus change, and so on), and so IntelliSense is not blocked from operating while typing. Error generation can happen in all of the states, depending on the type of error, and is a consequence of moving to a different state unless compilation is changed or restarted for any reason. But the listing of errors to the task list or error list is specifically coded to wait for keystrokes to end if the user continues typing (since you would just be recompiling anyway on the next commit), so again, there's no blockage. Figure 2 provides some examples of features and the compiler states that they rely on.

Figure 2 Some Compiler Features
Figure 2 Some Compiler Features

To improve performance and functionality, the members of the compiler team sometimes modify existing features to use information from the BC. In Visual Studio .NET 2003, dropdowns, the object browser, and the class view began using the BC information. Previously, in Visual Studio .NET 2002, these features were supported by a slow XML-based library manager that was essentially duplicating what the compiler was doing (with much less accuracy). In switching over to compiler symbols, the performance and accuracy of those features dramatically improved. Based on this success, the team (the Visual Basic Compiler, Editor, and Debugger team) also changed the code model engine to use the BC information in Visual Studio 2005 and this has resulted in substantial savings for the new class designer and various forms and macros. All of these features now require "bound" information, since they need to know the precise identity of an object in order to function correctly.

New features are also being developed to take advantage of the information in the BC. Error correction is a brand new feature for Visual Basic 2005 that uses compiler information to suggest fixes to the user. Although errors can happen at any state, the current set of error corrections relate to errors generated during the Declared and Bound compilations. Therefore, error correction is dependent on those states—first to recognize that there are indeed errors, and then to determine which fixes to suggest. Edit and Continue, which is being reintroduced in Visual Basic 2005, relies heavily on compiler information to detect which edits would require a reset of the debugger, as well to generate the actual code to be injected and verify that it creates no errors before committing it. The Find Symbol and Rename Symbol features, which are also new in Visual Basic 2005 rely on Bound state information to correctly identify symbols.

Scaling Up the Background Compiler

One of the biggest changes the team made involves how we handle decompilation—that is, resetting the compiler to an earlier state based on some change that the customer has made. Decompilation, followed by recompilation, is required to some extent anytime a change is made to the project, but sometimes it can be taken too far. For example, deleting a resource file should not take your project down to No State—the information collected through Bound state is still going to be valid. Although decompiling to No State on a change is always totally safe, too much decompilation can severely impact the performance of the compiler. Visual Studio .NET 2002 sometimes does too much decompilation, and therefore runs into this problem. In order to address it, the team made major modifications all across our architecture to be more efficient about when we decompile, and how far we need to go when we do. For example, sometimes we decompile to No State in Visual Studio .NET 2002 when you add another carriage return after an End Sub, under the assumption that you've modified the procedure (since the change was on the boundary of the procedure and may have redefined it). That issue, along with many others, was fixed in Visual Studio .NET 2003, so when making sweeping changes in large solutions, the response is much faster.

Many more changes of this sort have been made for Visual Studio 2005 along with more streamlined handling of references to reduce startup and operation time.

Performance Tips for Earlier Versions

Now you may be wondering how to improve the performance of your Visual Basic .NET 2002 and Visual Basic .NET 2003 development sessions. While you can't turn off the background compiler there are other steps you can take to improve performance.

Probably the most important step is to avoid complicated references. For any set of projects in a solution that reference each other, the BC needs to wait until all projects are at the same state before moving onto the next one. Thus, if project A references project B, which references C, which references D, and changes are made to D, then A, B, and C will all need to be decompiled to some level in order to incorporate the new information from D. More complicated dependency graphs cause even further delays.

The way you reference code in other assemblies is also important. Visual Basic supports two kinds of references (excluding Web references, which are another issue entirely): file references, which are simply references to assemblies on disk, and project-to-project references, which are references to another project within the same solution. In Visual Studio .NET 2002 and 2003, both types of references can be added by right-clicking on the References node in the Solution Explorer and selecting Add Reference. This will bring up the Add Reference dialog which has tabs for .NET file references, COM file references, and project references (see Figure 3).

Figure 3 Add Reference Dialog
Figure 3 Add Reference Dialog

You should use project-to-project references only when you are developing both the application's project and its referenced project at the same time. If, however, you have already finalized a referenced assembly, then a file reference to it would be a more appropriate reference type to use, since you'd have no need for the BC to even be thinking about recompiling the reference—in fact, there's no need for the source of the referenced assembly to be in the solution at all. (Note that, in Visual Studio 2005 the BC is much more clever about handling project-to-project references, but as a rule it's going to be faster when you don't load another project unnecessarily.)

I don't like to recommend that people turn off features to improve performance (and my team is working hard to ensure that that's never necessary), but in large-scale applications you can turn off whatever features you don't need to use at that moment. For example, you may not need the pretty-lister when you quickly need to modify large blocks of code (pasting a large chunk of code into it or modifying large if blocks, for instance). Most of these features don't require the BC to even be at Declared state, so technically this is not a BC issue, but these other operations do take some processing time. The various editing options can be controlled from Tools | Options. IntelliSense is controlled from the General page (where it is called Statement Completion, as shown in Figure 4).

Figure 4 IntelliSense Options
Figure 4 IntelliSense Options

Smart indentation (the only indentation choice that relies on Declared information) can be changed from the Tabs page, shown in Figure 5. Pretty-listing and other code-formatting options can be controlled from the Visual Basic page, shown in Figure 6.

Figure 5 Smart Indenting in Tabs Page in Tools
Figure 5 Smart Indenting in Tabs Page in Tools

Avoiding situations that result in decompilation is also a big time-saver. Decompilation in Visual Basic happens per-file, which generally translates to per-class for most people, and so the more referenced code that you have between your various classes and structures in different files, the more likely it is that decompilation will occur. (Note that Visual Studio 2005 will support partial classes—that is, classes defined across multiple files—and in such cases all relevant files may need to decompile to update a given class.) A general rule of thumb is to organize classes and structures so that the info requests are all one way—when class A makes a request on class B, class B shouldn't have to call back into class A to complete its job (which would create a dependency on the compilation state of A's code).

Figure 6 Visual Basic Editing Features
Figure 6 Visual Basic Editing Features

The factoring of your code is another important consideration for editor performance, as certain features (such as the pretty-lister) are geared towards well-factored code. Adding or removing an if construct that is 5,000 lines of code is definitely going to slow down the pretty-lister, as the indentations and scopes for the altered block of code must be recalculated and relisted in the editor. (The compiler team actually disabled pretty-listing at about 8000 affected lines for this reason, even if the pretty-listing option was turned on.) Well-factored code will always respond better than poorly factored code, since there will be less affected code in any block when listing back (especially important in those rare cases when the pretty-lister needs "bound" information).

Going Forward

High-performance code is a journey for my team, not a destination, and although we've made great strides in improving the performance of the BC and the features affected by it, both in Visual Studio .NET 2003 and in the upcoming Visual Studio 2005 release, we are continuing to research and implement improvements in our codebase. The goal is to make the performance implications of background compilation so minimal that users don't even realize it's happening. Ideas we're exploring include delay-starting of the BC (to improve load time for solutions), tightening up decompilation so that even less code needs to be reset on any given change, and better caching of information that is likely not to change during compilation/decompilation (removing the need to block on state in certain cases).

Send your questions and comments to basics@microsoft.com.

Matthew Gertz has worked at Microsoft for over 10 years as a developer on ActiveX Control Pad, Visual InterDev, Visual Studio Enterprise Edition, and Visual Basic. He is currently the Development Lead for the Microsoft Visual Basic Compiler, Editor, and Debugger team.


Data Points: XML Features in SQL Server 2000

Data Points
XML Features in SQL Server 2000
John Papa

Code download available at: DataPoints0506.exe (152KB)
Contents
Returning XML
Transforming and Hierarchies
Returning XML via ADO.NET
Working with FOR XML Restrictions
OPENXML
Inserts via OPENXML
Inserts and Updates
Bulk Inserts via OPENXML
Using the Sample
Wrapping Up

SQL Server™ 2000 includes several XML features that let you transform relational rowsets into hierarchical XML documents, read XML documents, and bulk load data via XML. For example, you can pass an XML document to a stored procedure, join the XML to some tables and return a rowset, or even modify data in the database. The inclusion of the OPENXML function and the FOR XML statement have been prompted by the ever-expanding role of XML in today's enterprise systems. Some of these features not only support XML, but also offer improved performance when loading data in bulk.

In this month's column, I will discuss how you can return XML from SQL Server via T-SQL's FOR XML clause. I'll walk through examples that show the various ways you can return XML data and schema information, and show how to transform the XML to a more palatable format. Then I will discuss OPENXML, how to join an XML document to database tables, and how to pull XML out of a DataSet using the WriteXml and GetXml methods. The SQL for the examples as well as a sample ASP.NET project that executes some of the examples and streams them to text files are all in the download available from the MSDN®Magazine Web site. The sample project also includes code to insert and update records to the database from XML.

Returning XML

When used in a SELECT statement, the FOR XML clause tells SQL Server to return the data as XML as opposed to a standard rowset. You can specify the mode: RAW, AUTO, or EXPLICIT. Each offers a different way to transform the XML (Figure 1 shows a quick overview of each mode).

Figure 1 Overview of the FOR XML Modes (Close Figure 1)

Mode | Description
RAW | Each record in the rowset is transformed to an XML element named row. The <row> elements will contain an attribute that represent each column that is retrieved.
AUTO | Records in the rowset can be transformed into nested XML elements named after the tables in the FROM clause. Each column that is retrieved will be represented as an attribute.
EXPLICIT | You have a lot of control over how the XML is formatted. However, the syntax to use the EXPLICIT mode is much more involved. XSLT is a much more common and accepted practice of transforming XML.

For example, if you query the Northwind database's Employees table using FOR XML RAW, it would return each employee row within a <row> element. Each column that you included in the SELECT statement would be represented as an attribute on the <row> element. The following FOR XML RAW query selects two employee records and then returns them in the RAW format:

SELECT EmployeeID,
    FirstName, LastName
FROM Employees
WHERE LastName LIKE 'D%'
FOR XML RAW

<row EmployeeID="1" FirstName="Nancy" LastName="Davolio"/>
<row EmployeeID="9" FirstName="Anne" LastName="Dodsworth"/>

The same SELECT statement could be modified to use the FOR XML AUTO clause instead. This time the elements are named Employees, matching the name of the source table. The columns are still attributes of the main elements:

SELECT EmployeeID, FirstName, LastName
FROM Employees
WHERE LastName LIKE 'D%'
FOR XML AUTO

<Employees EmployeeID="1" FirstName="Nancy" LastName="Davolio"/>
<Employees EmployeeID="9" FirstName="Anne" LastName="Dodsworth"/>

Transforming and Hierarchies

While the differences in the previous example are slight, the differences between AUTO and RAW are more pronounced when used with a query that joins tables. A query that uses FOR XML RAW will only return <row> elements whether the data comes from one or more tables. Therefore, the RAW mode does not take advantage of the inherent hierarchical structure of XML documents. Imagine the following SQL statement:

SELECT   Customers.CustomerID,
         CompanyName,
         OrderID,
         CONVERT(VARCHAR(10), OrderDate, 101) AS OrderDate
FROM     Customers
INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID
ORDER BY Customers.CustomerID

In this example I want to retrieve a one-to-many parent-child relationship. If I execute this SQL statement, I will get back a series of customers and their corresponding orders. If I throw on the FOR XML RAW clause and execute it once again, the resulting XML will contain a single <row> element representing each row that is returned. For example, the XML data in Figure 2 represents the rows that the FOR XML RAW will return where the CustomerID is ALFKI.

Figure 2 The XML Data (Close Figure 2)
<row CustomerID="ALFKI" CompanyName="Alfreds Futterkiste" OrderID="10643" OrderDate="08/25/1997" />
<row CustomerID="ALFKI" CompanyName="Alfreds Futterkiste" OrderID="10692" OrderDate="10/03/1997" />
<row CustomerID="ALFKI" CompanyName="Alfreds Futterkiste" OrderID="10702" OrderDate="10/13/1997" />
<row CustomerID="ALFKI" CompanyName="Alfreds Futterkiste" OrderID="10835" OrderDate="01/15/1998" />
<row CustomerID="ALFKI" CompanyName="Alfreds Futterkiste" OrderID="10952" OrderDate="03/16/1998" />
<row CustomerID="ALFKI" CompanyName="Alfreds Futterkiste" OrderID="11011" OrderDate="04/09/1998" />

Notice that the data is not laid out in a parent-child hierarchy. If I wanted the data to appear as a series of <Customers> elements that contain a series of their associated <Orders> elements, I could use the FOR XML AUTO clause instead (the nesting of the parents and their children depends on the parent rows being grouped together). The resulting XML for the CustomerID of ALFKI would look like the following:

<Customers CustomerID="ALFKI" CompanyName="Alfreds Futterkiste">
  <Orders OrderID="10643" OrderDate="08/25/1997" />
  <Orders OrderID="10692" OrderDate="10/03/1997" />
  <Orders OrderID="10702" OrderDate="10/13/1997" />
  <Orders OrderID="10835" OrderDate="01/15/1998" />
  <Orders OrderID="10952" OrderDate="03/16/1998" />
  <Orders OrderID="11011" OrderDate="04/09/1998" />
</Customers>
This XML is much more readable since it uses the names of the tables as the element names. It also contains less data since it does not repeat the CustomerID and CompanyName attributes for every Orders element, as does the FOR XML RAW example.

If you prefer the values of the columns to be presented as elements instead of attributes, then you are in luck. By specifying the ELEMENTS option in the FOR XML clause, all column values will become elements within the XML. Sometimes seeing is believing, so I have shown the output of the previous query when the ELEMENTS condition is applied in Figure 3. (For brevity, the XML samples I show include only the XML for the CustomerID ALFKI. The actual XML generated by these queries would contain the XML for all customers and their orders.) The query that uses the ELEMENTS option looks like this:

\SELECT   Customers.CustomerID,
         CompanyName,
         OrderID,
         CONVERT(VARCHAR(10), OrderDate, 101) AS OrderDate
FROM     Customers
INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID
ORDER BY Customers.CustomerID
FOR XML AUTO, ELEMENTS

Figure 3 Using FOR XML AUTO, ELEMENTS (Close Figure 3)
<Customers>
    <CustomerID>ALFKI</CustomerID>
    <CompanyName>Alfreds Futterkiste</CompanyName>
    <Orders>
        <OrderID>10643</OrderID>
        <OrderDate>08/25/1997</OrderDate>
    </Orders>
    <Orders>
        <OrderID>10692</OrderID>
        <OrderDate>10/03/1997</OrderDate>
    </Orders>
    <Orders>
        <OrderID>10702</OrderID>
        <OrderDate>10/13/1997</OrderDate>
    </Orders>
    <Orders>
        <OrderID>10835</OrderID>
        <OrderDate>01/15/1998</OrderDate>
    </Orders>
    <Orders>
        <OrderID>10952</OrderID>
        <OrderDate>03/16/1998</OrderDate>
    </Orders>
    <Orders>
        <OrderID>11011</OrderID>
        <OrderDate>04/09/1998</OrderDate>
    </Orders>
</Customers>

Returning XML via ADO.NET

I have included with the downloadable sample code an ASP.NET project that allows you to run some of the code samples (shown in Figure 4). The sample for this project uses a method called GetFORXML that runs any SQL with a FOR XML clause and writes the resulting XML to a file. The XML is also wrapped within a <root> element for simplicity.

Figure 4 Using ADO.NET to Get XML from SQL Server
Figure 4 Using ADO.NET to Get XML from SQL Server

The code for the GetFORXML method, as shown in Figure 5, demonstrates how you can use the standard SqlCommand object to execute the SQL statement containing the FOR XML clause. I used the ExecuteXmlReader method so I could return the XML as an XmlReader object. The DataSet's ReadXml method accepts the XmlReader, with the second argument indicating that the XML is a fragment. Because it is a fragment, it will only load all of the Customer nodes if it has a schema to match. So prior to loading the XML, I load the schema (inferred in this case) using the ReadXmlSchema method. Finally, I set the DataSetName property of the DataSet to "root", which will result in the read XML being enclosed within a parent tag <root/>.

Figure 5 Traversing XML with the XmlTextReader (Close Figure 5)
private string GetFORXML(string sFile, string sSQL) {
    // Create and open the connection to Northwind
    using(SqlConnection oCn = new SqlConnection(this.sCnNW)) {
        oCn.Open();

        // Create the SQL command to execute
        SqlCommand oCmd = new SqlCommand(sSQL, oCn);

        DataSet oDs = new DataSet();

        // Execute the SQL statement and return the data to an XmlReader.
        // Then read the schema and the fragment
        XmlReader oXml = oCmd.ExecuteXmlReader();
        oDs.ReadXmlSchema(oXml);
        oDs.ReadXml(oXml, XmlReadMode.Fragment);

        oCn.Close();

        oDs.DataSetName = "root";
        oDs.WriteXml(this.sPath + sFile);
        return "Wrote XML to file " + sFile;
    }
}

The ASP.NET page in Figure 4 can execute and output the XML for the examples I have reviewed thus far and can also get employee data using FOR XML RAW, BINARY BASE64. Plus, it can execute and output XML to get customer and order data using:

  • FOR XML RAW
  • FOR XML AUTO
  • FOR XML AUTO, ELEMENTS
  • FOR XML AUTO, ELEMENTS, XMLDATA

Working with FOR XML Restrictions

Neither FOR XML AUTO nor FOR XML RAW offer complete support for returning binary data. For example, if you try to select a binary field (such as the Employees.Photo column) and use the FOR XML RAW, you will receive a nasty error message. One way around this is to return a URL to the binary data field. This requires you to have a SQL Server virtual directory set up in IIS first. Another option, which does not require an IIS configuration, is to specify the BINARY BASE64 option in the FOR XML clause, as shown in the following code:

SELECT EmployeeID, FirstName, LastName, Photo
FROM Employees
WHERE LastName LIKE 'D%'
FOR XML RAW, BINARY BASE64

This tells SQL Server to format the binary data using the BASE64 format. The good news is that your SQL statement will no longer blow up. The bad news is that reading the results of a query that uses BINARY BASE64 is not pretty. The following XML is an example of what is returned when using BINARY BASE64. I cropped the contents of the Photo attribute because it was over 14,000 characters long!

<row EmployeeID="1" FirstName="Nancy"
    LastName="Davolio" Photo="FRw...f4="/>

Another restriction on the FOR XML AUTO mode is that it does not support the GROUP BY clause or aggregate functions. However, you can get around this by selecting the rows to form a TABLE variable and then using the FOR XML AUTO clause to retrieve the rows from that intermediary table. The following SQL code accomplishes this:

DECLARE @TempTable TABLE (OrderID INT, Total MONEY)

INSERT INTO @TempTable
SELECT OrderID, SUM(UnitPrice * Quantity) AS Total
FROM [Order Details]
GROUP BY OrderID

SELECT OrderID, Total FROM @TempTable AS OrderDetails FOR XML AUTO

Computed columns also must be accounted for when using FOR XML. While computed columns are supported by FOR XML, you need to always make sure you name the columns. The problem arises when FOR XML tries to create an attribute for each column. The name of the attribute is created by using the name of the column that it is associated with. So if the column is a computed column and has not been aliased, then SQL Server will balk at you. The easy solution is to make sure you alias your computed column, like in the following SQL statement and its resulting XML:

SELECT TOP 1 LastName + ', ' + FirstName AS FullName
FROM Employees
FOR XML AUTO

<Employees FullName="Davolio, Nancy"/>

One last item worth noting when using FOR XML is that any of the special XML characters will be converted using XML encoding. Just like HTML converts special characters in a URL, properly formed XML encodes special characters. For example, if the < character is contained within the data, then it will be converted to "&lt;".

OPENXML

So far I have reviewed how to get XML from SQL Server. The next logical step is to take XML and use it to modify data in the database. Enter the OPENXML function. The T-SQL OPENXML function can build a relational rowset from an XML stream. This rowset can act like a table so it can be used in other SQL statements perhaps joined with other tables or even used to insert or update data. This can be very handy because an application can be passed an XML stream containing new or updated records that need to be modified in the database. The OPENXML function uses a form of XPath so that you can tell it where to look for the data that you want to pull out.

To begin, I'll take an XML document containing a customer's data and insert it into the Customers table from the Northwind database. This XML document could use elements or attributes to represent the customer data, like so:

<root>
  <customer>
    <custid>77777</custid>
    <custname>fake customer</custname>
    <city>Somewhere</city>
    <country>USA</country>
  </customer>
</root>
The OPENXML statement allows you to drill into the XML document using XPath expressions, which means the format of the XML can be quite flexible.

I created a stored procedure (shown in Figure 6) that accepts this XML document, prepares it for a relational rowset, reads the customer data, inserts it into the Customers table, and then removes the XML document from memory. Now, let's see how this all works. First, you can pass in the XML as a variable of any string type (such as VARCHAR(8000) or even a large object type like TEXT or NTEXT). I used NTEXT so the XML did not have the 8,000 character limitation of VARCHAR. NTEXT has a maximum length of 230 - 1 (1,073,741,823) characters. (Of course, in SQL Server 2005 you could use the XML datatype instead.)

Figure 6 Inserting a Customer via OPENXML (Close Figure 6)
CREATE PROCEDURE prInsertCustomerFromXML (
    @sXML NTEXT
)
AS
    DECLARE @iDoc INT
    EXEC sp_xml_preparedocument @iDoc OUTPUT, @sXML

    INSERT INTO Customers (CustomerID, CompanyName, City, Country)
    SELECT CustomerID, CompanyName, City, Country
    FROM OPENXML(@iDoc,'/root/customer',2)
    WITH (CustomerID NCHAR(5) 'custid',
          CompanyName NVARCHAR(40) 'custname',
          City NVARCHAR(15) 'city',
          Country NVARCHAR(15) 'country')

    EXEC sp_xml_removedocument @iDoc

RETURN

The first thing I do to the XML is to pass it to the sp_xml_preparedocument system stored procedure. This procedure takes the XML and translates it into an internal DOM so that OPENXML can process it as a rowset. It also creates a reference to the DOM in memory (the @iDoc variable in Figure 6). The XML can be accessed as a rowset within a SELECT statement's FROM clause by passing the @iDoc variable to the OPENXML function.

Once I am finished using the XML for this procedure, I should remove it from memory using the sp_xml_removedocument system stored procedure.

At the core of the code in Figure 6 is the OPENXML function, which operates on the in-memory representation of the XML document. The OPENXML method accepts the reference to the XML document as its first argument. Its second argument is used to tell OPENXML which nodes in the XML DOM you want to map to rows. In this example, I want to identify the Customers nodes in order to grab the customer's data values, so I specify "/root/customer" as the second argument. The third argument to the OPENXML function is used to indicate what type of mapping to use. A value of 1 tells OPENXML to map to attributes and a value of 2 tells it to map to elements.

The WITH clause can be used to specify the fields to grab out of the XML document and the datatypes to convert them to. It can also be used to map an attribute or element in the XML using an XPath expression or even alias the XML field to be used in a query. The OPENXML code in Figure 6 basically takes the four customer fields from the XML data and turns them into a rowset. That rowset is then selected from and inserted into the Customers table.

Inserts via OPENXML

Now I will take the following XML sample and insert an order and two order detail rows from it:

<Customer CustomerID='ALFKI'>
    <Order OrderDate='1/1/1972' Freight='3'>
        <Detail ProductID='1' Price='4' Quantity='10' Discount='0'/>
        <Detail ProductID='2' Price='5' Quantity='2' Discount='0'/>
    </Order>
</Customer>
First, I pass this XML to the stored procedure shown in Figure 7 and prepare it using the sp_xml_preparedocument system stored procedure. Then, since I want to insert an order and its children records, I begin a transaction to wrap the INSERT statements. This allows me to roll back the transaction if part of it fails. Next, I open the XML document using OPENXML starting at the Customer\Order nodes. Using the WITH clause, I grab the CustomerID by traveling back up the XML document to the parent node of the Order element (the Customer element) and getting the value by looking at the CustomerID attribute of the Customer element. This is a great feature of the OPENXML function as it allows you to travel up and down the XML document using limited XPath expressions to grab values of attributes and elements.

Figure 7 Inserting a Parent and Its Children (Close Figure 7)
CREATE PROCEDURE prInsertOrderAndOrderDetailsFromXML @sXML NTEXT
AS
    DECLARE @iDoc int, @OrderID int

    EXEC sp_xml_preparedocument @iDoc output, @sXML
    IF @@Error<>0
    BEGIN
        RETURN
    END

    BEGIN TRANSACTION
    INSERT INTO Orders (CustomerID , OrderDate,  Freight)
    SELECT CustomerID, CAST(OrderDate As datetime), CAST(Freight AS money)
    FROM OpenXML(@iDoc,'/Customer/Order', 1)
    WITH (CustomerID nchar(5) '../@CustomerID' ,
        OrderDate varchar(10) '@OrderDate',
        Freight varchar(12) '@Freight')

    IF @@Error<>0
    BEGIN
        EXEC sp_xml_removedocument @iDoc
        ROLLBACK TRANSACTION
        RETURN
    END

    SET @OrderID=SCOPE_IDENTITY()

    INSERT INTO [Order Details]
        (OrderID, ProductID, UnitPrice, Quantity, Discount)
    SELECT @OrderID, ProductID, ProductPrice, ProductQTY, Discount
    FROM OpenXML(@iDoc,'/Customer/Order/Detail')
    WITH (ProductID int '@ProductID', ProductPrice money '@Price',
        ProductQTY smallint '@Quantity', Discount real '@Discount')

    IF @@Error<>0
    BEGIN
        EXEC sp_xml_removedocument @iDoc
        ROLLBACK TRANSACTION
        RETURN
    END

    EXEC sp_xml_removedocument @iDoc
    COMMIT TRANSACTION
    SELECT @OrderID
GO

Once the Order has been inserted, I grab the OrderID value that was just generated by the built-in SQL Server SCOPE_IDENTITY function. Then I move on to insert the Order Details rows using yet another INSERT statement that uses the OPENXML function to grab the Order Details from the XML data. As long as no errors are encountered, the order and its children order detail rows are inserted into their respective database tables.

The SCOPE_IDENTITY approach works with a single Order and its children. However, when inserting multiple Orders and their children all in one XML batch, it gets more involved. The issue is that with multiple Order records, you still want to be able to map the appropriate order to its children. Since you won't know which rows to associate, you would have to add some code to handle this. One way to tackle this is to use the @mp:id/@mp:parentid meta properties in the WITH clause to provide a way to get the parent's new OrderID and map it to its children's OrderID fields.

Inserts and Updates

XML documents that are prepared by the SQL Server sp_xml_preparedocument system stored procedure can be used in a JOIN just like any other table. They can also be used to INSERT, UPDATE, or DELETE records, just like any other rowset. To demonstrate this, I first selected a series of Order Details rows from OrderID 10285 and stuffed them into an ADO.NET DataSet. Then, I modified some of the existing Order Details rows by changing their quantities and added several Order Details rows to the DataSet, as shown in the following code snippet:

DataSet oDs = new DataSet("NorthwindOrderDetailsData");
oDa.Fill(oDs, "OrderDetails");

//-- Modify the existing 2 line items (for updates)
oDs.Tables["OrderDetails"].Rows[0]["Quantity"] = "2";
oDs.Tables["OrderDetails"].Rows[1]["Quantity"] = "7";

// Add a line item for several other products (inserts)
for (int i = 10; i <= 30; i++) {
    oDs.Tables["OrderDetails"].Rows.Add(
        new object[]{iOrderID, i, 1.50, 3, 0});
}

Once I make these changes, the DataSet gives me several options on how to take the changed data within the DataSet and turn it into XML. I can use the WriteXml method to write out the data as a DiffGram, the data with its schema, or the data without its schema. I can also use the GetXml method to get the data into XML. For this example I will use the DiffGram since it will contain the modified rows along with the before and after state of each row (for UPDATES).

The DiffGram is passed to a stored procedure (shown in Figure 8) which picks out the new and updated rows from the XML document and inserts them into a TABLE variable (@tblTemp). Notice that the OPENXML function in Figure 8 uses an XPath expression to grab the value of the hasChanges attribute. This will be an i for the inserted rows and an m for the modified rows. Because the OPENXML function can filter out rows, only a single XML document must be passed into the stored procedure. Next, the appropriate rows are inserted into the Order Details table and then the appropriate Order Details rows are updated. Both the INSERT and the UPDATE are wrapped inside of a transaction so it can be rolled back if either fails.

Figure 8 Inserting and Updating Using a DiffGram (Close Figure 8)
CREATE PROCEDURE prInsertUpdateOrderDetailsFromXML @sXML NTEXT
AS

    DECLARE @iDoc int, @OrderID int

    EXEC sp_xml_preparedocument @iDoc output, @sXML,
        '<ns xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1"
        xmlns:msdata="urn:schemas-microsoft-com:xml-msdata"/>'

    IF @@Error<>0
    BEGIN
        RETURN
    END

    BEGIN TRANSACTION

    DECLARE @tblTemp TABLE (
        OrderID INT,
        ProductID INT,
        UnitPrice MONEY,
        Quantity SMALLINT,
        Discount REAL,
        ChangeType CHAR(1)
    )

    INSERT INTO @tblTemp
        (OrderID, ProductID, UnitPrice, Quantity, Discount, ChangeType)
    SELECT OrderID, ProductID, ProductPrice,
        ProductQTY, Discount, ChangeType
    FROM OpenXML(@iDoc,
        '/diffgr:diffgram/NorthwindOrderDetailsData/OrderDetails', 2)
    WITH (OrderID INT 'OrderID', ProductID INT 'ProductID',
        ProductPrice MONEY 'UnitPrice',
        ProductQTY SMALLINT 'Quantity', Discount REAL 'Discount',
        ChangeType CHAR(1) '@diffgr:hasChanges')

    IF @@Error<>0
    BEGIN
        EXEC sp_xml_removedocument @iDoc
        ROLLBACK TRANSACTION
        RETURN
    END

    EXEC sp_xml_removedocument @iDoc

    INSERT INTO [Order Details]
        (OrderID, ProductID, UnitPrice, Quantity, Discount)
    SELECT OrderID, ProductID, UnitPrice, Quantity, Discount
    FROM @tblTemp
    WHERE ChangeType = 'i'

    IF @@Error<>0
    BEGIN
        ROLLBACK TRANSACTION
        RETURN
    END

    UPDATE [Order Details]
    SET
        UnitPrice = t.UnitPrice,
        Quantity = t.Quantity,
        Discount = t.Discount
    FROM
        @tblTemp t, [Order Details] od
    WHERE
        t.ChangeType = 'm'
      AND od.OrderID = t.OrderID
      AND od.ProductID = t.ProductID

    IF @@Error<>0
    BEGIN
        ROLLBACK TRANSACTION
        RETURN
    END

    COMMIT TRANSACTION
GO

Bulk Inserts via OPENXML

Inserting 10 rows from a DataSet via a DataAdapter and a stored procedure is easy enough. But the DataAdapter traverses the DataSet's rows one at time looking for the rows whose rowstate is inserted, and it executes the stored procedure associated with the DataAdapter's InsertCommand for each row. This means that 10 inserts would cause 10 calls to the database. This back and forth between the ADO.NET code and the database barely makes a noticeable difference when there are only a few necessary updates. When other factors are introduced, however, such as a large increase in the number of concurrent users or the insertion of hundreds of rows, performance could degrade rapidly. Instead of calling a stored procedure 100 times to insert 100 rows you could pass the 100 rows as XML into a stored procedure all in one shot.

The sample ASP.NET application included with the downloadable code will execute a stored procedure that will insert n number of customers into the Customers table. The ASP.NET code creates and adds 100 customer records in a DataSet. Then it uses the WriteXml method to output the XML data without its schema. This XML is then passed to a stored procedure that inserts the 100 customers all in a single shot. This technique of inserting bulk data is more efficient than inserting each customer one at a time by executing a stored procedure for each insert.

Using the Sample

If you use SQL Query Analyzer to try out any of the FOR XML SQL included in this column you might want to change some of the default options. For example, if you want to query and return XML, you'll want to start by increasing the maximum characters per column in the results pane. Go to Options | Tools | Results and increase the Maximum Characters per Column setting to something like 4096. Since the XML that is returned appears to be a single column in the results pane, this will not cut off the XML at the default of 256 characters. In the end, you will be able to see your XML and copy and paste it into an XML editor if you prefer.

Wrapping Up

The first step towards XML and SQL Server integration was the introduction of the FOR XML and OPENXML features. Thanks to that advance, you can get XML out of a SQL Server database without having to transform it using some intermediary component. You can also pass XML back into a database by combining the use of the OPENXML function and the old standbys, the SQL INSERT, UPDATE, and DELETE statements. These XML features are only the first steps in the gathering support for XML that SQL Server 2005 will offer, but they are valid tools that can be used to begin integrating your XML applications with SQL Server today.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a .NET architect and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. He has authored articles and several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive or blogging at http://codebetter.com/blogs/john.papa.


Service Station: Techniques for Contract-First Development

Service Station
Techniques for Contract-First Development
Aaron Skonnard

Contents
Contract-First in .NET Today
Author XSD
Author WSDL
Generate ASMX Code
Implement Service Operations
Iterate
Hybrid Approach
Where Are We?
Additional Resources

In my May 2005 column, I discussed contract-first development and appropriate times to use it (see Service Station: Contract-First Service Development). In this second part, I'm going to focus on some techniques for contract-first development within the ASMX framework. I'll look at various tools that come with the Microsoft® .NET Framework and Visual Studio® .NET, as well as some useful third-party tools. I'll also discuss an interesting hybrid technique that offers a nice compromise between interoperability and productivity given the current tool situation.

Contract-First in .NET Today

If you use a contract-first development model with Visual Studio .NET and the ASMX framework, you'll probably need third-party tools for ease-of-use and overall productivity. With contract-first, you typically go through the development cycle that is shown in Figure 1 and follow these steps:

  1. Define XML messages using XSD
  2. Define operations using WSDL
  3. Generate ASMX code from WSDL
  4. Implement operations
  5. Iterate

Figure 1 Contract-First Development
Figure 1 Contract-First Development

I'll walk you through each of these steps and explain the different tool options available for assisting you. Then I'll give you a few ideas on simplifying the process.

Author XSD

When using contract-first, you begin by identifying the data exchanged by the service and then model it using XML Schema Definition (XSD). During this step, focus on naming and structuring the data in the messages that interact with the service. When using the document/literal style, the resulting XSD contains all the information about what goes in the SOAP envelopes on the wire. This is often referred to as the service's data contract.

There are a variety of ways to author XSD, starting with opening Notepad and typing. Most developers, however, prefer to use a more sophisticated XSD editor. Visual Studio .NET makes authoring XSD easy through its built-in schema designer. The designer makes it possible to drag and drop schema constructs from the toolbox onto the design surface, shielding developers from the angle brackets and other complexities of the language. You can always switch between the design view and the XML view and edit within either.

Let's look at an example. You want to design a purchase order (PO) service that will have two operations: Submit and GetPending. Using contract-first development, you create a new XSD file in Visual Studio .NET and then begin authoring the XML messages that will be needed by these operations (see Figure 2). In this example, Submit will be a one-way operation so it needs only a single XML element. GetPending needs to be a request/response operation so it will need two elements.

Figure 2 Visual Studio .NET XSD Designer (Close Figure 2)

The XSD shown in Figure 3 illustrates what a sample schema might contain for the messages needed by the PO service. Notice that there are three elements defined: SubmitRequest, GetPendingRequest, and GetPendingResponse. The associated complex type definitions define the structure of these elements.

Figure 3 PO.xsd (Close Figure 3)
<xs:schema xmlns="http://msdn.microsoft.com/purchasing"
   targetNamespace="http://msdn.microsoft.com/purchasing"
   xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
    <xs:element name="SubmitRequest" type="SubmitRequest" />
    <xs:element name="GetPendingRequest" type="GetPendingRequest" />
    <xs:element name="GetPendingResponse" type="GetPendingResponse" />

    <xs:complexType name="SubmitRequest">
        <xs:sequence>
            <xs:element name="PO" type="PO" />
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name="GetPendingRequest" />
    <xs:complexType name="GetPendingResponse">
        <xs:sequence>
            <xs:element name="PendingPOs" type="ListOfPO" />
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name="PO">
        <xs:sequence>
          <xs:element name="CustomerID" type="xs:string" />
          <xs:element name="CompanyName" type="xs:string" />
          <xs:element name="CustomerName" type="xs:string" />
          <xs:element name="Phone" type="xs:string" />
          <xs:element name="EMail" type="xs:string" />
          <xs:element name="LineItem" type="LineItem"
                 minOccurs="1" maxOccurs="unbounded" />
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name="LineItem">
        <xs:sequence>
          <xs:element name="ProductID" type="xs:string" />
          <xs:element name="ProductName" type="xs:string" />
          <xs:element name="Qty" type="xs:int" />
          <xs:element name="UnitPrice" type="xs:double" />
          <xs:element name="ExtendedPrice" type="xs:double" />
        </xs:sequence>
    </xs:complexType>
    <xs:complexType name="ListOfPO">
        <xs:sequence>
          <xs:element name="PO" type="PO"
                 maxOccurs="unbounded" />
        </xs:sequence>
    </xs:complexType>
  </xs:schema>

Visual Studio 2005 comes with a new XML code snippet feature, which will make it possible to package common XML constructs and tie them to short aliases for automatic expansion (similar to the code snippet feature already available for C#), though as always with beta code, things are subject to change before the final release. Microsoft is planning to ship numerous XSD code snippets and expansions to make hand-authoring XSD more efficient for those who prefer to work directly with angle brackets. This feature alone could greatly increase the number of developers who feel comfortable authoring raw XSD.

In addition to Visual Studio .NET, there are also a variety of third-party tools that can be used to author XSD. Altova's XML Spy is one of the industry favorites that also integrates directly with Visual Studio .NET. Several others can be used standalone. The important thing is to find one that your developers will use.

Assuming a document/literal service, the resulting schema defines the valid XML elements that are used when interacting with the service. What is still not known is how the different elements map to logical operations or how they're correlated with each other (for example, GetPendingResponse follows a GetPendingRequest, and so on). This is where Web Services Description Language (WSDL) comes in.

Starting with XSD helps avoid most interoperability problems. It forces you to define your messages in the language that everyone understands, and that means you avoid using types specific to your environment that don't have well-known schema equivalents (like DataSet or Hashtable). Thinking more about the XML will improve interoperability. Because Visual Studio .NET has a built-in designer for XSD, most developers today have at least tinkered with it. Unfortunately, the next step is a bit harder.

Author WSDL

Once you have your schemas in place, you then define your service's operations and transport bindings using WSDL. A WSDL operation simply defines an exchange of XSD messages. The operations are then grouped into a WSDL interface (or portType). WSDL also defines transport binding details that are needed to transmit messages to the service. WSDL is a fairly thin layer over XSD, but it's a necessary one. It contains all the information required by code generators to produce implementation code on either side of the wire.

Authoring WSDL is the most confounding part of the contract-first model. Developers get frustrated by this step because the language is complex and confusing and there isn't much in the way of decent tool support. Writing WSDL in Notepad is a long shot—it takes a strong understanding of XML namespaces, XSD, SOAP, and of course, the WSDL language itself. Even most Web services experts have a hard time hand-authoring a bug-free WSDL definition on the fly.

Visual Studio .NET does not provide the XSD designer equivalent for WSDL. So unless you use a third-party tool, you're basically stuck doing it by hand. However, as with XSD, Visual Studio 2005 comes with several canned WSDL code snippets and expansions that will provide a valuable crutch for developers hand-authoring WSDL.

Another way that you can get around hand-authoring WSDL is to prototype the contract with ASMX and then use the generated WSDL as a starting point. You can inspect the generated WSDL, make necessary changes, and iterate until you've produced the contract you desire. This approach requires you to know enough about WSDL to make good decisions and guide the outcome during the process. Many developers take this approach, but it can still cause some shops to fumble.

There are a handful of third-party editors that can be used to author WSDL definitions—examples include Altova's XML Spy and Cape Clear's SOA Editor. Most of these tools make a noble attempt at an intuitive WSDL designer, but they all make the mistake of attempting full coverage of the language. As a result, most tools in this camp end up being hard to use, not because of flaws in the product necessarily, but because WSDL itself is too hard. Because the language is so flexible and complex, there are many decisions to be made while authoring a WSDL definition. What most developers need is a simple wizard model that helps them produce a WS-I compliant definition, avoiding the parts of the language they don't need to know about.

There are a few third-party tools that make this possible. My personal favorite is Thinktecture's free WsContractFirst tool. WsContractFirst integrates with Visual Studio .NET and provides a step-by-step wizard for generating WS-I-compliant WSDL definitions around existing XSD files, letting you avoid editing the WSDL altogether. It's easy to use and can help you produce WSDL without having to get knee-deep in the complexities of the language.

Figure 4 WsContractFirst Step 1
Figure 4 WsContractFirst Step 1

Once you've installed WsContractFirst, you can right-click on any .xsd file in your project and select "Create WSDL Interface Description." Doing so will launch a wizard that allows you to fill in the WSDL details. On the first step, specify the name and namespace of your service (see Figure 4). Then you specify the names of the operations your service will contain along with the message exchange pattern (see Figure 5). And finally you specify which XSD elements will be used by the operations (see Figure 6). Then WsContractFirst generates a WSDL definition and adds it to your project (see Figure 7).

Figure 7 POService.wsdl (Close Figure 7)
<definitions xmlns:xsd="http://www.w3.org/2001/XMLSchema"
   xmlns:tns="http://msdn.microsoft.com/purchasing"
   xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
   targetNamespace="http://msdn.microsoft.com.com/purchasing"
   name="POService" xmlns="http://schemas.xmlsoap.org/wsdl/"
>
  <types>
    <xsd:schema>
      <xsd:import schemaLocation="PO.xsd"
        namespace="http://msdn.microsoft.com/purchasing" />
    </xsd:schema>
  </types>
  <message name="submitIn">
    <part name="messagePart" element="tns:SubmitRequest" />
  </message>
  <message name="getPendingIn">
    <part name="messagePart" element="tns:GetPendingRequest" />
  </message>
  <message name="getPendingOut">
    <part name="messagePart" element="tns:GetPendingResponse" />
  </message>
  <portType name="POServiceInterface">
    <operation name="Submit">
      <input message="tns:submitIn" />
    </operation>
    <operation name="GetPending">
      <input message="tns:getPendingIn" />
      <output message="tns:getPendingOut" />
    </operation>
  </portType>
  <binding name="POService" type="tns:POServiceInterface">
    <soap:binding transport="http://schemas.xmlsoap.org/soap/http"
       style="document" />
    <operation name="Submit">
      <soap:operation
         soapAction="http://msdn.microsoft.com/purchasing:Submit"
         style="document" />
      <input>
        <soap:body use="literal" />
      </input>
    </operation>
    <operation name="GetPending">
      <soap:operation
         soapAction="http://msdn.microsoft.com/purchasing:GetPending"
         style="document" />
      <input>
        <soap:body use="literal" />
      </input>
      <output>
        <soap:body use="literal" />
      </output>
    </operation>
  </binding>
</definitions>
Figure 5 WsContractFirst Step 2
Figure 5 WsContractFirst Step 2

WsContractFirst is a great example of how simple contract-first development can be with the proper tool support. It's unfortunate that WSDL support isn't deeply integrated in Visual Studio .NET. That turns many developers away from the model despite the obvious benefits. However, thanks to emerging tools like WsContractFirst, it's definitely becoming more feasible. Shortly I'll show you a hybrid mechanism that works around this problem if you would still prefer to avoid WSDL altogether.

Figure 6 WsContractFirst Step 3
Figure 6 WsContractFirst Step 3

Once you've produced the WSDL contract for your service, you're ready to generate code and implement the operations.

Generate ASMX Code

With contract-first, you do the heavy lifting in the first two steps. This third step, code generation, is the easiest part of the whole process because you get to use a code generator to produce code that conforms to the contract, for either side of the wire.

If you're building code to consume the service, simply run it through wsdl.exe and it will generate a proxy class for you. If you need to implement the service, run "wsdl.exe /server" and it will generate an abstract class for you as a starting point—all you have to do is finish implementing it.

Figure 8 illustrates how to run wsdl.exe from the command line. It first shows how to generate a client-side proxy class. Depending on how you run wsdl.exe, you may have to specify the location of the XSD file as well as the WSDL file, as in this example. After generating this class, you can simply add it to your project and begin using it. The generation of client proxy code is also fully integrated with Visual Studio .NET via the "Add Web Reference" feature. After you point it to the WSDL contract, it generates the code, and automatically adds the files to your project. Since most developers are used to the client side, I'm going to focus attention on implementing a service.

Figure 8 Running wsdl.exe
Figure 8 Running wsdl.exe

Figure 8 also shows how to use wsdl.exe to generate a service implementation from a WSDL file. This time specify "/server" on the command line.

Figure 9 shows part of the generated file. Notice that it creates a class that can be used with the ASMX infrastructure—the class in this case is called POService. POService is automatically decorated with all the necessary Web service attributes to ensure that the service implementation exposes the original WSDL contract (POService.wsdl). Wsdl.exe also generates all the data classes that model the original XSD types used by the WSDL definition (PO.xsd). These classes are also decorated with the appropriate XML serialization attributes to conform with the original schema.

Figure 9 Code Generated by wsdl.exe /server (Close Figure 9)
//----------------------------------------------------------------
// <autogenerated>
//     This code was generated by a tool.
//     Runtime Version: 1.1.4322.573
// </autogenerated>
//----------------------------------------------------------------

using System.Diagnostics;
using System.Xml.Serialization;
using System;
using System.Web.Services.Protocols;
using System.ComponentModel;
using System.Web.Services;

/// <remarks/>
[WebServiceBinding(Name="POService",
  Namespace="http://msdn.microsoft.com/purchasing")]
public abstract class POService : WebService
{

    /// <remarks/>
    [WebMethod()]
    [SoapDocumentMethod("http://msdn.microsoft.com/purchasing:Submit",
      OneWay=true, Use= SoapBindingUse.Literal, ParameterStyle=
      SoapParameterStyle.Bare)]
    public abstract void Submit
    (
      [XmlElement(Namespace="http://msdn.microsoft.com/purchasing")]
      SubmitRequest SubmitRequest);

    /// <remarks/>
    [WebMethod()]
    [SoapDocumentMethod(
      "http://msdn.microsoft.com/purchasing:GetPending",
      Use=SoapBindingUse.Literal, ParameterStyle=
      SoapParameterStyle.Bare)]
    [return: XmlElement("GetPendingResponse",
      Namespace="http://msdn.microsoft.com/purchasing")]
    public abstract GetPendingResponse GetPending([XmlElement(
      Namespace="http://msdn.microsoft.com/purchasing")]
      GetPendingRequest GetPendingRequest);
}

/// <remarks/>
[XmlType(Namespace="http://msdn.microsoft.com/purchasing")]
public class SubmitRequest
{

    /// <remarks/>
    [XmlElement(Form=XmlSchemaForm.Unqualified)]
    public PO PO;
}
... // remaining classes omitted for brevity

To implement the service, add the generated files to your project and begin writing code (see the next step). Unfortunately, Visual Studio .NET does not provide any built-in support (similar to "Add Web Reference" on the client) for implementing services from WSDL, but Visual Studio 2005 Team System introduces support for this via the new Application Connection Designer (ACD).

If you want Visual Studio .NET integration today, you can use WsContractFirst. WsContractFirst also generates code from the WSDL contract. Assuming you have the tool installed in Visual Studio .NET, you can right-click on any WSDL definition in your project and select "Generate Web Service Code." This brings up the dialog shown in Figure 10. Options there include whether you want to generate a client-side proxy or a server-side interface (as in wsdl.exe /server). Other handy options are generating properties instead of fields, marking the classes serializable, and enabling XSD validation for incoming messages. And when you use this tool to generate a server-side interface, it greatly simplifies what you have to do in the next step, as you'll see shortly.

Figure 10 WsContractFirst Code Generator
Figure 10 WsContractFirst Code Generator

When you generate .NET classes from WSDL, they'll contain all the attributes needed to ensure conformance to the predefined contract, shielding you from having to learn much about the .NET attributes themselves and letting you focus your time and energy on learning more about XSD and WSDL. When the ASMX framework is asked to generate WSDL for this class, it will produce a WSDL definition that is equivalent to the original. It's important for you to note that it doesn't return the original WSDL file—it actually generates a new one from the attributes found on the class. You can, however, tell ASMX to return the original WSDL file by specifying its location through the Location property on the [WebServiceBinding] attribute.

Now that you have the service framework code generated, the next step is to implement the service operations.

Implement Service Operations

In this step your focus is on taking the generated code and implementing your business logic. You don't have to worry about modifying attributes since they've all been generated for you in the previous step. You should only have to add code within the method blocks. Then you simply need to bind the class to an .asmx file.

If you inspect the generated class definition shown in Figure 9 you'll notice it is marked "abstract" and it is missing the [WebService] attribute. It was designed to have you derive a new class from the generated abstract class and implement the operations there. The [WebService] attribute belongs on the implementation class, which is why it's left off the generated abstract class. Given this approach, if you need to modify the WSDL contract down the road and regenerate the service interface, you won't accidentally overwrite your implementation code. The flaw is that the Web service attributes don't actually propagate during derivation. This ultimately means that you have to copy and paste all Web service attributes to the new derived class regardless.

A simpler approach is to simply make the generated class concrete, removing the abstract keyword from the class and each method. You will also need to add a [WebService] attribute to the class (specifying at least the service namespace). Then you can add an implementation block to each method and begin writing your business logic. The only problem here, of course, is that you have to be careful if you need to modify the WSDL contract and regenerate the service interface. Make sure always to use a different file name via the /out switch so you don't accidentally overwrite your implementation code. Then after you generate the new abstract class, you'll have to repeat the process, making the class concrete, adding the [WebService] attribute, and then copying over your existing implementation code.

WsContractFirst shields you from some of these complexities. When you use the WsContractFirst code generator it will produce the typical abstract base class as well as a derived class with all the Web service attributes copied over. It also injects the [WebService] attribute for you on the derived class, so all you have to do is begin writing your code.

Figure 11 shows a sample derived class generated using WsContractFirst. You could still overwrite your implementation using this tool, so be careful when making changes. And if you generate a new service interface, you'll still need to copy over your existing method implementations.

Figure 11 Service-Interface Code (Close Figure 11)
//----------------------------------------------------------------
// <autogenerated code>
//     This code was generated by WsContractFirst.
//     Changes to this file may cause incorrect behavior and will
//     be lost if the code is regenerated.
// </autogenerated code>
//----------------------------------------------------------------

namespace MSDN.Services
{
    using System.Diagnostics;
    using System.Xml.Serialization;
    using System;
    using System.Web.Services.Protocols;
    using System.ComponentModel;
    using System.Web.Services;

    /// <remarks/>
    [WebServiceBinding(Name="POService",
       Namespace="http://msdn.microsoft.com/purchasing")]
    [WebService(Namespace="http://msdn.microsoft.com/purchasing")]
    public class POServiceImpl : POService
    {

        /// <remarks/>
        [WebMethod()]
        [SoapDocumentMethod(
          "http://msdn.microsoft.com/purchasing:Submit", OneWay=true,
          Use=SoapBindingUse.Literal, ParameterStyle=
          SoapParameterStyle.Bare)]
        public override void Submit([XmlElement(
          Namespace="http://msdn.microsoft.com/purchasing")]
          SubmitRequest SubmitRequest)
        {
            ... // put your code here
        }

        /// <remarks/>
        [WebMethod()]
        [SoapDocumentMethod(
           "http://msdn.microsoft.com/purchasing:GetPending",
           Use= SoapBindingUse.Literal, ParameterStyle=
           SoapParameterStyle.Bare)]
        [return: XmlElement("GetPendingResponse",
           Namespace="http://msdn.microsoft.com/purchasing")]
        public override GetPendingResponse GetPending([XmlElement(
           Namespace="http://msdn.microsoft.com/purchasing")]
           GetPendingRequest GetPendingRequest)
        {
            ... // put your code here
        }
    }
}

There is a new feature coming in the .NET Framework 2.0 that will give developers a little more flexibility when making changes to the contract during development. The new /serverInterface switch tells the code generator to produce an interface definition instead of an abstract base class. The ASMX framework in the .NET Framework 2.0 now supports decorating interface definitions with Web service attributes in order to specify WSDL contract details. Then when you derive a class from the interface, all of the attributes propagate down as they should have before. The only attribute that you need to have on the derived class is [WebService].

Using an interface to hold all of the WSDL contract information makes a lot of sense because it cleanly separates the contract from the implementation code. If you need to make changes to the WSDL later in the development cycle, you can rerun wsdl.exe with /serverInterface and let it override the previous interface definition. Then you just need to make sure that your existing implementation still works—and the compiler can help you with that. You don't have to worry about overwriting your implementation code, since all of the WSDL metadata is cleanly factored into a separate artifact. For more details on this new feature, check out my column in the April 2005 issue of MSDN®Magazine at Service Station: Developing .NET Web Services with Beta 2.

Iterate

In the last step, one of the issues that comes up is how to deal with changes to the WSDL contract during the development cycle. For example, when you need to change the WSDL contract after you're already well into development, you also need to regenerate the proxy and service interfaces, rationalize those changes with your existing implementation code, and make sure that everything is still working.

Support for an iterative development process is crucial to long-term success with contract-first development. Obviously it's rare to get the contract definition right the first go-around, so you need the ability to go back to the original XSD and WSDL, make modifications, and keep everything in sync. This is the other place where the current tools let you down.

In the previous step I discussed all of the issues surrounding wsdl.exe and regenerating ASMX code. Today you have to manually copy code (either the attributes or the method implementations) in order to make this work. Even third-party tools like WsContractFirst don't currently address this issue.

With a fully integrated iterative development environment, you shouldn't have to care about where the contract metadata is stored (whether on the class or an interface). It should let you make changes to the contract and keep everything in sync automatically without losing changes to your existing code. This is the direction Microsoft is headed with future versions of Visual Studio.

Visual Studio 2005 Team System introduces an ACD that allows you to author service contracts before implementing them. You can model service contracts and specify which applications implement them and which consume them. Once the application framework has been completely modeled, the designer will generate all the service-related code behind the scenes, adding proxy code to consuming applications and implementation code to services implementing the contract. But the real beauty of the Application Connection Designer is how it lets you make changes to the contract while keeping all the generated code in sync, throughout all applications using the contract, without causing you to worry about losing implementation code.

The main problem with the ACD, from a contract-first perspective, is that it doesn't support modeling XML messages with XSD and service operations using WSDL. It still focuses on methods and parameters as opposed to schema types and elements. The good news is Microsoft plans to provide better support for a full message-oriented contract-first design experience in a future release of Visual Studio. Ill explore developing services using the new ACD in an upcoming column.

The overall lack of tool support for an iterative and integrated contract-first development process is why contract-first requires more discipline today. However, there is a hybrid technique that can provide a simpler process given today's tools.

Hybrid Approach

The two biggest problems with the current tool support are lack of WSDL support and lack of support for an iterative development process. The current support for XSD design, however, is quite reasonable. It turns out that using contract-first for XSD message design addresses most of the interoperability problems developers run into. One way to compromise between your interoperability and productivity needs is to use a hybrid approach, where you use contract-first to model XSD and .NET code to model WSDL.

In other words, you model the XSD as I showed you here (for your request and response messages) and then you run it through xsd.exe with the /classes switch (instead of layering WSDL and using wsdl.exe). This generates the .NET serialization classes that will produce the XML format described by the XSD. Then you can use those classes in your [WebMethod] signatures.

When you use this approach, the [WebMethod] should take a single argument (representing the request message) and specify a return type (representing the response message). Both should be decorated with an [XmlElement] attribute specifying the correct element names for the request and response messages (ASMX uses the argument name by default).

The [WebMethod] should also be marked with [SoapDocumentMethod], specifying a ParameterStyle of SoapParameterStyle.Bare. This prevents ASMX from generating additional wrapper elements for the messages. This general style is referred to as document/literal/bare. Check out Figure 9, Figure 11, and Figure 12, all of which use document/literal/bare signatures. You can also use this technique with the new .NET Framework 2.0 interface model, which probably makes the most sense down the road.

Figure 12 Classes Generated Using wsdl.exe /serverinterface (Close Figure 12)
//----------------------------------------------------------------
// <autogenerated>
//     This code was generated by a tool.
//     Runtime Version:2.0.40903.19
// </autogenerated>
//----------------------------------------------------------------
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Xml.Serialization;

/// <remarks/>
[WebService(Namespace="http://msdn.microsoft.com/purchasing")]
[WebServiceBinding(Name="POService",
   Namespace="http://msdn.microsoft.com/purchasing")]
public interface IPOService {

    /// <remarks/>
    [WebMethod()]
    [SoapDocumentMethod("http://msdn.microsoft.com/purchasing:Submit",
       OneWay=true, Use= SoapBindingUse.Literal, ParameterStyle=
       SoapParameterStyle.Bare)]
    void Submit([XmlElement(
       Namespace="http://msdn.microsoft.com/purchasing")]
       SubmitRequest SubmitRequest);

    /// <remarks/>
    [WebMethod()]
    [SoapDocumentMethod(
      "http://msdn.microsoft.com/purchasing:GetPending",
      Use= SoapBindingUse.Literal, ParameterStyle=
      SoapParameterStyle.Bare)]
    [return: XmlElement("GetPendingResponse",
      Namespace="http://msdn.microsoft.com/purchasing")]
    GetPendingResponse GetPending([XmlElement(
      Namespace="http://msdn.microsoft.com/purchasing")]
      GetPendingRequest GetPendingRequest);
}

/// <remarks/>
[Serializable()]
[XmlType(Namespace="http://msdn.microsoft.com/purchasing")]
public partial class SubmitRequest {

    private PO poField;

    /// <remarks/>
    [XmlElement(Form=XmlSchemaForm.Unqualified)]
    public PO PO {
        get {
            return this.poField;
        }
        set {
            this.poField = value;
        }
    }
}
... // remaining classes omitted for brevity

As far as the iterative experience goes, a free third-party tool from www.sellsbrothers.com/tools called XsdClassesGen integrates with Visual Studio .NET and greatly simplifies this approach. This tool integrates running xsd.exe /classes within Visual Studio .NET. All you have to do is set the Custom Tool property for each .xsd file in your project to SBXsdClassesGenerator and the add-in will automatically generate the classes behind the scenes and keep them attached to your .xsd files. When you modify the XSD contracts, the tool automatically runs and the classes are updated.

Figure 13 Document/literal/bare WebMethod
Figure 13 Document/literal/bare WebMethod

When you develop ASMX this way, as shown in Figure 13, you're using XSD to define the message formats, and you're using the ASMX interface to correlate the XSD messages into operations. The focus on XSD helps you avoid most interoperability concerns, and authoring the WSDL operations via the ASMX class gives you a more pleasant and productive experience.

Where Are We?

There are a variety of ways to employ contract-first design when building Web services with .NET. Whether you bite off authoring the complete WSDL definition by hand or you prefer to stop at XSD using the hybrid technique I described, your systems will benefit from increased interoperability and simplified versioning.

Additional Resources

  • "Contract First Web Services Interoperability between Microsoft .NET and IBM WebSphere"
  • Service Station, "SOA: More Integration, Less Renovation," MSDN Magazine
  • MSDN Web Services Interoperability and Integration
  • Service Station, "Improving Web Services Interoperability," MSDN Magazine

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, a premier Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services, Applied BizTalk Server 2004, and Introducing Indigo courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at http://pluralsight.com/aaron.


Extreme ASP.NET: Tools of the Trade: Application Center Test

Extreme ASP.NET
Tools of the Trade: Application Center Test
Rob Howard

Contents
Application Center Test
Using Application Center Test
Running a Test and Establishing the Baseline
Know What You're Improving
Setting Up a Test Environment
Next Steps and Test Coverage
Conclusion

When you sit down to write an ASP.NET application, how much time do you spend thinking about performance? It's unfortunate, but for most developers performance is an afterthought. Performance planning and design really need to be front and center. Think about your goals and make sure one of those goals is good performance. Then make sure to measure your progress. The more you measure, the more opportunity you have to improve performance.

In this month's column I'm going to walk you through one of the great tools included with Microsoft® Visual Studio® Enterprise Edition: Microsoft Application Center Test. It's one tool that serious Web developers must have in their utility belts.

Application Center Test

Prior to leaving Microsoft, I did a 12-city roadshow for ASP.NET. One of the segments covered performance and introduced many developers to Microsoft Application Center Test. The tool invariably generated a lot of interest and I got lots of questions about it.

You can find the Application Center Test tool as part of Application Center (just hunt through your old stack of MSDN® CDs or DVDs) or as part of your Visual Studio .NET Enterprise Edition installation at Visual Studio .NET 2003\Visual Studio .NET Enterprise Features. When you first open Application Center Test you are presented with a tree view for navigating the available Tests, Results, and Users. First, I want to show you how easy it is to create Tests.

Using Application Center Test

First, create a simple Web application. For example, I'll use the page shown in Figure 1 (note that I've used a little-known trick when writing code inline with an ASP.NET page: you don't have to write the full event declaration for Page_Load).

Figure 1 Simple Web App (Close Figure 1)
<%@ Page Language="C#" %>
<%@ Import Namespace="System.Data " %>
<%@ Import Namespace="System.Data.SqlClient" %>
<%@ Import Namespace=" System.Configuration" %>

<script runat="server">
  public void Page_Load() {
      using(SqlConnection connection =
        new SqlConnection(ConfigurationSettings.AppSettings["Northwind"]))
      {
          SqlCommand command = new SqlCommand(
              "SELECT * FROM Products", connection);
          connection.Open();

          DataGrid1.DataSource = command.ExecuteReader();
          DataGrid1.DataBind();
      }
  }
</script>

<form runat="server">
  <asp:DataGrid id="DataGrid1" runat="server" />
</form>

The code in the figure, though not a recommended way to structure your application, is simple enough to perform some rudimentary tests on. Opening the page in a Web browser should return a populated DataGrid rendered as an HTML table.

Now that you know the page works, copy the URL to your clipboard, as you'll need it in a second. The example URL in my case is http://localhost/blackbelt/outputcache/test.aspx.

Next, navigate to Application Center Test, right-click on Tests and choose New Test. This opens the New Test Wizard Welcome screen. Click Next to choose the source for the new test and choose "Record a new test." Click Next again to select the Test Type, and then click Next again when prompted to select the Script Language (leaving the defaults), bringing you to the screen in Figure 2.

Figure 2 New Test Wizard
Figure 2 New Test Wizard

Recording tests is what makes Application Center Test so easy to use. Click on "Start recording" and a new instance of Internet Explorer will open. Do not type a URL into the Address bar just yet (it should read about:blank). Instead, from within this new Internet Explorer instance select Tools | Internet Options and browse to the Connections tab. Next, click on the LAN Settings button. You should see something similar to the screen shot in Figure 3.

Figure 3 Connection Settings
Figure 3 Connection Settings

You will notice that the Proxy server settings are populated and are different from their normal values. That's because Application Center Test has opened a new instance of Internet Explorer and instructed it to use a private proxy server run by Application Center Test. Any requests made through Internet Explorer can be captured by the Application Center Test proxy, as you will soon see.

To complete the test, close the Internet Explorer dialogs and paste the URL from your test ASP.NET page into the Address bar. Clicking the green Go button in Internet Explorer or simply hitting Enter will render the DataGrid again. Next, close Internet Explorer and you should see something similar to Figure 4.

Figure 4 Captured Requests
Figure 4 Captured Requests

The request details portion of the dialog is now populated with the requests that the Application Center Test proxy has captured. These are the same HTTP requests sent by Internet Explorer. Now click "Stop recording," then Next. You will be prompted to give the test a name (I used "My Test") and you can then finish the wizard.

Congratulations you're now a performance testing jockey—pretty easy huh?

There are many other settings and configuration options for your new test. You can find these by right-clicking on the My Test node in the Tests list and selecting Properties. From there you can simulate multiple browsers, multiple users, parameters that include a warm-up period (where results are not reported), and the duration for the test. (You can explore these settings later and read some of the many other great articles that discuss the philosophy of testing and the best strategies for running your tests.) Instead of dwelling on the minutia, let's jump right into running the test.

Running a Test and Establishing the Baseline

To run the test you just created, simply right-click on your test and select Start Test. Once the test begins running, click on the Show Details button. The details box shows a graph of the running test as well as any errors that may be occurring during the execution of the test (see Figure 5). Running the test the first time establishes the baseline, which is used to measure against the current and future performance. In the figure, the graph shows the baseline at approximately 90 requests per second.

Figure 5 Baseline Graph
Figure 5 Baseline Graph

Now that you have an established baseline you can make some changes to the application to either improve or degrade overall performance in order to gauge the effects of such changes. Granted, the test case I used is exceedingly simple, but I want to show you a few tweaks that you can make to this small piece of code that have a large effect on the performance of the app.

Know What You're Improving

The more you measure the more opportunity you have to improve. In this case I am going to make some small changes to the application and measure after each change. While this is not something you can realistically expect to do with every check-in, you should get in the habit of checking performance regularly. For my company's Community Server product, we have a baseline set of tests to measure the overall app performance. If major changes are made, a developer can investigate any performance improvement or regression using previous test data.

This first change I'm going to make to the sample application is to limit the amount of data returned. I'll do this by changing the SQL query from SELECT * FROM Products to SELECT TOP 25 * FROM Products. This may seem like a minor change to the code. After all, all I have done is limit the amount of data output to the screen, but the results are surprising. The performance jumped from ~90 RPS (requests per second) to more than 200—that's greater than a 100 percent gain! Because you had a baseline, you know that limiting the amount of data bound in the DataGrid definitely affects performance. I've still got some tweaks to make.

Next, modify the <asp:DataGrid/> server control by adding EnableViewState="false":

<asp:DataGrid EnableViewState="false" id="DataGrid1" runat="server" />

ViewState is a great feature of ASP.NET, but it is not always needed. In fact, in most cases where the DataGrid is used ViewState is not needed. By disabling ViewState, I was able to increase performance 166 percent above the baseline in my test case. Hold on, I'm not finished yet!

Next, add the following line to enable OutputCaching:

<%@ OutputCache Duration="60" VaryByParam="none" %>

Figure 6 Output Caching
Figure 6 Output Caching

Now rerun the test. Surprised! When I did this, I experienced a performance gain of more than 600 percent over the baseline, as you can see in Figure 6. Now if you really want to have some fun, play with the OutputCache duration numbers. For example, try an OutputCache duration value of 1 second—you'll be surprised again by the results.

Setting Up a Test Environment

Tests are CPU-intensive, so when running a test you should expect to see your CPU running at nearly 100 percent. This takes away resources from the application you are testing as it is sharing CPU time with the test harness. All kinds of configurations will affect your test results, and some will simulate the real world better than others. For example, when running a test with both SQL Server™ and the ASP.NET application on the same machine, you are not simulating the cost of network I/O. Most applications use a database on a separate server from their Web server.

To set up a real test environment, dig up a couple of old developer machines to use as your clients. Do not use virtual machines. Run Application Center Test on these machines. Next try to simulate your real environment as much as possible. Run the Web application on a server operating system with no other applications running and connect to a database on a separate server.

Those caveats aside, there is nothing wrong with running smoke tests on your developer box. Smoke tests that don't simulate a real environment, such as the examples shown earlier, still provide lots of valuable data and often can be extrapolated to forecast what those same tests would look like in a real-world scenario.

Next Steps and Test Coverage

Now that you have an idea of how to test and measure I would recommend trying to use Application Center Test on your own applications. It's a good idea to find out how your users typically use your application. Which pages have to perform really well, and which pages really don't?

For example, in Community Server we have several different types of performance tests we run. Mainline tests include anonymous scenarios (when no user is logged in) as well as authenticated scenarios. In applications that do heavy personalization this can significantly change the performance characteristics.

Mainline tests also include common paths through the system, such as the main view for blogs, galleries, or forums as well as the various views for each screen. It's obviously important that these mainline scenarios perform very well and it's best to spend a lot of time here to ensure you get them right.

If you manage or run any project that must support two or more simultaneous users and you don't know how many requests per second your main pages can handle, you've got a problem.

Once you have your performance tests in place, measure after any significant milestone. If you've structured your code effectively you can always go into the source and comment out sections and recompile. This can help you determine if the problem is a poorly written routine or something else. The something else, nine times out of ten, is going to be your data access code. In a later column I'll detail two tools in SQL Server, Profiler and Query Analyzer, that can be used to isolate performance problems in the database—and you don't need to be a DBA to understand them.

You can also test common paths through the application. When recording your test, just follow some of the common navigation paths you expect a user to follow. Application Center Test will record those and you can replay that exact script. If you want to really get fancy you can edit the generated VBScript files to introduce delays or other meaningful inputs into your test script—that's a separate article though.

The last test I would recommend requires a bit more work. For example, in Community Server our developers wanted to test how many posts per minute the application can support. To test this, rather than attempting to post content to a form, we created a new ASP.NET page that used the APIs to post the content. This page was then run through Application Center Test and a benchmark was established for how many posts per second the application can support. In other words, sometimes you need to do a little bit of work to test all your scenarios.

Conclusion

I didn't cover everything that Application Center Test provides, but I hopefully did give you enough working knowledge of Application Center Test so that you can begin using it to measure your applications and improve them. Just remember, establish a baseline, measure often (at least after every major application change), and identify your key scenarios. By following these simple rules you'll develop a better understanding of your applications and hopefully identify opportunities to improve their performance. In my next column I'll introduce some debugging tips and tricks that I use when writing Web applications.

Send your questions and comments for Rob to xtrmasp@microsoft.com.

Rob Howard is a founder of Telligent Systems, specializing in high-performance Web apps and knowledge management and collaboration systems. Previously, Rob was employed by Microsoft where he helped design the infrastructure features of ASP.NET 1.0, 1.1, and 2.0. You can contact Rob at rhoward@telligentsystems.com.


Test Run: The Analytic Hierarchy Process

Test Run
The Analytic Hierarchy Process
James McCaffrey

Contents
The System Under Analysis
Setting Up the Problem
Determining the Relative Weights of the Comparison Attributes
Compare the Builds on Each Attribute
Aggregate Weights to Produce Final Evaluation
Discussion
Conclusion

Most software testing takes place at a relatively low level. Testing an application's individual methods for functional correctness is one example. However, some important testing must take place at a very high level—for example, determining if a current build is significantly better overall than a previous build. In this column I'll show you a powerful technique called the analytic hierarchy process that will enable you to perform high-level quality analysis of your software systems.

The best way to illustrate where I'm headed is with a concrete example. Suppose you are developing a product that searches for files across a local network. You want to know if the current build is better overall than the previous builds, and if so, how much better? Comparing the overall quality of the builds with each other is not so easy because you have to consider many attributes such as performance, functionality, and so forth. Using the techniques I'll explain in this month's column will enable you to compute metrics like those shown in Figure 1.

Figure 1 Multi-Criteria Quality Analysis (Close Figure 1)

Performance |  |  | Functionality |  |  | Overall
 | Startup | File Save | User Interface | Database | Network |
Build A | 0.648 | 0.571 | 0.443 | 0.639 | 0.638 | 0.531
Build B | 0.230 | 0.286 | 0.387 | 0.274 | 0.273 | 0.329
Build C | 0.122 | 0.143 | 0.170 | 0.087 | 0.089 | 0.140

If you examine the table in Figure 1 you'll see that the overall quality of each build consists of a performance attribute and a functionality attribute. Each of these attributes is composed of subattributes: Startup and File Save in the case of performance, and User Interface, Database, and Network in the case of functionality.

The technique I'll demonstrate is very flexible and can be adapted to any number of attributes with any number of subattributes. Based on build comparison data that I'll show you in the following sections, the overall quality metric for the current build, Build A, is 0.531 and the overall quality for the two previous builds is 0.329 and 0.140, respectively. I'll explain how to interpret these quality metrics later, but for now it's enough to say larger numbers are better. So, you have a fairly strong indication that the current build is significantly better than the previous two builds. The overall quality metrics for each build depend on the quality metrics for each attribute. In this example, for startup performance, Build A is best at 0.648, Build B is next at 0.230, and Build C is worst at 0.122.

In the sections that follow I will briefly describe the hypothetical Windows®-based application I am analyzing for quality, walk you through the analysis of the app using the analytic hierarchy process, explain how to interpret the resulting quality metrics, and conclude with a discussion of how you can adapt this technique to meet your own needs. I think you'll find that the ability to quickly evaluate the overall quality of a software system composed of many attributes is a valuable addition to your software testing, development, and management skill sets.

The System Under Analysis

Let's briefly look at the application under test so you'll understand the goal of quality analysis. The dummy search application is a simple form-based app. If I want to compare the overall quality of a particular build with previous builds, I have to take into account many system attributes. For the purposes of this column I'm using only performance and functionality, but the technique I'll show you can deal with any number of attributes. Although each software system is different, the most common attributes to consider when evaluating overall quality include accessibility, documentation, functionality, internationalization, module/API, performance, security, setup, scalability, stability, usability, and UI.

Of course there are many attributes, and each will have many subcomponents. In addition, many of these attributes overlap each other. But in practice, this list is a good place to start. This technique can deal with virtually any kind of software system—Windows-based applications, ASP.NET Web applications, class libraries, and so forth.

My goal is to produce a single value that represents the overall quality of my software system as a whole. There are several multi-attribute analysis techniques available, but the one I prefer is called the analytic hierarchy process. The analytic hierarchy process has been around since the early 1980s and has been used in a wide variety of fields. The seminal reference on the topic is the book The Analytic Hierarchy Process for Decisions in a Complex World by Thomas Saaty (McGraw-Hill, 1980).

Without a structured technique, trying to evaluate the overall quality of different builds of even this small dummy application would be very difficult. After choosing your attributes, you'd have to determine how to compare each build on each attribute, how to quantify that information, and how to aggregate all that data into a meaningful metric. Then you'd have to decide how to interpret your results. A much better approach is to use the elegant analytic hierarchy process to quickly evaluate your builds.

Setting Up the Problem

The first step in the analytic hierarchy process is to set up the problem. In the case of multi-attribute software quality analysis this means deciding which builds you want to evaluate, and which attributes you will use as a basis for the evaluation. I have chosen to compare the three most recent builds of my hypothetical system—Build A, Build B, and Build C. The technique can accommodate any number of builds, but based on my experience three or four is a good number for evaluation purposes.

Next you must decide which attributes you will use to evaluate the overall quality of each build. Then, you'll decompose each attribute into subattributes. I decided to break down performance into startup performance and file save performance, and to break down functionality into user interface, database, and network functionality. The technique here allows you to decompose subattributes into even smaller sub-subattributes, and so forth, to any depth. As a general rule, you'll most often have top-level attributes and between two and five subattributes.

Figure 2 Traditional Representation of the Problem
Figure 2 Traditional Representation of the Problem

Figure 2 shows a diagram traditionally used to represent the problem. I don't like it because it combines the goal with the build alternatives and the quality attributes in a way that is difficult for software engineers to interpret. I prefer to represent the problem in tabular form, as shown in Figure 3. Whichever representation format you decide to use, it must clearly summarize the problem in order to convey it to others.

Figure 3 Setup (Close Figure 3)

Problem
Evaluate build quality
Alternatives
Build A, Build B, Build C
Attributes
Performance
Startup
File Save
Functionality
User Interface
Database
Network

Determining the Relative Weights of the Comparison Attributes

After I've set up my problem, the next step is to determine the relative weights of each of the comparison attributes. In my example so far, I have not determined exactly how important my chosen attributes are relative to each other. For instance, startup performance may be much more important to me than file save performance, or vice versa, or they may be equally important.

The analytic hierarchy process uses a pair-wise comparison technique. It works like this: I start with my highest-level attribute classifications, which in this example are performance and functionality. I create a table structure like so:

 | Performance | Functionality
Performance |  |
Functionality |  |
Because I have two highest-level attributes, my table is size 2 × 2. If I had three highest level attributes (for example, an additional security attribute), then my table would have been size 3 × 3, and so forth. Next I will compare the relative importance of product functionality versus product performance using the table that is shown in Figure 4.

Figure 4 Comparison Values (Close Figure 4)

Relative Importance | Value
Equal importance/quality | 1
Somewhat more important/better | 3
Definitely more important/better | 5
Much more important/better | 7
Extremely more important/better | 9

For my example suppose I determine that functionality is between "somewhat more important" (value = 3) and "definitely more important" (value = 5) than performance, and so I get a value of 4 from the table. It is most common to use the values 1, 3, 5, 7, and 9, but you can use an intermediate value as I've done in my example. The loose interpretation of this value is that functionality is four times as important to me as performance in this software system.

Now the upper-left to lower-right diagonal elements of the table are comparing the relative importance of performance against itself, and the relative importance of functionality against itself. Using the values from the table in Figure 4, you can see I get values of 1 because each attribute has equal importance compared to itself. By definition, all the upper-left to lower-right diagonal elements of an analytic hierarchy process comparison table will be 1's. This just leaves the upper-right cell of my comparison table which is the relative importance of performance versus functionality. Because I previously specified that functionality versus performance is a 4 (that is, that functionality is about 4 times as important to me as performance), the relative importance of performance versus functionality must be ¼ or 0.250. In other words, performance is ¼ as important as functionality to me. Placing these values in the table gives me the following:

 | Performance | Functionality
Performance | 1 | 0.250
Functionality | 4 | 1
By definition, the values of cells in a comparison table that are diagonal (45 degrees) from each other will be mathematical inverses of each other. After completing the relative importance table, I've got to compute the "priority vector" from the table. Again, the process is best explained by example. I first sum each column, getting 5 and 1.250, respectively. Next I divide each entry in the relative importance table by its column sum. In this case I have 1/5 and 4/5 in the first column, and 0.250/1.250 and 1/1.250 in the second column. This gives me the matrix:
 | Performance | Functionality
Performance | 0.200 | 0.200
Functionality | 0.800 | 0.800
The last step to determine the priority vector is to take the average of each row. This is (0.200 + 0.200) / 2 = 0.200 and (0.800 + 0.800) / 2 = 0.800, or expressed as a column vector:
Performance | 0.200
Functionality | 0.800
The calculations are very easy to produce using Microsoft Excel, or if you are more ambitious you can write a C# or Visual Basic .NET program to do it.

After producing the priority vector for the highest level of comparison attributes (performance and functionality), I need to compute priority vectors for each sublevel. The process is just like that for the higher-level attribute categories. I'll illustrate by showing how I calculate the priority vector for the functionality attributes (user interface, database, and network).

I start by making an empty 3×3 table. Next I compare the relative importance of each attribute in a pair-wise fashion. In this example I decided that user interface functionality is "somewhat more important" than database functionality (value = 3), user interface functionality is "much more important" than network functionality (value = 7), and database functionality is "slightly more important" (value = 2) than network functionality. I enter these three values into the comparison table, place 1s on the main diagonal, and enter inverses for corresponding cells across the main diagonal. Then I compute each column sum. At this point I have the table shown in the top of Figure 5. Then I divide each table cell by its corresponding column total to get the table at the bottom of Figure 5. And then I compute each row average to obtain the final priority vector, as shown here:

User Interface | 0.681
Database | 0.216
Network | 0.103
Notice that each individual value in any priority vector will be between 0.0 and 1.0, and that the values in any priority vector will sum to 1 (subject to rounding error). What you end up with are so-called eigenvalues—normalized priority weights of each attribute. You can think of the priority weights as the values that are the most consistent with the pair-wise comparison values you enter. You could generate these weights directly, but one of the nice features of the analytic hierarchy process is the pair-wise comparison process that allows you to generate the weights in a consistent manner.

Figure 5 Two Tables (Close Figure 5)

 | User Interface | Database | Network
User Interface | 1 | 3 | 7
Database | 0.333 | 1 | 2
Network | 0.143 | 0.500 | 1
Sum | 1.476 | 4.500 | 10.000
 | User Interface | Database | Network
User Interface | 0.678 | 0.667 | 0.700
Database | 0.226 | 0.222 | 0.200
Network | 0.097 | 0.111 | 0.100

After I compute the priority vector for the functionality attributes, I will then compute the priority vector for the two performance attributes. The following shows my comparison table and the resulting priority vector:

 | Startup | File Save
Startup | 1 | 5
File Save | 0.200 | 1
Startup | 0.833 |
File Save | 0.167 |
So at this point, I have set up my problem (evaluate build quality for Build A, Build B, and Build C), determined my comparison attributes (startup performance, file save performance, user interface functionality, database functionality, network functionality), and computed the priority vectors to be used for each level of the comparison attributes.

Compare the Builds on Each Attribute

After setting up the build quality evaluation problem and determining the priority vectors for the comparison attributes, the next step in the analytic hierarchy process is to perform a comparison of each build based on each of the lowest level comparison attributes. In this example that means I need to compare Build A versus Build B versus Build C on each of the five comparison attributes: startup, file save, user interface, database, and network. The build comparison process is exactly the same as the attribute comparison process. So, for startup performance for example, I compare each build using the criteria in the table in Figure 4. In my example I got the following values:

Startup | Build A | Build B | Build C
Build A | 1 | 3 | 5
Build B | 0.333 | 1 | 2
Build C | 0.200 | 0.500 | 1
In other words, I determined that startup performance for Build A is "somewhat better" than it is for Build B, that startup performance for Build A is "definitely better" than that for Build C, and that startup performance for Build B is "slightly better" than that of Build C. Using the same algorithm as in the previous section (compute column totals, divide each cell by its column total, compute each row average), I establish the startup performance ranking vector:
Startup |
Build A | 0.648
Build B | 0.230
Build C | 0.122
Using the same process for the remaining four comparison attributes, I compare builds and compute ranking vectors, as shown in Figure 6.

Figure 6 Quality Attributes and Ranking Vectors (Close Figure 6)

File Save | Build A | Build B | Build C | File Save Ranking Vector
Build A | 1 | 2 | 4 | Build A | 0.571
Build B | 0.500 | 1 | 2 | Build B | 0.286
Build C | 0.250 | 0.500 | 1 | Build C | 0.143
User Interface | Build A | Build B | Build C | User Interface Ranking Vector
Build A | 1 | 1 | 3 | Build A | 0.443
Build B | 1 | 1 | 2 | Build B | 0.387
Build C | 0.333 | 0.500 | 1 | Build C | 0.170
Database | Build A | Build B | Build C | Database Ranking Vector
Build A | 1 | 3 | 6 | Build A | 0.639
Build B | 0.333 | 1 | 4 | Build B | 0.274
Build C | 0.167 | 0.250 | 1 | Build C | 0.087
Network | Build A | Build B | Build C | Network Ranking Vector
Build A | 1 | 4 | 5 | Build A | 0.638
Build B | 0.250 | 1 | 5 | Build B | 0.273
Build C | 0.200 | 0.200 | 1 | Build C | 0.089

Aggregate Weights to Produce Final Evaluation

After setting up the build comparison problem, computing the attribute priority vectors, and computing the build comparison ranking vectors, the next step is to aggregate all the intermediate data to produce the final evaluation metrics. As with the previous steps, it's easiest to explain by example. If I put the intermediate results into a table, I get one like you see in Figure 7.

Figure 7 Intermediate Results (Close Figure 7)

Performance (.200) |  |  | Functionality (.800)
 | Startup (.833) | File Save (.167) | User Interface (.681) | Database (.216) | Network (.103)
Build A | 0.648 | 0.571 | 0.443 | 0.639 | 0.638
Build B | 0.230 | 0.286 | 0.387 | 0.274 | 0.273
Build C | 0.122 | 0.143 | 0.170 | 0.087 | 0.089

The final quality value for Build A is:

(.200)(.833)(.648) +
(.200)(.167)(.571) +
(.800)(.681)(.443) +
(.800)(.216)(.639) +
(.800)(.103)(.638) = 0.531
In other words, the final quality metric for each build is the weighted sum of its attribute rankings. In the same way, the final quality value for Build B is:
(.200)(.833)(.230) +
(.200)(.167)(.286) +
(.800)(.681)(.387) +
(.800)(.216)(.274) +
(.800)(.103)(.273) = 0.329
And the final quality value for Build C is:
(.200)(.833)(.122) +
(.200)(.167)(.143) +
(.800)(.681)(.170) +
(.800)(.216)(.087) +
(.800)(.103)(.089) = 0.140

At this point I can now interpret my build quality. Because the ranking vector values sum to 1.0 you can roughly compare each build on a percentage basis. In this example, given the three builds, Build A is loosely 0.53 - 0.33 = 0.20, or 20 percent better than Build B. However this interpretation is somewhat misleading because it only applies when all three builds are taken into account. In other words, if I added a fourth build to my analysis, my quality vector would change and Build A would be less than 20 percent better than Build B. And let me caution you that the 3-decimal precision of the data is somewhat deceptive. Because the input data derived from the table in Figure 4 is relatively crude, it's a good idea to compare builds using only two decimals at most.

In my opinion the best way to interpret the results of build quality using the analytic hierarchy process is to view it strictly as trend data. Here you can see a relatively clear improvement trend. Based on my experience, as a general rule of thumb when comparing three builds, a difference of 0.15 or more is a significant (in the normal sense of the word, not in the statistical sense) difference. After you have some experience working with the analytic hierarchy process in your environment you'll gain an understanding of the metrics you get.

Discussion

As I've shown, the analytic hierarchy process is an elegant and powerful method for quantifying the overall quality of software systems. There are many other techniques you could use for meta-quality analysis, but the analytic hierarchy process has several appealing characteristics. The use of a subjective scale ("much more important") rather than a quantitative scale is particularly useful in software quality meta analysis. For example, suppose you are measuring performance. A difference of 2 seconds versus 3 seconds (50 percent difference) for a file save operation may not be equally as important as a 4 second versus 6 second (also 50 percent) time difference. Also useful is the pair-wise comparison technique used by analytic hierarchy process. It's usually much easier to compare two items at a time than to compare many items all at once.

There are certain disadvantages to using the analytic process to measure your build quality. As I mentioned, the subjective scale is very useful in software quality analysis but precisely because the scale is subjective, it is subject to human error. Take for example my comparison of the three builds based on network functionality:

Database | Build A | Build B | Build C
Build A | 1 | 3 | 6
Build B |  | 1 | 4
Build C |  |  | 1
If you examine the data closely you'll see it is not entirely consistent. If Build A is 3 "good units" better than Build B, and Build A is 6 "good units" better than Build C, then you would think Build B should be exactly 3 "good units" better than Build C, but I rated Build B as 4 relative to Build C.

The analytic hierarchy process has a neat way to estimate a so-called inconsistency index for each comparison matrix. Showing you how to compute an inconsistency index estimate is outside the scope of this column, but it is quite easy to calculate. For the comparison matrix shown earlier, the inconsistency index is 0.05—which is below the generally accepted threshold value of 0.10. And, in fact, my comparison of the three builds based on network functionality is very inconsistent and in a production scenario I would investigate and revise those comparisons. Some variations of the analytic hierarchy process modify the attribute ranking vector values based on the corresponding inconsistency index.

Another disadvantage of the analytic hierarchy's subjective scale is that it is vulnerable to human psychology—there is a tendency to want to see improvement in builds and so you must be careful not to subconsciously inflate rankings of more recent builds. As always, your final metrics will only be as good as your input data.

In my opinion the biggest disadvantage of using the analytic hierarchy process is that the number of comparison tables can become very large if you use a lot of comparison attributes. This can lead to a tendency to exclude valid comparison attributes in order to keep the number of calculations manageable. In this column I used Excel to manage the analytic hierarchy process, but there are several excellent commercial tools available that simplify the data input process, perform calculations automatically, and have nice reporting capabilities.

Conclusion

Although the analytic hierarchy process has been the subject of many research papers and the general consensus is that the technique is both technically valid and practically useful, there are critics of the method. My view on the matter is that you should use the analytic hierarchy process as just one component of your overall build quality analysis. It's a valuable supplement to other objective and subjective techniques that measure product quality. In general you should not rely completely on any single software system quality metric or technique.

Here I've shown you how easy and powerful the technique is when used to monitor build quality. Based on my experience, using the analytic hierarchy process to measure overall software quality represents a significant enhancement to traditional techniques like bug count metrics and milestone checklists.

Furthermore, you can use the technique in other areas. On one project I used it for competitive analysis to assess the overall quality of my product relative to major competing products. I've heard of other testers using similar methodologies to report on the health of a new build of a product based on weighting the results of automated build verification tests. As the software development environment continues to mature, techniques like the analytic hierarchy process will become increasingly important components of your software engineering skill set.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Security Briefs: Customizing GINA, Part 2

Security Briefs
Customizing GINA, Part 2
Keith Brown

Code download available at: SecurityBriefs0506.exe (274KB)
Contents
WlxNegotiate and WlxInitialize
WlxDisplaySASNotice
WlxLoggedOutSAS
Calling LsaLogonUser
Other WlxLoggedOutSAS Considerations
WlxActivateUserShell
WlxLoggedOnSAS and WlxDisplayLockedNotice
WlxWkstaLockedSAS
Trivial Functions
Supporting Remote Desktop
Conclusion

GINA, the Graphical Identification and Authentication component, is a part of WinLogon that you can customize or replace. Last month I introduced GINA customization; this month, I'm going to drill down to implement each of the GINA entry points. If you have not read last month's Security Briefs column, I strongly suggest that you start there before diving into this one (see Security Briefs: Customizing GINA, Part 1).

WlxNegotiate and WlxInitialize

I covered the two simple functions WlxNegotiate and WlxInitialize in last month's column. WlxNegotiate and WlxInitialize allow you to negotiate versions with WinLogon and exchange context handles for state management. WinLogon gives GINA a handle (hWlx), and GINA gives WinLogon a pointer to its internal state (pWlxContext). My implementation uses a class called Gina to hold this state, which dispatches each exported GINA function to member functions on the Gina class. I instantiate an instance of the Gina class during the second of these functions, WlxInitialize, so both Negotiate and Initialize are implemented as static methods on the Gina class.

My sample GINA uses Ctrl+Alt+Del secure attention sequence (SAS) events just like the default GINA does. If you need this functionality, in WlxInitialize you should call WlxSetOption and set the WLX_OPTION_USE_CTRL_ALT_DEL option to TRUE. If you forget to do this, you will not get any SAS events when the user presses Ctrl+Alt+Del.

WlxDisplaySASNotice

The next function normally called by WinLogon is WlxDisplaySASNotice, and as with all the rest of the functions you'll see, the export from my DLL simply takes the GINA context (pWlxContext), casts it to my Gina class, and calls a corresponding method, as shown here:

VOID WINAPI WlxDisplaySASNotice(PVOID pWlxContext) {
    LDB(L"-->WlxDisplaySASNotice");
    ((Gina*)pWlxContext)->DisplaySASNotice();
    LDB(L"<--WlxDisplaySASNotice");
}

Note the logging macros before and after the call. This is how each of my exported functions is implemented: I log that the function was called, I dispatch the call to the Gina class, then I log that I'm returning. LDB (which stands for Log Debug) compiles to NULL for release builds. In debug builds, this generates a line of output in a log file on the local hard disk. Since debugging a deployed GINA is tricky, I find that a liberal sprinkling of debug log messages coupled with the debug entrypoint I discussed last month can help you avoid the need for a symbolic debugger entirely during development.

Now let's look at the implementation of Gina::DisplaySASNotice:

void Gina::DisplaySASNotice() {
    NoticeDialog dlg(_pWinLogon, IDD_SASNOTICE);
    dlg.Show();
}
This really is a simple function. You should display a modal dialog box that tells the user how to log in. The odd thing about this dialog is that it has no buttons on it, and no close box, so there's no way for the user to dismiss it (see Figure 1).

Figure 1 The SAS Notice Dialog
Figure 1 The SAS Notice Dialog

As I discussed last month, you'll call back into WinLogon whenever you want to display modal dialogs. If you look at the NoticeDialog class, you'll see that it derives from a base class called GinaModalDialog, which provides a very basic framework for handling modal dialogs. It calls into WinLogon's WlxDialogBoxParam method to display the dialog, passing a pointer to itself as the parameter, which the dialog procedure then grabs via WM_INITDIALOG and tucks away as part of the window state. This is coupled with a virtual method named DialogProc that each derived dialog may implement however it likes, which makes dialog state management very simple. I just use member variables for dialogs that need to maintain state. Conceptually this is similar to the way MFC worked, although it's considerably simpler.

The notice dialog doesn't do a thing. It just waits for WinLogon to dismiss it, typically because the user pressed Ctrl+Alt+Del. If your GINA needs to listen for custom SAS events on another thread, your notice dialog should watch for a custom window message that you define (WM_USER, for example), and call WlxSasNotify when this message comes in. Your background thread should post this message whenever it detects the custom SAS. As I mentioned last month, this keeps all your interaction with WinLogon on a single thread, as it should be.

WlxLoggedOutSAS

WlxLoggedOutSAS is the next function WinLogon will call in your GINA, when the user presses Ctrl+Alt+Del or you generate a custom SAS from your notice dialog. This is by far the most complicated function your GINA will implement, although conceptually it's not that difficult. This is where GINA collects credentials of some sort from the user, and either logs the user in or rejects the logon attempt. My sample displays a dialog asking for a user name and password, as shown in Figure 2.

int WlxLoggedOutSAS(
  IN PVOID pWlxContext,
  IN DWORD dwSasType,
  OUT PLUID pAuthenticationId,
  OUT PSID pLogonSid,
  OUT PDWORD pdwOptions,
  OUT PHANDLE phToken,
  OUT PWLX_MPR_NOTIFY_INFO pNprNotifyInfo,
  OUT PVOID* pProfile
);

Figure 2 The Logged Out SAS Dialog
Figure 2 The Logged Out SAS Dialog

Besides your context pointer (pWlxContext), WinLogon only passes in one other parameter: dwSasType. Microsoft defines several SAS types, reserving the range of values 0 to 127. If you define your own SAS type, make sure its value is 128 or greater. There are two SAS types my sample cares about:

WLX_SAS_TYPE_CTRL_ALT_DEL
WLX_SAS_TYPE_AUTHENTICATED
The second SAS type (WLX_SAS_TYPE_AUTHENTICATED) is currently undocumented, but you must handle it if you want to fully support the Windows® XP Remote Desktop feature. I'll discuss it later.

The return value from WlxLoggedOutSAS tells WinLogon what GINA wants it to do:

WLX_SAS_ACTION_NONE
WLX_SAS_ACTION_SHUTDOWN
WLX_SAS_ACTION_LOGON
You should return NONE if you want to cancel the logon attempt (for example, if the user presses the Cancel button or supplies invalid credentials too many times). SHUTDOWN tells WinLogon to shut down the system. To demonstrate this, my sample allows any user to press the Shutdown button on the logon dialog. However, in your own GINA, you may not want an anonymous user to be able to do this.

LOGON indicates that GINA has successfully logged on the user. If you return this value, you must also supply the other out parameters from WlxLoggedOutSAS that indicate the details of the logon. This includes a token handle, profile path information, and other details. Your best bet for obtaining these values comes from calling the low-level function LsaLogonUser, which is complicated enough to warrant its own section.

Calling LsaLogonUser

The LsaLogonUser function accepts credentials and produces a logon session if the credentials are valid. More specifically, this function dispatches the request to an authentication package such as Kerberos or MSV1_0 (the Windows NT® logon provider), which will then verify the credentials and establish a session. Because there are many different forms of credentials, ranging from a simple user name and password to a Kerberos server ticket and authenticator, the input to this function is defined as a binary BLOB that will be interpreted by the underlying authentication package. This BLOB is passed via the AuthenticationInformation parameter shown in Figure 3.

Figure 3 LsaLogonUser (Close Figure 3)
NTSTATUS LsaLogonUser(
  IN HANDLE LsaHandle,
  IN PLSA_STRING OriginName,
  IN SECURITY_LOGON_TYPE LogonType,
  IN ULONG AuthenticationPackage,
  IN PVOID AuthenticationInformation, // blob
  IN ULONG AuthenticationInformationLength,
  IN PTOKEN_GROUPS LocalGroups,
  IN PTOKEN_SOURCE SourceContext,
  OUT PVOID* ProfileBuffer,
  OUT PULONG ProfileBufferLength,
  OUT PLUID LogonId,
  OUT PHANDLE Token,
  OUT PQUOTA_LIMITS Quotas,
  OUT PNTSTATUS SubStatus
);

There are a lot of parameters to this method, so I've taken the liberty of marking the input and output parameters. I'm not going to explain what each and every one of these parameters is for; you can find those details in the documentation. Instead I'll focus on what the current documentation doesn't tell you, and give you tips on using this function from a custom GINA.

My sample supports both domain logons (Kerberos) and local workstation logons (MSV1_0); fortunately the input BLOB for both of these providers looks exactly the same. Here's the MSV1_0 structure definition:

typedef struct _MSV1_0_INTERACTIVE_LOGON {
  MSV1_0_LOGON_SUBMIT_TYPE MessageType;
  UNICODE_STRING LogonDomainName;
  UNICODE_STRING UserName;
  UNICODE_STRING Password;
} MSV1_0_INTERACTIVE_LOGON;

Now here's the tricky part. The UNICODE_STRING fields are pointers to buffers that contain Unicode characters for the domain, user name, and password. You might be tempted to allocate these string buffers separately. If you do this, LsaLogonUser will fail. Instead you must dynamically allocate a block of memory large enough to hold the data structure shown in Figure 3, plus all the string buffers that go along with it. In other words, you must serialize this data structure into a single contiguous buffer before calling LsaLogonUser. Rather than bore you with the code, I'll simply point you to the helper function that forms this request. It's called _allocLogonRequest, and it can be found in the SecurityHelper.cpp file in the sample code available for download from the MSDN®Magazine Web site.

GINA should specify a LogonType of "Interactive" when logging in a user from WlxLoggedOutSAS, unless that user is logging in remotely via Remote Desktop, in which case "RemoteInteractive" should be used. Specify a LogonType of "Unlock" when allowing a user to unlock the workstation.

The only other tricky parameter to LsaLogonUser is LocalGroups, which allows the caller to specify any number of extra Security Identifiers (SIDs) that should be added to the resulting token as it's returned. Think about that for a moment. You have the power to make any user an administrator, for example. Heck, your GINA could include a checkbox, "Make me an admin for this session." If checked, you could specify the well-known SID for the local Administrators group via LocalGroups, and the user would suddenly be an administrator for the length of that logon session.

Don't get me wrong; I'm not suggesting that you do this, just like I'm not suggesting that you hardcode a backdoor into your GINA that allows anyone with a user name of "h@x0r" to log in using the built-in administrator account without knowing the password. But GINA can do these sorts of things, which is why I mentioned last month that you must put a strong access control list (ACL) on your custom GINA DLL file to ensure that a normal user doesn't overwrite it with a malicious GINA. And your GINA had better be bulletproof. You don't want an attacker to exploit a buffer overflow in a GINA and subsequently run arbitrary code there. Note that LsaLogonUser may only be called by code running in the trusted computing base, which GINA is part of by virtue of running inside WinLogon, which runs as SYSTEM. A normal user cannot call LsaLogonUser to elevate privileges.

My sample GINA passes NULL for LocalGroups, and unless you have a good reason not to, yours should as well.

Several outputs from LsaLogonUser are useful in your implementation of WlxLoggedOutSAS. For example, GINA must cache the Token parameter as part of its state. Also, the LogonId LUID is the unique 64-bit identifier for the new logon session, and it maps directly to the pAuthenticationId out parameter for WlxLoggedOutSAS.

The ProfileBuffer output from LsaLogonUser is a BLOB whose format depends on the logon provider. Once again, the Kerberos and MSV1_0 providers agree on the format of this BLOB, which is shown in Figure 4. This structure contains a wealth of information that a fully featured GINA needs. For example, the default GINA checks the PasswordMustChange field to see if the user's password is due to expire soon, and gives the user a chance to change it right then and there. Besides conveniences like this, you'll need to look at the ProfilePath field to be able to properly fill out the pProfile out parameter in WlxLoggedOutSAS, which is another BLOB that must be serialized.

Figure 4 The Interactive Profile (Close Figure 4)
typedef struct _MSV1_0_INTERACTIVE_PROFILE {
  MSV1_0_PROFILE_BUFFER_TYPE MessageType;
  USHORT LogonCount;
  USHORT BadPasswordCount;
  LARGE_INTEGER LogonTime;
  LARGE_INTEGER LogoffTime;
  LARGE_INTEGER KickOffTime;
  LARGE_INTEGER PasswordLastSet;
  LARGE_INTEGER PasswordCanChange;
  LARGE_INTEGER PasswordMustChange;
  UNICODE_STRING LogonScript;
  UNICODE_STRING HomeDirectory;
  UNICODE_STRING FullName;
  UNICODE_STRING ProfilePath;
  UNICODE_STRING HomeDirectoryDrive;
  UNICODE_STRING LogonServer;
  ULONG UserFlags;
} MSV1_0_INTERACTIVE_PROFILE, *PMSV1_0_INTERACTIVE_PROFILE;

Other WlxLoggedOutSAS Considerations

Once LsaLogonUser returns, if you've successfully established a logon, you can cache the user name and domain as part of your state for convenience, as they will be needed elsewhere. Another alternative would be to simply look up the user name and domain whenever they are needed, since you're already caching the user's token, but for domain accounts that will require round-trips to a domain controller, so I simply cache these values in member variables on the Gina class.

WinLogon normally loads the user's profile after WlxLoggedOutSAS returns with a successful logon. To get this default behavior, my sample sets *pdwOptions to 0. Your GINA can customize how the profile is loaded by setting this flag to WLX_LOGON_OPT_NO_PROFILE, which tells WinLogon that GINA has already loaded the profile.

The pNprNotifyInfo output parameter allows other network providers such as Novell Netware to get a peek at the user name and password being used so they can automatically log the user into their networks as well. This prevents the user from having to log on multiple times.

The pLogonSid output parameter points to a fixed sized buffer provided by WinLogon that is large enough to hold a logon SID of the form S-1-5-5-x-y, where x and y are unique values that identify the new logon session. Unfortunately, LsaLogonUser doesn't supply this SID, but you can find it by looking in the token. Ask GetTokenInformation for the TokenGroups class of information, then enumerate the group SIDs in the token until you find the one with the SE_GROUP_LOGON_ID flag. You can then use the CopySID function to copy this value into the buffer provided by pLogonSid. WinLogon uses this SID to grant permissions to the interactive window station and desktop. You can read more about this at What Is A Window Station.

Pay close attention to the return value from LsaLogonUser. If it fails, my sample is careful to check if the account requires an immediate password change or if the user's password has expired. If so, I pop up a password change dialog to give the user a chance to change her password. On any other failures I look up the corresponding error message by calling FormatMessage, display it to the user, and then return WLX_SAS_ACTION_NONE.

WlxActivateUserShell

Once the user is logged on, GINA is asked to launch the shell.

BOOL WlxActivateUserShell(
  IN PVOID pWlxContext,
  IN PWSTR pszDesktopName,
  IN PWSTR pszMprLogonScript,
  IN PVOID pEnvironment
);

My sample does what most GINAs should do: it launches USERINIT.EXE. Well, technically it looks in WinLogon's registry key for a named value called Userinit, and launches whatever programs are specified in this comma-delimited string. Most machines will have a value that looks something like this:

"C:\WINDOWS\system32\userinit.exe,"
This little program is responsible for running logon scripts and calling CreateProcess to start the user's shell, which is named in another value aptly called "Shell". Most machines will have the following value for Shell:
"Explorer.exe"

The trick here is that you can't simply call CreateProcess to launch USERINIT.EXE. If you did, it would run as SYSTEM just like GINA! Instead, you must call CreateProcessAsUser, which takes one extra argument: the handle to the token that your GINA got earlier from calling LsaLogonUser. This will cause USERINIT.EXE and consequently the user's shell to run in the new logon session you've created.

Also, be careful to specify the desktop and environment given to you by WinLogon. You can see my call to CreateProcessAsUser in the SecurityHelper.cpp file in the sample code. Once you return from this function, your GINA won't be called again until something interesting happens, such as the user pressing Ctrl+Alt+Del, in which case you'll see a call to WlxLoggedOnSAS.

WlxLoggedOnSAS and WlxDisplayLockedNotice

WinLogon calls WlxLoggedOnSAS when a user is logged on and a SAS occurs. The only interesting argument is dwSasType. My sample watches for WLX_SAS_TYPE_CTRL_ALT_DEL and pops up the dialog shown in Figure 5.

Figure 5 The Logged-On SAS Dialog
Figure 5 The Logged-On SAS Dialog

The implementation of this function is simply an exercise in getting some user input and returning a value to WinLogon indicating what the user needs. The only significant functionality you need to implement is a change password dialog, which should call NetUserChangePassword to attempt a password change.

To give you an idea of how easy this function is to implement, here are the constants you can return from it:

WLX_SAS_ACTION_NONE
WLX_SAS_ACTION_LOCK_WKSTA
WLX_SAS_ACTION_LOGOFF
WLX_SAS_ACTION_PWD_CHANGE
WLX_SAS_ACTION_TASKLIST
WLX_SAS_ACTION_SHUTDOWN_REBOOT
WLX_SAS_ACTION_SHUTDOWN
WLX_SAS_ACTION_SHUTDOWN_SLEEP
WLX_SAS_ACTION_SHUTDOWN_HIBERNATE
WLX_SAS_ACTION_SHUTDOWN_POWER_OFF
So if the user presses the Task Manager button, GINA simply returns the TASKLIST value, and WinLogon will launch the user's task manager. To lock the workstation, return LOCK_WKSTA, and so on. If the user asks to shut down the computer, my sample asks for confirmation and then returns the SHUTDOWN value, but if you wanted to, you could call the power management APIs to determine what other options are available, such as suspend or hibernate, and give the user further options.

The WlxDisplayLockedNotice function is as simple as WlxDisplaySASNotice. In fact, my sample code uses the same dialog class to display this dialog; it just uses a different dialog resource to get an appropriate interface. This dialog also has no buttons and will be dismissed only when a SAS is detected.

WlxWkstaLockedSAS

WinLogon calls WlxWkstaLockedSAS when a logged-on user has locked her workstation and a SAS occurs. This function is similar to WlxLoggedOutSAS, except that it takes no out parameters:

int WlxWkstaLockedSAS(
  IN PVOID pWlxContext,
  IN DWORD dwSasType
);

The trick to implementing this function is realizing that there are two use cases to be considered. The normal case is where the user who locked her workstation is returning and unlocking it. The corner case is when someone other than the logged-on user tries to unlock the computer. If that person is an administrator, they should be allowed to forcefully log off the user in order to access the workstation themselves.

Here's the approach I take in the sample. I pop-up my logon dialog, and populate the name and domain of the currently logged-on user that I cached earlier in WlxLoggedOutSAS. I then give keyboard fo-cus to the password edit box. This simplifies the normal case where the user is simply unlocking her own workstation.

Once the password prompt returns, I attempt to log the user by calling LsaLogonUser with a logon type of Unlock, which is a special type of logon designed just for GINA, and properly audits the attempt to unlock the workstation.

If the logon succeeds, I look at the resulting token and compare the user SID with the SID of the currently logged-on user. If these SIDs are the same, then I know the user has simply returned to unlock her workstation. I close the new token and return WLX_SAS_ACTION_UNLOCK_WKSTA to WinLogon to indicate that the workstation should be unlocked.

If the SIDs don't match, I check to see if the new user is an administrator by calling CheckTokenMembership, looking for the well-known local Administrators group SID.

If the user is an administrator, I return WLX_SAS_ACTION_FORCE_LOGOFF, and WinLogon logs off the user and brings GINA back around to display the logon prompt.

Trivial Functions

There are several trivial functions that my sample implements, so including WlxIsLockOk, WlxIsLogoffOk, WlxLogoff, WlxShutdown, and others, I'll mention them briefly here. Their names are usually self-explanatory.

In WlxIsLockOk and WlxIsLogoffOk, I simply return TRUE. My sample GINA never stops anyone from logging off or locking the workstation, although yours might need to do this.

In WlxLogoff, I clear any variables that have to do with the currently logged-on user. This means closing the user's token and freeing the cached strings for the user name and domain.

WlxShutdown is a pretty obvious notification. I originally assumed that this was the last message I'd ever receive from WinLogon (that's my experience so far). But I've been told by reliable sources that it may be possible to receive status notifications even after WlxShutdown has been called. With this in mind, I keep enough of the GINA alive to process any of these extra messages that happen to come along.

WlxNetworkProviderLoad is obsolete, according to sources inside Microsoft. My GINA simply returns FALSE here, since the entry point is never actually called by WinLogon.

And finally, DisplayStatusMessage, RemoveStatusMessage, and GetStatusMessage allow WinLogon to give me messages for the user from time to time, which I display in a modeless dialog box.

Supporting Remote Desktop

In order to support Terminal Services and the Windows XP Remote Desktop feature, you need to make special considerations in your code. Think about what happens when you use the remote desktop client, MSTSC.EXE, to log onto a remote workstation. What happens when you press Ctrl+Alt+Del? You're raising an interrupt on your local hardware, not on the remote machine.

If you want to be friendly to Remote Desktop, the first thing to do is detect whether you're running in a session for a remote user. That information will be useful to you throughout your GINA's lifetime, and it's easy to discover:

bool UserIsRemote() {
  return 0 != GetSystemMetrics(SM_REMOTESESSION);
}
My sample GINA uses this to skip the Ctrl+Alt+Del requirement for remote desktop users by calling WlxSasNotify to simulate Ctrl+Alt+Del on their behalf. You can see this in my implementation of WlxDisplaySASNotice.

When a user first logs on through Remote Desktop on Windows XP, something rather magical happens inside WinLogon, and GINA must help facilitate it. If you have two workstations side by side you can see this happen. Say workstation A just booted up and is waiting for someone to log on. Its GINA is waiting in WlxDisplaySASNotice. On workstation B, you connect to A via remote desktop and log on. As soon as you enter your logon credentials via B's screen, you'll see something change back on A's screen. What's happening here is a shuffle of terminal services sessions. In Remote Desktop, the operating system ensures that only one user is ever connected to the machine, either via the console or via a remote session. Part of this bookkeeping is ensuring that the logged-on user is always working in Terminal Services session zero.

In this case, the GINA that was loaded on workstation A is running in session 0, but the user is connecting via a new session. Session 1 is a temporary session with its own instance of WINLOGON.EXE, which has also loaded your GINA. This copy of GINA authenticates the user in session 1, and then the magic shuffle starts. After you return from WlxLoggedOutSAS with a successful logon, you'll see a call to WlxGetConsoleSwitchCredentials in session 1. This is the last chance your GINA has to communicate the results of the logon before it disappears forever!

The data structure defined for this function is pretty complicated, but you don't need to fill it all out. One thing you absolutely need to pass is the token you just obtained from LsaLogonUser. WinLogon also expects the UserName field (things won't work properly if you don't pass at least this field).

WinLogon in session 1 now marshals this data, pipes it over to session 0, and calls the GINA's WlxLoggedOutSAS there with the special value WLX_SAS_TYPE_AUTHENTICATED. This constant isn't in the documentation, but that's an oversight, not a secret. This is your signal to grab the data from the session 1 GINA by calling WlxQueryConsoleSwitchCredentials, and return the user's token to WinLogon in session 0.

Interestingly enough, at this point if you call the UserIsRemote function I described earlier, you'll see that session 0 has suddenly gone remote! The user on workstation B is now using session 0, and the login will proceed as normal. Session 1 has been terminated. Back on workstation A's console, yet another temporary Terminal Services session has been constructed, typically session 2 in the scenario I've described. This session exists to inform the user that the workstation is in use, and allow the user to switch back to the console if so desired.

Another thing you'll want to check for is whether the user supplied credentials via the Remote Desktop client. You can discover this by calling WinLogon's WlxQueryTsLogonCredentials function. If the user has already provided credentials, you don't want to prompt her to supply them again, so this is mainly a convenience.

If this all sounds complicated, you're right, it really is. That's why the team working on the next version of Windows (code-named "Longhorn") is trying to get rid of this rather messy interface once and for all. But if you're customizing the logon experience on Windows XP and need to support Remote Desktop, you must jump through these hoops. The sample accompanying this article illustrates how it's done, including a number of details that I didn't have room to discuss in these two columns.

Conclusion

Writing a custom GINA is not easy. MSGINA is a very complicated piece of machinery, and replacing it is not trivial. But a lot of folks have found it necessary over the years. If you can't use the stub type GINA that I discussed last month and therefore are forced to write a custom GINA from scratch, I hope you find these columns with their accompanying samples helpful in your quest. Just remember that you are implementing the heart of the interactive logon plumbing in Windows, and it's critical that your code be absolutely correct and bulletproof! To share knowledge with other GINA developers, please visit my wiki at Customizing GINA.

Send your questions or comments for Keith at briefs@microsoft.com.

Keith Brown is a co-founder of Pluralsight, specializing in developing and delivering high quality training for software developers. Keith's most recent book is The .NET Developer's Guide to Windows Security. Keith also keeps a web log and contact info at www.pluralsight.com/keith.


Wicked Code: Power Programming Tips for ASP.NET 2.0

Wicked Code
Power Programming Tips for ASP.NET 2.0
Jeff Prosise

Code download available at: WickedCode0506.exe (118KB)
Contents
Storing Site Maps in SQL Server
Forwarding and Capturing Trace Output
Fetching Images from Databases
Building Asynchronous Pages

In the February 2005 issue, I introduced five lesser-known features of ASP.NET 2.0 that have the potential to make a significant impact on the security, performance, and robustness of your code (see Wicked Code: Five Undiscovered Features on ASP.NET 2.0). This month I'd like to continue down that path by offering still more power programming tips for ASP.NET 2.0.

The code samples presented in this column were written and tested using the Beta 1 Refresh version of Visual Studio 2005 and the .NET Framework 2.0. They could require changes when the final versions of the products ship.

Storing Site Maps in SQL Server

One of the major new features in ASP.NET 2.0 is data-driven site navigation. You start by providing an XML site map detailing your site's structure. Then you add a SiteMapDataSource control to the page and bind it to a TreeView or Menu control to produce a navigation UI. The SiteMapDataSource control reads the XML site map and provides the data to the TreeView or Menu as well as to any SiteMapPath controls also present in the page. If the structure of your site changes, you update the site map and the navigation UI updates automatically.

One shortcoming of the site navigation bits is that they include only one built-in site map provider. That provider, currently named XmlSiteMapProvider, requires that site maps be stored in XML files conforming to a particular schema. Developers are already asking for a SQL site map provider so they can store site maps in databases alongside other content on their site.

Fortunately, site map providers are not incredibly difficult to write. Figure 1 contains the source code for a custom site map provider named SqlSiteMapProvider that retrieves site map information from a SQL Server™ database. It supports hierarchical site maps of unlimited depth. It also supports security trimming—a feature of XmlSiteMapProvider that enables site map nodes to be hidden from users who lack permission to view them. SqlSiteMapProvider looks and behaves like XmlSiteMapProvider in almost every way, but because it uses SQL Server as the site map data store, it adds a useful new capability to ASP.NET 2.0.

Figure 1 SqlSiteMapProvider.cs (Close Figure 1)
public class SqlSiteMapProvider : StaticSiteMapProvider
{
    static readonly string _errmsg1 =
        "Missing connectionStringName attribute";
    static readonly string _errmsg2 = "Duplicate node ID";
    SiteMapNode _root = null;
    string _connect;

    public override void Initialize (string name,
        NameValueCollection attributes)
    {
        base.Initialize (name, attributes);

        if (attributes == null)
            throw new ConfigurationException (_errmsg1);

        _connect = attributes["connectionStringName"];
        if (String.IsNullOrEmpty (_connect))
            throw new ConfigurationException (_errmsg1);
    }

    [MethodImpl(MethodImplOptions.Synchronized)]
    public override SiteMapNode BuildSiteMap()
    {
        // Return immediately if this method has been called before
        if (_root != null) return _root;

        // Create a dictionary for temporary node storage and lookup
        Dictionary<int, SiteMapNode> nodes =
            new Dictionary<int, SiteMapNode> (16);

        // Query the database for site map nodes
        SqlConnection connection = new SqlConnection(
           ConfigurationSettings.ConnectionStrings[
               _connect].ConnectionString);

        try
        {
            connection.Open ();
            SqlCommand command = new SqlCommand(
                "SELECT ID, Title, Description, Url, " +
                "Roles, Parent FROM SiteMap ORDER BY ID", connection);
            SqlDataReader reader = command.ExecuteReader ();
            int id = reader.GetOrdinal("ID");
            int url = reader.GetOrdinal ("Url");
            int title = reader.GetOrdinal ("Title");
            int desc = reader.GetOrdinal("Description");
            int roles = reader.GetOrdinal ("Roles");
            int parent = reader.GetOrdinal("Parent");

            if (reader.Read())
            {
                // Create the root SiteMapNode
                _root = new SiteMapNode(this,
                   reader.GetInt32(id).ToString(),
                   reader.IsDBNull(url) ? null : reader.GetString(url),
                   reader.GetString(title),
                   reader.IsDBNull(desc) ? null : reader.GetString(desc));

                if (!reader.IsDBNull(roles))
                {
                    string rolenames = reader.GetString(roles).Trim ();
                    if (!String.IsNullOrEmpty(rolenames))
                    {
                        string[] rolelist = rolenames.Split(
                            new char[] { ',', ';' }, 512);
                        _root.Roles = rolelist;
                    }
                }

                //  Add "*" to the roles list if no roles are specified
                if (_root.Roles == null) _root.Roles = new string[]{"*"};

                // Record the root node in the dictionary
                if (nodes.ContainsKey (reader.GetInt32(id)))
                    throw new ConfigurationException (_errmsg2);
                nodes.Add(reader.GetInt32(id), _root);

                // Add the node to the site map
                AddNode(_root, null);

                // Build a tree of SiteMapNodes underneath the root
                while (reader.Read())
                {
                    SiteMapNode node = new SiteMapNode(this,
                      reader.GetInt32(id).ToString(),
                      reader.IsDBNull(url) ? null : reader.GetString(url),
                      reader.GetString(title), reader.IsDBNull(desc) ?
                          null : reader.GetString(desc));

                    if (!reader.IsDBNull(roles))
                    {
                         string rolenames =
                             reader.GetString(roles).Trim ();
                         if (!String.IsNullOrEmpty(rolenames))
                         {
                             string[] rolelist = rolenames.Split
                                 (new char[] { ',', ';' }, 512);
                             node.Roles = rolelist;
                         }
                     }

                     // If the node lacks roles information,
                     // "inherit" that information from its parent
                     SiteMapNode parentnode =
                         nodes[reader.GetInt32(parent)];
                     if (node.Roles == null)
                         node.Roles = parentnode.Roles;

                     // Record the node in the dictionary
                     if (nodes.ContainsKey(reader.GetInt32(id)))
                         throw new ConfigurationException(_errmsg2);
                     nodes.Add(reader.GetInt32(id), node);

                     // Add the node to the site map
                     AddNode(node, parentnode);
                 }
             }
        }
        finally
        {
            connection.Close ();
        }

        // Return the root SiteMapNode
        return _root;
    }

    protected override SiteMapNode GetRootNodeCore ()
    {
        BuildSiteMap ();
        return _root;
    }
}

The key method in SqlSiteMapProvider is BuildSiteMap. This method is called by ASP.NET shortly after the provider is loaded to build the site map, which is basically a collection of SiteMapNodes linked together to form a tree. Each SiteMapNode represents one node in the site map. Figure 2 shows the properties.

Figure 2 Site Map Node Properties (Close Figure 2)

Property | Description
Title | Specifies the text that a navigation control displays for the node.
Url | Specifies the URL the user is sent to when the node is clicked.
Description | Specifies the descriptive text that's displayed if the cursor hovers over the node.
Roles | Specifies the role or roles that are permitted to view the node if security trimming is enabled ("*" if anyone can view it). Multiple roles can be specified using commas or semicolons as separators.

SqlSiteMapProvider's implementation of BuildSiteMap queries the site map database. Then it iterates over the records, one by one transforming them into SiteMapNodes. At the end, it hands the site map over to ASP.NET by returning a reference to the root site map node. And because providers must be thread-safe, SqlSiteMapProvider wraps the entire BuildSiteMap method in a MethodImpl attribute that serializes concurrent thread accesses.

This raises the question: how do you structure the site map database? The SQL script in Figure 3 creates a SiteMap table that's compatible with SqlSiteMapProvider. Each record added to the table represents one site map node, and each has fields that map to the SiteMapNode properties mentioned previously as well as fields used to denote the relationship between nodes. Each node must have a unique ID, which is stored in the ID field. To parent one node to another, you set the child node's Parent field equal to the ID of the parent. All nodes except the root node must have a parent. In addition, because of the way BuildSiteMap is implemented, a node can only be parented to another node with a lesser ID. (Note the ORDER BY clause in the database query.) For example, a node with an ID of 100 can be the child of a node with an ID of 99, but it can't be the child of a node with an ID of 101.

Figure 3 Site Map Database Schema (Close Figure 3)
CREATE TABLE [dbo].[SiteMap] (
    [ID]           [int]     NOT NULL,
    [Title]        [varchar] (32) NOT NULL,
    [Description]  [varchar] (512),
    [Url]          [varchar] (512),
    [Roles]        [varchar] (512),
    [Parent]       [int]
) ON [PRIMARY]
GO

ALTER TABLE [dbo].[SiteMap] ADD
    CONSTRAINT [PK_ID] PRIMARY KEY  CLUSTERED
    (
        [ID]
    ) ON [PRIMARY]
GO

Figure 4 shows one of the databases that I used to test SqlSiteMapProvider. Note that I didn't explicitly initialize the Roles field of every node. Nodes that lack Roles values inherit the Roles values of their parents, and if the root node's Roles value isn't specified, then "*" is assumed. Roles values are ignored if security trimming is disabled, and like XmlSiteMapProvider, SqlSiteMapProvider disables security trimming by default. You can enable it by including a securityTrimmingEnabled="true" attribute in the element that registers SqlSiteMapProvider.

Figure 4 Sample Site Map Database (Close Figure 4)

Figure 5 shows how to register SqlSiteMapProvider in Web.config and activate it by making it the default provider. Note the connectionStringName attribute, which specifies the name of a connection string in the <connectionStrings> section of Web.config. This required attribute identifies the connection string that SqlSiteMapProvider uses to connect to the site map database.

Figure 5 Registering SqlSiteMapProvider (Close Figure 5)
<configuration>
  <connectionStrings>
    <add name="SiteMapConnectionString" connectionString="..."
      providerName="System.Data.SqlClient" />
  </connectionStrings>
  <system.web>
    <siteMap defaultProvider="SqlSiteMapProvider" enabled="true">
      <providers>
        <add name="SqlSiteMapProvider" type="SqlSiteMapProvider"
          securityTrimmingEnabled="true"
          connectionStringName="SiteMapConnectionString" />
      </providers>
    </siteMap>
  </system.web>
</configuration>

Forwarding and Capturing Trace Output

ASP.NET 1.x includes a handy tracing feature that lets you produce diagnostic output using the TraceContext class, which is normally accessed through Page.Trace. Trace output can be directed to the page or to the trace log front-ended by Trace.axd. It cannot be directed elsewhere without resorting to rather exotic means, such as custom HTTP modules incorporating response filters built from custom Stream-derived classes.

ASP.NET 2.0 introduces flexibility to trace output. For starters, you can declaratively forward ASP.NET trace output to System.Diagnostics.Trace output and vice versa. The following Web.config file shows how. Apply this configuration to a project and run it in Visual Studio® 2005 in debug mode, and the trace output will magically appear in the Visual Studio output window:

<configuration>
  <system.web>
    <trace enabled="true" writeToDiagnosticsTrace="true" />
  </system.web>
</configuration>

Here's another example of the ASP.NET 2.0 kinder, gentler approach to trace output:

void Page_Load (object sender, EventArgs e) {
    Trace.TraceFinished +=
        new TraceContextEventHandler (OnTraceFinished);
}

void OnTraceFinished (object sender, TraceContextEventArgs e) {
    foreach (TraceContextRecord record in e.TraceRecords) {
        Response.Write (String.Format ("{0}: {1}<br>",
            record.Category, record.Message));
    }
}
The TraceContext class now features a TraceFinished event that provides programmatic access to trace output. The sample just shown echoes trace output to the page using Response.Write. You could just as easily write the trace output to a database, an XML file, or another kind of storage medium, perhaps for viewing it with a custom diagnostic tool.

Fetching Images from Databases

One of my favorite features originally scheduled for ASP.NET 2.0 is the new DynamicImage control, which, among other things, makes it very easy to display images retrieved from databases. The related ImageField type makes it a snap to display database images in GridView and DetailsView controls, and the new ASIX file type simplifies dynamic image generation.

Unfortunately, these features have been cut from the product for Beta 2 (ImageField won't be removed, but it will lose its ability to bind to database images.) So what's a poor developer to do? You can still display database images the old-fashioned way with custom HTTP handlers, but all too often I see developers use ASPX files as image generators. That impedes performance. And while writing HTTP handlers isn't terribly difficult, it's something most developers would prefer not to do if given a choice.

That's why I decided to build the SqlImage control (see Figure 6). While certainly not the last word in displaying database images in Web pages, it's capable of handling routine image retrieval and display chores that might make it useful as a replacement for DynamicImage. SqlImage derives from System.Web.UI.WebControls.Image and extends it by adding the properties that are listed in Figure 7.

Figure 7 SqlImage Properties (Close Figure 7)

Property | Description
ConnectionString | Specifies the connection string SqlImage uses to query the database for images
SelectCommand | Specifies the SQL command SqlImage uses to query the database for images
MimeType | Specifies the image format (default="image/jpeg")
EnableCaching | Tells SqlImage whether to store the images it retrieves in the ASP.NET application cache (default=false)
CacheDuration | Specifies how long (in seconds) images should be cached if EnableCaching is true
Figure 6 SqlImage.cs (Close Figure 6)
namespace MsdnMag
{
    [ToolboxData ("<{0}:SqlImage Runat=\"server\">")]
    public class SqlImage : System.Web.UI.WebControls.Image
    {
        [Browsable(true), Category("Database"), DefaultValue(""),
         Description("Database connection string")]
        public string ConnectionString
        {
            get
            {
                object o = ViewState["ConnectionString"];
                return (o == null) ? String.Empty : (string)o;
            }
            set { ViewState["ConnectionString"] = value; }
        }

        ... // implementations of the other properties listed in Figure 7

        protected override void OnPreRender(EventArgs e)
        {
            if (!String.IsNullOrEmpty(ConnectionString) &&
                !String.IsNullOrEmpty(SelectCommand))
            {
                ImageUrl = String.Format (
                    "~/__ImageGrabber.axd?n={0}&s={1}&m={2}&c={3}&d={4}",
                    EncryptionHelper.Encrypt(ConnectionString),
                    EncryptionHelper.Encrypt(SelectCommand),
                    MimeType, EnableCaching ? "1" : "0", CacheDuration);
            }
        }
    }

    public class ImageGrabber : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            // Read the query string parameters
            string connectionString = context.Request["n"];
            if (String.IsNullOrEmpty(connectionString)) return;

            string selectCommand = context.Request["s"];
            if (String.IsNullOrEmpty(selectCommand)) return;

            string mimeType = context.Request["m"];
            if (String.IsNullOrEmpty(mimeType)) mimeType = "image/jpeg";

            bool enableCaching = false;
            string caching = context.Request["c"];
            if (!String.IsNullOrEmpty(caching) && caching == "1")
                enableCaching = true;

            int cacheDuration = 0;
            string duration = context.Request["d"];
            if (!String.IsNullOrEmpty(duration))
                cacheDuration = Convert.ToInt32 (duration);

            byte[] image = null;
            string cacheKey = String.Format("__ImageGrabber_{0}_{1}",
                connectionString, selectCommand);

            // Read the image from the cache if caching is enabled
            if (enableCaching) image = (byte[])context.Cache[cacheKey];

            // If image isn't cached, query the database
            if (image == null)
            {
                SqlConnection connection = new SqlConnection(
                    EncryptionHelper.Decrypt(connectionString));
                try
                {
                    connection.Open();
                    SqlCommand command = new SqlCommand(
                        EncryptionHelper.Decrypt (selectCommand),
                        connection);
                    image = (byte[])command.ExecuteScalar();
                    if (enableCaching)
                        context.Cache.Insert(cacheKey, image, null,
                            DateTime.Now.AddSeconds(cacheDuration),
                            Cache.NoSlidingExpiration);
                }
                finally
                {
                    connection.Close();
                }
            }

            // Stream the image bits back to the browser
            context.Response.ContentType = mimeType;
            context.Response.OutputStream.Write(image, 0, image.Length);
        }

        public bool IsReusable { get { return true; } }
    }

    class EncryptionHelper { ... }
}

Using the control is simplicity itself:

<%@ Register TagPrefix="msdn" Namespace="MsdnMag" Assembly="__code" %>
  ...
<msdn:SqlImage ID="SqlImage1" Runat="server"
  ConnectionString="<%$ ConnectionStrings:Contoso %>"
  SelectCommand= "SELECT Photo FROM Employees WHERE EmployeeID=1313"
  MimeType="image/gif" EnableCaching="true" CacheDuration="60" />
Simply declare a control instance and initialize it with a SelectCommand that extracts an image from a SQL Server database. Be sure to provide a connection string, too. You can declare it explicitly or use a "$ ConnectionStrings" expression like the one in the example. If you'd like to cache the image to avoid redundant database accesses, initialize the EnableCaching and CacheDuration properties as well.

If you're curious about the reference to "__code" in the "@ Register" directive, that's the ASP.NET 2.0 way of referring to an autocompiled assembly. I took this example from a site in which I deployed SqlImage.cs in the Code directory, where it was autocompiled by ASP.NET. If you build the assembly yourself and deploy it in the bin directory, or in the Global Assembly Cache (GAC), substitute your assembly name for "__code".

Before you use the SqlImage control, you'll need to make one change to Web.config. Add an element to <httpHandlers> mapping GET requests for __ImageGrabber.axd to MsdnMag.ImageGrabber, as shown here:

<configuration>
  <system.web>
    <httpHandlers>
      <add verb="GET" path="__ImageGrabber.axd"
        type="MsdnMag.ImageGrabber" />
    </httpHandlers>
  </system.web>
</configuration>
The reason for the change is simple. SqlImage outputs <img> tags with src attributes that refer to __ImageGrabber.axd. This Web.config entry configures ASP.NET to activate the MsdnMag.ImageGrabber class whenever it receives a request for __ImageGrabber.axd. MsdnMag.ImageGrabber is implemented in the same source code file as the SqlImage control, and it's the class that fetches the image from the database and streams it back to the browser. The bulk of the work done in displaying a database image is performed by MsdnMag.ImageGrabber, not by SqlImage.

You might have noticed that SqlImage.cs contains a private helper class named EncryptionHelper. Here's why: Beta 1's DynamicImage control generates a GUID for each image it references. Then it emits <img> tags referencing an internal HTTP handler that uses the GUID, which is passed in a query string, to return the image bits. There are several problems with this approach; for one, the framework has to cache that GUID for an indeterminate period of time since it doesn't know when or how often the image will be requested. (That's why DynamicImage tends to work reliably for a while and then starts returning images containing text indicating that the image is no longer available. Argh!)

SqlImage takes a more robust approach to image identification: it passes connection strings and SQL commands (the ConnectionString and SelectCommand values you provide to it) to __ImageGrabber.axd in query strings. Of course, passing these values in plaintext would pose a potentially serious security problem. To prevent the connection string and SQL command from being exposed to users smart enough to do a View Source, SqlImage encrypts these strings and MsdnMag.ImageGrabber decrypts them. EncryptionHelper methods provide the encryption and decryption logic.

EncryptionHelper uses the same symmetric encryption/decryption key that ASP.NET uses—the one specified with <machineKey>'s decryptionKey attribute. Therefore, if you deploy an application that uses SqlImage on a Web farm, you'll need to set decryptionKey to the same value on every machine. That shouldn't require any additional work because ASP.NET applications deployed on Web farms already need identical decryptionKeys, even if they don't employ SqlImage controls.

Building Asynchronous Pages

The fact that ADO.NET 2.0 contains cool new support for asynchronous database operations has been well documented. But it's not as well known that ASP.NET 2.0 features built-in support for asynchronous pages. This information rarely appears in bulleted lists of new and notable features.

By default, page processing in ASP.NET is synchronous. A request comes in, it's assigned to a thread, and that thread does nothing else until the request completes. The problem is that if the thread becomes I/O-bound (that is, if it calls out to a Web service or queries a remote database or does something else to cause it to wait for a response) it's unavailable to ASP.NET for the duration. Because ASP.NET has a limited number of threads at its disposal to process requests, other requests pile up in the application's request queue. If the queue fills to capacity (by default, it's limited to 100 requests), additional requests are rejected with 503 "Server Unavailable" errors. Even in less extreme circumstances, too many I/O-bound threads can significantly reduce throughput.

ASP.NET has supported asynchronous HTTP handlers from the beginning, and even in version 1.x, one could build asynchronous pages (pages that free request threads during I/O-intensive operations) with a bit of ingenuity. But ASP.NET 2.0 makes asynchronous pages much simpler. You begin by attributing the page as asynchronous with the following @ Page directive:

<%@ Page Async="true" ... %>

Then, sometime before the page's OnPreRenderComplete event, you register a pair of methods: a begin method to start an asynchronous operation and an end method to end the operation. You wrap the former method in a BeginEventHandler delegate, and the latter in an EndEventHandler delegate. ASP.NET calls the begin method before the page begins its render cycle. You launch an async operation (for example, an async ADO.NET call or an async call to a Web service) and return an IAsyncResult interface that ASP.NET uses to detect completion of the asynchronous operation. When the operation completes, ASP.NET calls your end method, where you do whatever's necessary to finish up (for example, bind the results of an async database query to a control).

The code in Figure 8 uses this technique to asynchronously fetch the contents of another Web page using System.Net.WebRequest.BeginGetResponse and System.Net.WebRequest.EndGetResponse. When the call completes and ASP.NET calls EndAsyncOperation, the page parses the content and displays the targets of all the HREFs it finds. What differentiates this page from a synchronous page is that the thread assigned to process the request is returned to the ASP.NET thread pool the moment BeginAsyncOperation returns. This is powerful stuff that can dramatically increase the throughput of I/O-intensive pages.

Figure 8 Asynchronous Page (Close Figure 8)
public partial class AsyncPage_aspx {
    WebRequest _request;

    void Page_Load (object sender, EventArgs e) {
        AddOnPreRenderCompleteAsync (
            new BeginEventHandler(BeginAsyncOperation),
            new EndEventHandler (EndAsyncOperation)
        );
    }

    IAsyncResult BeginAsyncOperation (object sender, EventArgs e,
        AsyncCallback cb, object state) {
        _request = WebRequest.Create("http://msdn.microsoft.com");
        return _request.BeginGetResponse (cb, state);
    }

    void EndAsyncOperation (IAsyncResult ar) {
        WebResponse response = _request.EndGetResponse (ar);
        string text = new StreamReader
            (response.GetResponseStream ()).ReadToEnd ();
        response.Close();

        Regex regex = new Regex ("href\\s*=\\s*\"([^\"]*)\"",
            RegexOptions.IgnoreCase);
        MatchCollection matches = regex.Matches(text);

        StringBuilder builder = new StringBuilder(1024);
        foreach (Match match in matches) {
            builder.Append (match.Groups[1]);
            builder.Append("<br/>");
        }

        Label1.Text = builder.ToString ();
    }
}

That's just one way to use the new asynchronous page infrastructure in ASP.NET 2.0. The whole story could easily fill a column or feature-length article. I'll have a lot more to say about asynchronous pages in a future installment of this column.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect, a software consulting and education firm that specializes in Microsoft .NET.


Pure C++: CLR Generics Versus C++ Templates

Pure C++
CLR Generics Versus C++ Templates
Stanley B. Lippman

Contents
Parameter List Redux
Additional Template Functionality Within the Parameter List
Type Argument Constraints
Generic Constraints
Wrapping It Up

Visual Studio® 2005 brings the type parameter model of generic programming to the Microsoft® .NET Framework. C++/CLI supports two type parameter mechanisms—common language runtime (CLR) generics and C++ templates. In my last column, I looked at the characteristics that these two mechanisms have in common (see Pure C++: Generic Programming Under .NET). In this column, I'll look at some of the differences—in particular, differences in the parameter list and type constraint model.

Parameter List Redux

The parameter list works like the signature of a function: it identifies the number and kind of each parameter, and associates a unique identifier with each one so that each parameter can be uniquely referred to within the template definition.

The parameter serves as a placeholder within the definition of the template or generic type. The user creates object instances of the type by providing actual values to bind to the parameters. The instantiation of a parameterized type is not a simple text substitution such as that which a macro expansion mechanism might employ. Rather, it consists of binding the actual user values to the associated formal parameter within the definition.

Under generics, each parameter represents either an Object type or a type derived from Object. As you'll see later, this constrains the kinds of operations you can perform on or through objects declared through a type parameter. You can offset these constraints by providing more explicit (and therefore less restrictive for the generic author) constraints of your own. These explicit constraints refer to a base class or set of interfaces from which the actual type argument must derive.

In addition to supporting type parameters, templates support expression and template parameters. Plus, templates support default parameter values. These are resolved positionally rather than by name. Under both mechanisms, type parameters are introduced with either the class or typename keyword.

Additional Template Functionality Within the Parameter List

In addition to type parameters, templates permit two other kinds of parameters: non-type parameters and template parameters. Let's briefly look at each.

A non-type parameter is constrained to a constant expression. What should spring to mind immediately is a numeric or string literal constant. For example, if you chose to provide a fixed-size stack, you might decide to specify a non-type size parameter as well as the element type parameter, thus partitioning your categories of stack instances by both element type and size. For example, see the fixed-size stack with a non-type parameter in Figure 1.

Figure 1 Fixed-Size Stack with a Non-Type Parameter (Close Figure 1)
// fixed-size stack with non-type parameter
template <class elemType, int size>
public ref class tStack
{
    // no reason to use a vector now...
    array<elemType> ^m_stack;
    int top;

public:
    tStack() : top( 0 )
           { m_stack = gcnew array<elemType>( size ); }

};

In addition, it would be convenient if the template class designer could specify a default value for either or both parameters. For example, it may prove optimal to have a buffer default to 1KB in size. Under the template mechanism, a parameter can be provided with a default value, as you see here:

// template declaration with default value
template <class elemType, int size = 1024>
public ref class FixedSizeStack {};
A user could then override the default size value by providing an explicit second value like so:
// A stack of at most 128 String instances
FixedSizeState<String^, 128> ^tbs = gcnew FixedSizeStack<String^, 128>;
Otherwise, by not providing a second argument, he uses the associated default value, like this:
// A stack of at most 1024 String instances
FixedSizeStack<String^> ^tbs = gcnew FixedSizeStack<String^>;

This use of default parameter values is an essential design aspect of the Standard Template Library (STL). For example, the following declarations are taken from the ISO-C++ standard:

// examples of default type parameter values in ISO-C++ namespace std
{
     template <class T, class Container = deque<T> >
     class queue;

     template <class T, class Allocator = allocator<T> >
     class vector;
     // ...
}
You could also provide a default element type, like this:
// template declaration with default element type as well
template <class elemType=String^, int size=1024>
public ref class tStack {};
But that would be more difficult to justify from a design standpoint since it is unlikely in general to converge on a single default type for a container.

A pointer can also serve as a non-type parameter since the address of an object or function is known at compile time and is therefore a constant expression. For example, you may want to provide a third parameter to the stack class, one that indicates a callback handler when some condition arises. A judicious use of a typedef can do wonders with the otherwise seemingly complex declaration, as shown here:

typedef void (*handler)( ... array<Object^>^ );
template <class elemType, int size, handler cback >
public ref class tStack {};

Of course, you can provide a default value for your handler—in this case, the address of an existing method. For example, this declaration of your buffer provides for a default size and handler:

void defaultHandler( ... array<Object^>^ ){ ... }

template < class elemType,
           int size = 1024,
           handler cback = &defaultHandler >
public ref class tStack {};

Because default values are positional rather than named, there is no way to provide an overriding handler without also providing an explicit size, even if that size were to duplicate the default value. Here are some ways in which your modified stack might be used:

void demonstration()
{
    // default size and handler
    tStack<String^> ^ts1 = nullptr;

    // default handler
    tStack<String^, 128> ^ts2 = gcnew tStack<String^, 128>;

    // override all three parameters
    tStack<String^, 512, &yourHandler> ^ts3;
}

The second additional kind of parameter supported by templates is that of a template template parameter—say that five times fast! That is, the template parameter itself represents a template. For example:

// template template parameter
template <template <class T> class arena, class arenaType>
class Editor {
    arena<arenaType> m_arena;
    // ...
};

The Editor template class lists two template parameters, arena and arenaType. arenaType is a template type parameter; you can pass it an int, String, yourType, myType, and so on. arena is a template template parameter. Any template class with a single template type parameter can be bound to arena. And m_arena is an instance of that template class bound to the arenaType template type parameter. For example:

// a template Buffer class
template <class elemType>
public ref class tBuffer {};

void f()
{
    Editor<tBuffer,String^> ^textEditor;
    Editor<tBuffer,char>    ^blitEditor;
    // ...
}

Type Argument Constraints

If you use your parameterized type simply as a container for the storage and retrieval of elements, as I did in my last column's stack class, then the whole topic of constraints can be ignored. It is only when you need to invoke an operation on a type parameter (such as when you compare two objects to see whether one object of that type is equal to or less than another, or when you invoke the name of a method or nested type through a type parameter) that the issue of constraints becomes a concern. For example, let's reconsider the typename declaration that was in my last column:

template <class T>
ref class Demonstration {
    int method() {
         typename T::A *aObj;
         // ...
    }
};

While this successfully declares aObj, it also constrains the type arguments that can successfully bind to your class template. For example, if you write the following code, the declaration of aObj turns out to be illegal (in this particular case) and the compiler flags it as an error:

int demoMethod()
{
    Demonstration<int> ^demi =
        gcnew Demonstration<int>( 1024 );
    return dm->method();
}
The specific constraint, of course, is that the type argument must contain a nested declaration of a type explicitly named A. It doesn't do you any good if it is called B or C or Z. A more general constraint is that the type argument must represent a class; otherwise, the use of the T:: scope operator is not permitted. My use of an int type argument violates both of these constraints. For example, the Visual C++® compiler generates the following error:
error C2825: 'T': must be a class or namespace when followed by '::'

One criticism of the C++ template mechanism is the absence of any formal syntax to delineate these sorts of type constraints. (Note that in his original design paper on parameterized types, Bjarne Stroustrup discusses having considered providing an explicit constraint syntax, but being dissatisfied with it and choosing not to provide the facility at that time.) That is, the user can only recognize the implicit constraints of a template either by reading the source code or the associated documentation or by compiling her code and reading the ensuing compiler error message.

What can you do if you need to provide a type argument that is unsuited to the template? On the one hand, very little. Any class you write has certain assumptions built into it, and those assumptions represent constraints on its use. It is rather difficult to design a class that is appropriate for every situation; it is even more difficult to design a template class that is appropriate for not just every situation but for every possible type argument as well.

On the other hand, there are a number of template facilities that allow the user some elbow room. For example, a class template member function is not bound to the type argument until there is a use of that function in your code. So, if you can use the template class without using the method that invalidates your type argument, you're okay.

If that is not feasible, a second alternative is to provide a specialized version of that method that is specifically associated with your type argument. In this case, you would provide a specialized instance of Demonstration<int>::method, or, more generally, you could provide a specialized implementation of the entire template class when provided with an integer type argument.

In general, when you say that a parameterized type can support an infinite variety of types, you are primarily speaking of passive uses of parameterization—that is, the storage and retrieval of a type rather than actively manipulating it.

As the designer of a template, you must always be conscious of the implicit constraints your implementation places on the type arguments and work to ensure that they are not superfluous. For example, it is reasonable to demand that a type argument provide support for the equality and less-than operators; however, it is somewhat less reasonable to demand that it provide support for the less-than-or-equal or XOR bitwise operators. (You can loosen the dependence on operators either by factoring these operations into an interface or by requiring an additional parameter that represents a function, delegate, or function object.) For example, Figure 2 shows a likely implementation of a search method by a native C++ programmer making use of the built-in equality operator.

Figure 2 Template-Unfriendly Search Implementation (Close Figure 2)
template <class elemType, int size=1024>
ref class Container
{
    array<elemType> ^m_buf;
    int next;

public:
    bool search( elemType et )
    {
        for each ( elemType e in m_buf )
            if ( et == e )
                 return true;

        return false;
    }

    Container()
    {
        m_buf = gcnew array<elemType>(size);
        next = 0;
    }

    void add( elemType et )
    {
         if ( next >= size )
              throw gcnew Exception;
         m_buf[ next++ ] = et;
    }

    elemType get( int ix )
    {
        if ( ix < next )
             return m_buf[ ix ];
        throw gcnew Exception;
    }

    // ...
};

There is nothing wrong with this implementation of a search function. However, it is somewhat template-unfriendly because the type argument is tightly coupled to the equality operator. A more flexible alternative is to provide a second search method that allows the user to pass in an object to use for comparison operations. You can do this with a function member template. The function member template provides for an additional type parameter. Take a look at Figure 3.

Figure 3 Using a Template (Close Figure 3)
template <class elemType, int size=1024>
ref class Container
{
    // everything else the same ...

    // this is a function member template ...
    // it can reference both of its containing class parameters, and
    // its own parameters ...

    template <class Comparer>
    bool search( elemType et, Comparer comp )
    {
        for each ( elemType e in m_buf )
            if ( comp( et, e ) )
                 return true;

        return false;
    }
    // ...
};

Now the user has a choice of methods by which to search the container: the tightly coupled equality operator search that is more efficient but not appropriate for all types, and the more flexible member template search that requires a comparison type to be passed in.

What sort of type might serve this comparison purpose? The function object is a common C++ design pattern for this purpose. For example, here is a quick and dirty function object to compare two string objects for equality:

class EqualGuy {
public:
    bool operator()( String^ s1, String^ s2 )
    {
         return s1->CompareTo( s2 ) == 0;
    }
};
The code in Figure 4 shows how you might invoke both the member function template version of search together with the more traditional version.

Figure 4 Two Search Functions (Close Figure 4)
int main()
{
    Container<String^> ^sxc = gcnew Container<String^>;
    sxc->add( "Pooh" );
    sxc->add( "Piglet" );
    // ... and so on ...

    // the member template search ...
    if ( sxc->search( "Pooh", EqualGuy() ) )
         Console::WriteLine( "found" );
    else Console::WriteLine( "not found" );

    // the traditional equality operator search ...
    if ( sxc->search( "Pooh" ) )
         Console::WriteLine( "found" );
    else Console::WriteLine( "not found" );
}

Once you get your head wrapped around templates, you discover that there is hardly anything you can't accomplish using them. Or at least it feels that way.

Generic Constraints

Unlike templates, a generic definition supports a formal constraint syntax for describing prerequisite conditions on the type parameters it can legally bind to. Before I drill down into the constraint facility, let's briefly consider the question why generics choose to provide a constraint facility while templates chose not to. The primary answer, I believe, is the different binding times between the two mechanisms.

Templates are bound during compilation, and therefore an invalid type prevents the program from building. The user must either immediately address the problem or retreat back to a non-template programming solution. The integrity of the executing program is not at risk.

Generics, on the other hand, are bound by the runtime, and that is a bit too late to discover that a user-specified type is invalid. And so the Common Language Infrastructure (CLI) needs some static—that is, compile-time—mechanism to insure that the runtime only binds valid types. The constraint list associated with a generic type is a compile-time filter that, when violated, prevents the program from building.

Let's look at an example. Figure 5 shows my container class reimplemented as a generic type. Its search method presumes that the type parameter is derived from IComparable and that it therefore implements an instance of its CompareTo method. (Notice that the container's size is provided by the user during construction rather than as a second, non-type parameter. Generics, you'll recall, do not support non-type parameters.)

Figure 5 Container Implemented as a Generic Type (Close Figure 5)
generic <class elemType>
public ref class Container
{
    array<elemType> ^m_buf;
    int next;
    int size;

public:

    // implicit constraint that elemType
    // implement the IComparable interface

    bool search( elemType et )
    {
        for each ( elemType e in m_buf )
            if ( et->CompareTo( e ))
                 return true;

        return false;
    }

    Container( int sz )
    {
        m_buf = gcnew array<elemType>(size = sz);
        next = 0;
    }

    // add() and get() are the same ...

};

This generic class implementation fails to compile with the following fatal compile diagnostic:

error C2039: 'CompareTo' : is not a member of 'System::Object'
You might mumble to yourself, okay, so what? Nobody is claiming that it is a member of System::Object. However, in this case, you are wrong. By default, a generic type parameter makes the strongest possible constraint: it constrains all its types to be of type Object. That constraint is guaranteed to be true since only CLI types are permitted to bind to a generic type, and, of course, all CLI types are derived more or less directly from Object. So by default as an author of a generic type you have a very safe but limited set of operations that you can apply.

You might think, well, the heck with being flexible, I just want to get the darn thing to compile, and so you substitute an equality operator for the CompareTo method, but that causes the compiler to bark even louder:

error C2676: binary '==' : 'elemType' does not define this operator
or a conversion to a type acceptable to the predefined operator
Again, what is going on is that each type parameter begins its life metaphorically walled in by the four public methods of Object: ToString, GetType, GetHashCode, and Equals. In effect, this work in listing constraints on the individual type parameters represents a progressive loosening of this initial strong constraint. Stated another way, your task as author of the generic type is to expand the permitted operations on a type parameter in a verifiable way using the conventions of the generic constraint list. Let's take a look at how to do that.

The list of constraints are referred to as the constraint clause, and it is introduced with the non-reserved word "where". It is placed between the parameter list and the type declaration. The actual constraints consist of the names of one or more interface types and/or a single class type. These constraints represent base types that the associated type parameter is expected to implement or be derived from. The public set of operations of each type is added to the operations available to be exercised by the type parameter. So, in order to allow your elemType parameter to invoke CompareTo, you must add a constraint clause associating it with the IComparable interface, like so:

generic <class elemType>
  where elemType : IComparable
public ref class Container
{
    // the body of the class is unchanged ...
};

The constraint clause, which expands the set of operations permitted to be invoked by an elemType instance, is the union of the public operations of the implicit Object constraint and the explicit IComparable constraint. The generic definition now compiles and is available for actual use. When you specify an actual type argument, as you see in the following code, it is the compiler that verifies that the actual type argument meets the constraints associated with the type parameter to which it will bind:

int main()
{
    // ok: String and int implements IComparable
    Container<String^> ^sc;
    Container<int> ^ic;

    // error: StringBuilder does not implement IComparable
    Container<StringBuilder^> ^sbc;
}
The compiler flags any violations, such as the definition of sbc. But the actual binding and construction of the generic type is done by the runtime.

A generic type, then, is verified for not violating its constraints both at its definition point (when the compiler processes your implementation) and at each construction point (when the compiler checks the type argument against the associated constraints). A failure at either point results in a compile-time error.

The constraint clause can contain one entry per type parameter. The entries are not required to be specified in the same order as that of the parameter list. Multiple constraints for a parameter are separated by a comma. The constraints must be unique within the list associated with each parameter, but of course can occur in more than a single constraint list. For example:

generic <class T1, class T2, class T3>
  where T1 : IComparable, ICloneable, Image
  where T2 : IComparable, ICloneable, Image
  where T3 : ISerializable, CompositeImage
public ref class Compositor
{
    // ...
};

In this case, you have three constraint clauses specifying both interface types and a single class type (listed at the end of each list). The constraints are additive—meaning that a type argument must meet all of the listed constraints and not merely a subset. As my colleague Jon Wray points out, as you expand the set of operations in your role as author of the generic type, thereby loosening your constraints, you in turn increase the constraints you place on users of the generic type in their choice of a type argument.

The T1, T2, and T3 clauses could have been specified in any order. It is not permitted, however, to specify the constraint list associated with a type parameter across two or more where clauses. For example, the following is flagged as a syntax violation:

generic <class T1, class T2, class T3>
  // error: two entries for same parameter not permitted
  where T1 : IComparable, ICloneable
  where T1 : Image
public ref class Compositor
{
    // ...
};

A class constraint type must be an unsealed reference class. (Neither value classes nor sealed classes are permitted since they do not allow inheritance.) Four specific System namespace classes are prohibited from appearing within a constraint clause: System::Array, System::Delegate, System::Enum, and System::ValueType. Because the CLI only supports single inheritance, a constraint clause only supports the inclusion of one class type. The constraint types must be at least as accessible as the generic type or function. For example, you cannot declare a public generic type and list one or more constraints with internal visibility.

Any of the type parameters can be bound to a constraint type. Here is a simple example:

generic <class T1, class T2>
  where T1 : IComparable<T1>
  where T2 : IComparable<T2>
public ref class Compositor
{
    // ...
};
Constraints are not inherited. For example, if I derive the following class from Compositor, the IComparable constraints on Compositor's T1 and T2 are not applied to the same-named parameters of the BlackWhite_Compositor class:
generic <class T1, class T2>
public ref class BlackWhite_Compositor : Compositor
{
    // ...
};

This is something of a design inconvenience when those parameters are used in conjunction with the base class. In order to guarantee the integrity of Compositor, BlackWhite_Compositor must propagate the Compositor constraints on all parameters that are passed to the Compositor subobject. For example, the correct declaration looks like this:

generic <class T1, class T2>
  where T1 : IComparable<T1>
  where T2 : IComparable<T2>
public ref class BlackWhite_Compositor : Compositor
{
    // ...
};

Wrapping It Up

As you've seen, under C++/CLI, you can choose either CLR generics or C++ templates.Now you should have the information to make an informed choice based on your particular needs. Under both mechanisms, a parameterized type that goes beyond the storage and retrieval of elements contains assumptions as to the operations that each type parameter must support.

With templates, these assumptions are implicit. This benefits the author of a template who has a greater degree of freedom in what she can implement. However, this does not benefit the consumer of a template who is often faced with an undocumented set of constraints on the permissible type arguments. While violations of this constraint set result in compile-time errors and is therefore not a threat to the integrity of the run-time execution, use of a template class can prove frustrating. The design bias of the mechanism leans in favor of the implementor.

With generics, these assumptions are made explicitly and are associated with a set of base types listed within the constraint clause. This benefits the consumer of the generic and verifies that any generic passed to the runtime for type construction is correct. This provides some constraints on the design freedom of the author of a generic, in my opinion, and makes certain template design idioms a little more difficult to support. Violations of the formal constraints, either at the point of definition or at the point of a user-specified type argument, result in compile-time errors. The design bias of the mechanism leans in favor of the consumer.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman is an Architect with the Visual C++ team at Microsoft. He began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. In between, he worked in feature animation at Disney and DreamWorks, was a Distinguished Consultant with JPL, and was a Software Technical Director on Fantasia 2000.


C++ at Work: Counting MDI Children, Browsing for Folders

New information has been added to this article since publication.
Refer to the Editor's Update below.

C++ At Work
Counting MDI Children, Browsing for Folders
Paul DiLascia

Code download available at: CAtWork0506.exe (195KB)

Q I'm writing a Multiple Document Interface (MDI) app using MFC. From the parent window, how can I check whether all the MDI child windows have been closed? If they are all closed, then I want to activate a panel in my main window.

Ramesh

A Windows® and MFC don't provide any functions specifically to get the number of MDI child windows, but implementing what you want is easy enough. In fact, I can think of half a dozen ways to skin this cat. You could trap WM_CREATE/WM_DESTROY messages; you could install a Windows hook with SetWindowsHookEx; you could use EnumWindows to enumerate the child windows and count how many there are. But often the simplest solution is the one most easily overlooked.

All you need to solve this problem—something that works whether you're using MDI or some other multiple-window user interface of your own design—is a lowly list. Figure 1 shows a class based on the Standard Template Library (STL) list that does the job. It's hardly even worth encapsulating, but I did just in case typing "push_back" is too weird for Windows-based programmers; CWinList lets you type "Add" instead. To use CWinList, just add a global instance somewhere, either as a global variable or data member in your main application class:

class CMyApp : public CWinApp {
public:
   CWinList m_winlist; // list of open windows
};

Figure 1 WinList.h (Close Figure 1)
////////////////////////////////////////////////////////////////
// MSDN Magazine — June 2005
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 (V7.1) on Windows XP. Tab size=3.
//
#pragma once
#include <list>
using namespace std;

//////////////////
// Simple list class to keep track of windows by adding them to a list.
// Use this to keep track of MDI child or other windows. See view.cpp for
// example of how to use. Based on STL list, CWinList is a list of CWnd
// pointers:
// list<CWnd*>, with methods Add and Remove. You can also call any other
// STL list functions you want, and use the list iterator to enumerate
// the items in the list.
//
class CWinList : public list<CWnd*> {
public:
   CWinList() { }

   ~CWinList() {
      clear();
   }

   // add window to list
   void Add(CWnd* pWnd) {
      push_back(pWnd);
   }

   // remove window from list
   void Remove(CWnd* pWnd) {
      remove(pWnd);
   }
};

In order for your children to be tracked, all they have to do is add and remove themselves from the list as they're created and destroyed. The obvious place to do it is within the constructor and destructor for whichever window class you want to keep track of:

CMyView::CMyView()
{
   theApp.m_winlist.Add(this);
}
CMyView::~CMyView()
{
   theApp.m_winlist.Remove(this);
}

Alternatively, you can call Add and Remove from your OnCreate and OnDestroy handlers to ensure your list contains only window objects with valid HWNDs. Since CWinList is derived from list<CWnd*>, you have the full power of STL at your disposal. For example, you can use the STL list iterator to enumerate the windows in your list:

CWinList& wl = theApp.m_winlist;
for (CWinList::iterator it=wl.begin(); it!=wl.end(); it++) {
   CWnd* pWnd = *it;
   // do something
}

I wrote a little test program called WinCount that uses CWinList to count MDI child windows. Figure 2 shows it running. WinCount has a status bar panel in the lower-right corner that shows the number of windows open, and the application's About dialog lists the windows' captions. The About dialog uses CWinList::iterator to generate its feedback message; the status bar panel is a standard MFC indicator panel with an ON_UPDATE_COMMAND_UI handler that displays the number of views:

void CMainFrame::OnUpdateWinIndicator(CCmdUI* pCmdUI)
{
   CString s;
   s.Format(_T("Open:%d"), theApp.m_winlist.size());
   pCmdUI->SetText(s);
}

Figure 2 Counting MDI Children with WinCount
Figure 2 Counting MDI Children with WinCount

List::size is an STL list method that returns the number of items in the list. Note that for WinCount, there's a one-to-one correspondence between views and MDI child windows, so counting the number of views is the same as counting the number of MDI children. If you have a more complex user interface with multiple views inside each MDI child frame, you'd have to override CMDIChildWnd to put the Add/Remove calls in your derived class, or you could use some other window class that appears exactly once in each child frame. You can use as many CWinLists as you want to keep track of different kinds of window classes.

I know we live in the heady age of MFC, .NET, and GUI Frameworks that do everything for you—but don't forget how to use basic data structures!

[ Editor's Update - 5/9/2005: In the original implementation of CWinList shown in Figure 1, CWinList is derived from list<CWnd*>. Generally, however, it is considered bad practice to derive from STL containers as doing so can sometimes lead to unpredictable results. As such, the download for this article contains a new version where CWinList is now a typedef. This requires using push_back and remove instead of Add and Remove.]

Q I'm building an app in C++ using Visual Studio® .NET and MFC. In my program, the user has to pick a folder in which to copy some files. I can run the OpenFileDialog to let the user select a file, but how do I get the Open dialog to display only folders? Almost every installation program I've seen displays a dialog that shows only folders, but I can't seem to find the right flags.

Laine Chandler

A The reason you can't find the right flags is you're looking at the wrong function! The File Open dialog (the Win32® GetOpenFileName or MFC's CFileDialog) doesn't do folders. To display the folder-browser dialog, you have to call a special shell function, SHBrowseForFolder. To use it, you stuff a BROWSEINFO struct with a bunch of information, then call SHBrowseForFolder. Windows displays a dialog like the one in Figure 3. Users can navigate the folder hierarchy, expand and collapse items, and select the folder they want.

Figure 3 Calling SHBrowseForFolder
Figure 3 Calling SHBrowseForFolder

Alas, using SHBrowseForFolder isn't quite as easy as CFileDialog. It requires at least a minimal understanding of shellspeak, which makes heavy use of COM, IShellFolder, and PIDLs. In case you don't already know, PIDL (pronounced "piddle") is short for pointer-to-item-ID-list. The actual C type is LPITEMIDLIST, or LPCITEMIDLIST for the const variety. A PIDL is a byte string the shell uses to identify shell objects like files and folders, as well as pseudo-objects like My Computer or My Network Places. For ordinary files and folders, the bytes are the path name in Unicode, but for other objects the bytes are different so don't rely on that. The important thing is that when your user finally chooses a folder, SHBrowseForFolder returns it as a PIDL. To get the path name, you have to convert.

Since SHBrowseForFolder is so useful, and since MFC doesn't have a class to encapsulate it, I decided to write one for you. (I'm so nice, I know.) CFolderDialog hides the coding grungies and makes browsing for folders as easy as eating pie. All you have to do is instantiate and call BrowseForFolder:

CFolderDialog dlg(this);
LPCITEMIDLIST pidl = dlg.BrowseForFolder(
   _T("Pick a folder, dude!"), BIF_RETURNONLYFSDIRS | BIF_STATUSTEXT);
CString path = dlg.GetPathName(pidl);

CFolderDialog even has a handy function to convert the PIDL to a CString path name. SHBrowseForFolder can return the selected folder as a TCHAR string if you give it a buffer in BROWSEINFO::pszDisplayName, but the display name returned is only the final part of the full path—for example "Photos" if the folder is C:\MyStuff\Pub\Photos. If you want the full path name, you have to convert the PIDL using SHGetPathFromIDList or my own GetPathName. If the display name is what you want, call CFolderDialog::GetDisplayName.

If all you want to do is let the user pick a folder, then BrowseForFolder and GetPathName are all you need. But SHBrowseForFolder can do lots more. As with GetOpenFileName, it lets you supply a callback function to customize its behavior. If you provide a BrowseCallbackProc in BROWSEINFO::lpfn, Windows will call it when stuff happens. For example, Windows sends BFFM_INITIALIZED when the folder dialog has initialized itself, and BFFM_SELCHANGED when the user selects a new folder. Your callback procedure can process these notifications and do whatever it wants. For example, you can enable or disable the OK button by sending BFFM_ENABLEOK, or change the button text with BFFM_SETOKTEXT. CFolderDialog replaces the C-style callback with C++ virtual handler functions in typical MFC fashion, so instead of writing a callback proc you derive from CFolderDialog and override virtual functions like OnInitialized and OnSelChanged. Internally, CFolderDialog uses its own callback that calls these methods.

To exercise the more advanced features of CFolderDialog, I wrote a test app called FolderPick. It has two commands that run the folder dialog. One command displays the "old-style" dialog; the other, the new. The new style (BIF_NEWDIALOGSTYLE) creates a larger, sizeable dialog with a Make New Folder button and—if you specify BIF_EDITBOX—an edit box where the user can type the folder name. Other flags include BIF_BROWSEFORCOMPUTER to show computers and BIF_BROWSEFORPRINTER for printers. BIF_RETURNONLYFSDIRS tells Windows to return only file system directories, not pseudo-folders like My Network Places, and BIF_STATUSTEXT creates a status window whose text you can set. (BIF_STATUSTEXT is not supported for new-style dialogs.) For the full list of flags, see the documentation for BROWSEINFO.

FolderPick derives a new class, CMyFolderDialog, with overrides for OnInitialized and OnValidateFailed. When the dialog is initialized, FolderPick sets the status text and changes the name of the OK button to "Choose Me!"

void CMyFolderDialog::OnInitialized()
{
   SetStatusText(_T("Nice day, isn't it?"));
   SetOKText(L"Choose Me!");
}

There are a couple of things to underscore here. First, CFolderDialog has wrappers like SetStatusText and SetOKText for folder dialog messages like BFFM_SETSTATUSTEXT and BFFM_SETOKTEXT. If you were programming in C, you'd call ::SendMessage; with CFolderDialog you just call the wrappers. The only caveat is that you can only call these wrappers from within your virtual notification handlers (OnInitialized, OnSelChanged, and the rest) because m_hWnd is valid only while the folder dialog is actually running, not before or after calling BrowseForFolder. Internally, CFolderDialog subclasses the folder dialog the first time its callback receives a notification. The second thing to notice is that some BFFM_ messages require Unicode strings, not LPCTSTRs. That's why "Choose Me!" in the snippet is a wide character string (prefixed with L).

The Microsoft documentation has a couple of minor errors I should point out in case you try to program SHBrowseForFolder in C. The documentation says to pass the string for BFFM_SETOKTEXT in WPARAM; actually, it's LPARAM. It also says that BFFM_SETSELECTION requires a Unicode string, but BFFM_SETSELECTION is available in both A and W flavors, so you can use LPCTSTR.

If you use BIF_EDITBOX with the new-style dialog, Windows displays an edit control where the user can type the name of the folder. If the user types something bad, Windows calls the browser proc with BFFM_VALIDATEFAILED. CFolderDialog processes this by calling OnValidateFailed. FolderPick overrides OnValidateFailed to display an error message like the one in Figure 4.

BOOL CMyFolderDialog::OnValidateFailed(LPCTSTR lpsz)
{
   MessageBox(...);
   return TRUE; // don't dismiss dialog
}

Figure 4 Error Message from FolderPick
Figure 4 Error Message from FolderPick

Another cool feature that SHBrowseForFolder supports is custom filtering. This lets you control which items appear in the folder dialog on a per-item basis. As with callbacks, the mechanics are a bit tedious. You have to implement a COM interface, IFolderFilter, with two methods: GetEnumFlags and ShouldShow. When the folder dialog sends your callback BFFM_IUNKNOWN, you have to QueryInterface the IUnknown it passes for IFolderFilterSite, then call IFolderFilterSite::SetFilter with your IFolderFilter. Now the folder dialog calls your IFolderFilter::ShouldShow to filter each item. You can return S_OK to show the item or S_FALSE to hide it. Once you've installed your filter, IFolderFilterSite is no longer needed so you can Release it.

Naturally, I encapsulated all of this, too. To use custom filtering, all you have to do is call BrowseForFolder with bFilter=TRUE, and override two virtual functions: OnGetEnumFlags and OnShouldShow. No need to deal with COM, QueryInterface, or IFolderFilter. Figure 5 and Figure 6 show the code that performs this magic. CFolderDialog implements its own IFolderFilter internally, one that calls the corresponding virtual CFolderDialog functions. CFolderDialog::OnIUnknown uses the Active Template Library (ATL) CComQIPtr smart pointer class to make COM coding a breeze.

Figure 6 TraceWin (Close Figure 6)

Figure 5 FolderDlg (Close Figure 5)

FolderDlg.h
////////////////////////////////////////////////////////////////
// MSDN Magazine — June 2005
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 (V7.1) on Windows XP. Tab
// size=3.
//
#pragma once
#include "debug.h" // debugging tools

//////////////////
// Class to encapsulate SHBrowseForFolder. To use, instantiate in your
// app and call BrowseForFolder, which returns a PIDL. You can call
// GetPathName to get the path name from the PIDL.
//
class CFolderDialog : public CWnd {
public:
   CFolderDialog(CWnd* pWnd);
   ~CFolderDialog();

   LPCITEMIDLIST BrowseForFolder(LPCTSTR title, UINT flags,
      LPCITEMIDLIST pidRoot=NULL, BOOL bFilter=FALSE);
   CString GetDisplayName() { return m_sDisplayName; }

    // helpers
   static CString GetPathName(LPCITEMIDLIST pidl);
   static CString GetDisplayNameOf(
      IShellFolder* psf, LPCITEMIDLIST pidl, DWORD uFlags);
   static void FreePIDL(LPCITEMIDLIST pidl);

protected:
   BROWSEINFO m_brinfo;                 // used with SHBrowseForFolder
   CString m_sDisplayName;              // display name of folder chosen
   BOOL m_bFilter;                      // do custom filtering?
   CComQIPtr<IShellFolder> m_shfRoot;   // handy to have root folder

   static CALLBACK CallbackProc(
       HWND hwnd, UINT msg, LPARAM lp, LPARAM lpData);

   virtual int OnMessage(UINT msg, LPARAM lp);  // internal catch-all

   // Virtual message handlers: override these instead of using callback
   virtual void OnInitialized()                    { }
   virtual void OnIUnknown(IUnknown* punk);
   virtual void OnSelChanged(LPCITEMIDLIST pidl)   { }
   virtual BOOL OnValidateFailed(LPCTSTR lpsz)     { return TRUE; }

   // wrappers
   void EnableOK(BOOL bEnable) {
      SendMessage(BFFM_ENABLEOK,0,bEnable);
   }
   void SetOKText(LPCWSTR lpText) {
       SendMessage(BFFM_SETOKTEXT,0,(LPARAM)lpText);
   }
   ... // etc (more wrappers—download for details)

   // Override for custom filtering. You must call BrowseForFolder
   // with bFilter=TRUE.
   virtual HRESULT OnGetEnumFlags(...) { return S_OK; }
   virtual HRESULT OnShouldShow(...)   { return S_OK; }

   // IFolderFilter: used to do custom filtering. Download for details.
   DECLARE_INTERFACE_MAP()
   BEGIN_INTERFACE_PART(FolderFilter, IFolderFilter)
    STDMETHOD(GetEnumFlags)(IShellFolder* psf, ...);
    STDMETHOD(ShouldShow)(IShellFolder* psf, ...);
   END_INTERFACE_PART(FolderFilter)
};
FolderDlg.cpp
#include "stdafx.h"
#include "FolderDlg.h"
#include <shlwapi.h>

CFolderDialog::CFolderDialog(CWnd* pWnd)
{
   memset(&m_brinfo,0,sizeof(m_brinfo));
   m_brinfo.hwndOwner=pWnd->m_hWnd;     // use parent window
   m_bFilter = FALSE;                   // default: no filtering
   SHGetDesktopFolder(&m_shfRoot);      // get root IShellFolder
}

// Browse for folder. Args are same as for SHBrowseForFolder,
// but with extra bFilter that tells whether to do custom filtering.
LPCITEMIDLIST CFolderDialog::BrowseForFolder(LPCTSTR title, UINT flags,
   LPCITEMIDLIST root, BOOL bFilter)
{
   TCHAR* buf = m_sDisplayName.GetBuffer(MAX_PATH);
   m_brinfo.pidlRoot = root;
   m_brinfo.pszDisplayName = buf;
   m_brinfo.lpszTitle = title;
   m_brinfo.ulFlags = flags;
   m_brinfo.lpfn = CallbackProc;
   m_brinfo.lParam = (LPARAM)this;

   // filtering only supported for new-style dialogs
   m_bFilter = bFilter;
   ASSERT(!bFilter||(m_brinfo.ulFlags & BIF_NEWDIALOGSTYLE));
   LPCITEMIDLIST pidl = SHBrowseForFolder(&m_brinfo); // do it
   m_sDisplayName.ReleaseBuffer();
   return pidl;
}

// Handy function to get the string pathname from pidl.
CString CFolderDialog::GetPathName(LPCITEMIDLIST pidl)
{
   CString path;
   TCHAR* buf = path.GetBuffer(MAX_PATH);
   SHGetPathFromIDList(pidl, buf);
   path.ReleaseBuffer();
   return path;
}

// Handy function to get the display name from pidl.
CString CFolderDialog::GetDisplayNameOf(IShellFolder* psf,
   LPCITEMIDLIST pidl, DWORD uFlags)
{
   CString dn;
   STRRET strret;                // special struct for GetDisplayNameOf
   strret.uType = STRRET_CSTR;   // get as CSTR
   if (SUCCEEDED(psf->GetDisplayNameOf(pidl, uFlags, &strret))) {
      StrRetToBuf(&strret, pidl, dn.GetBuffer(MAX_PATH), MAX_PATH);
      dn.ReleaseBuffer();
   }
   return dn;
}

//////////////////
// Free PIDL using shell's IMalloc
//
void CFolderDialog::FreePIDL(LPCITEMIDLIST pidl)
{
   CComQIPtr<IMalloc> iMalloc;
   HRESULT hr = SHGetMalloc(&iMalloc);
   ASSERT(SUCCEEDED(hr));
   iMalloc->Free((void*)pidl);
}

// Internal callback proc used for SHBrowseForFolder passes control to
// appropriate virtual function after attaching browser window.
int CFolderDialog::CallbackProc(
    HWND hwnd, UINT msg, LPARAM lp, LPARAM lpData) {
   CFolderDialog* pDlg = (CFolderDialog*)lpData;
   ASSERT(pDlg);
   if (pDlg->m_hWnd!=hwnd) {
      if (pDlg->m_hWnd)
         pDlg->UnsubclassWindow();
      pDlg->SubclassWindow(hwnd);
   }
   return pDlg->OnMessage(msg, lp);
}

// Handle notification from browser window: pass to specific handler function.
int CFolderDialog::OnMessage(UINT msg, LPARAM lp) {
   switch (msg) {
   case BFFM_INITIALIZED: OnInitialized(); return 0;
   case BFFM_IUNKNOWN:    OnIUnknown((IUnknown*)lp); return 0;
   case BFFM_SELCHANGED:  OnSelChanged((LPCITEMIDLIST)lp); return 0;
   case BFFM_VALIDATEFAILED: return OnValidateFailed((LPCTSTR)lp);
   default: TRACE(
      _T("***Warning: unknown message %d in CFolderDialog::OnMessage\n"));
   }
   return 0;
}

// Browser is notifying me with its IUnknown: use it to set filter if
// requested. Note this can be called with punk=NULL when shutting down!
void CFolderDialog::OnIUnknown(IUnknown* punk) {
   if (punk && m_bFilter) {
      CComQIPtr<IFolderFilterSite> iffs;
      VERIFY(SUCCEEDED(punk->QueryInterface(
         IID_IFolderFilterSite, (void**)&iffs)));
      iffs->SetFilter((IFolderFilter*)&m_xFolderFilter);
   }
}

... // Standard MFC IUnknown not shown here — download source for details

//////////////////////////////// IFolderFilter ////////////////////////////
// Implementation passes control to parent class CFolderDialog (pThis)
//
BEGIN_INTERFACE_MAP(CFolderDialog, CCmdTarget)
   INTERFACE_PART(CFolderDialog, IID_IFolderFilter, FolderFilter)
END_INTERFACE_MAP()

STDMETHODIMP_(ULONG) CFolderDialog::XFolderFilter::AddRef() {
   METHOD_PROLOGUE(CFolderDialog, FolderFilter);
   return pThis->AddRef();
}

STDMETHODIMP_(ULONG) CFolderDialog::XFolderFilter::Release() {
   METHOD_PROLOGUE(CFolderDialog, FolderFilter);
   return pThis->Release();
}

STDMETHODIMP CFolderDialog::XFolderFilter::QueryInterface(
   REFIID iid, LPVOID* ppv) {
   METHOD_PROLOGUE(CFolderDialog, FolderFilter);
   return pThis->QueryInterface(iid, ppv);
}

// Note: pHwnd is always NULL here as far as I can tell.
STDMETHODIMP CFolderDialog::XFolderFilter::GetEnumFlags(IShellFolder* psf,
   LPCITEMIDLIST pidlFolder, HWND *pHwnd, DWORD *pgrfFlags) {
   METHOD_PROLOGUE(CFolderDialog, FolderFilter);
   return pThis->OnGetEnumFlags(psf, pidlFolder, pgrfFlags);
}

STDMETHODIMP CFolderDialog::XFolderFilter::ShouldShow(IShellFolder* psf,
   LPCITEMIDLIST pidlFolder, LPCITEMIDLIST pidlItem) {
   METHOD_PROLOGUE(CFolderDialog, FolderFilter);
   return pThis->OnShouldShow(psf, pidlFolder, pidlItem);
}

If you decide to use custom filtering, be careful because it overrides flags like BIF_RETURNONLYFSDIRS (return only file system dirs). Just for fun, I decided to implement BIF_RETURNONLYFSDIRS myself for FolderPick by manually disabling the OK button if the item selected is not a file system folder. To check for a file system object, you'd think the proper way would be to call IShellFolder::GetAttributesOf and look for SFGAO_FILESYSTEM. But when I tried this, "My Computer" had the SFGAO_FILESYSTEM attribute even though it's not really a file system folder! Go figure. The only reliable way I could discover to tell if a shell object is really a file or folder is to call GetPathName and check for an empty string. This is what I did in CMyFolderDialog to disable the OK button for non-folders. Download the source from the MSDN®Magazine Web site for details.

Finally, in order to help you understand what happens and when, I sprinkled CFolderDialog liberally with TRACE diagnostics. Figure 6 shows a sample run. You can turn the diagnostics on or off by setting a global variable CFolderDialog::bTRACE. Of course, the diagnostics appear only in debug builds. If you download the code, you also get a free copy of TraceWin to view the diagnostics without running under the debugger.

SHBrowseForFolder has lots of flags and features I haven't covered, but whichever features you decide to use, CFolderDialog alleviates the grunt work and lets you program SHBrowseForFolder the MFC way. Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: It's Not Just Fonts

{ End Bracket }
It's Not Just Fonts
Bill Hill

Creating a typeface that's truly readable on a screen is much more complex than people realize. The average core font in Windows®, for instance, has about 25,000 lines of executable code as well as the outline and other data. The constraints are brutal. You just can't make the text bigger and use more pixels. Humans need body text that's between 9 and 12 points high (there are 72 points in an inch). Reading involves a high-resolution section of the human retina called the fovea, which is only 0.2mm in diameter. Physiology is the driver.

There are some general features every font must have, even for print. The top and bottom curves of lowercase letters like "o" are generally slightly higher and lower than the height of the "x". These "overshoots" and "undershoots" are necessary; otherwise the "o" would appear smaller, and the type would look rather odd.

You need to incorporate 5,500 years of learning when you design type. How do the letters fit together? Each glyph is designed with general spacing around it. But when you bring two specific letters together in a word, they often don't fit well. The classic example is a capital "T" paired with the lowercase "o", in the word "To". Such pairs of letters need special optical spacing and the pair-kerning data for this is provided by the font designer.

There are some pairs, or even triplets, of letters that won't fit well even with kerning (an adjustment to the space between characters). A kerned lowercase "f" and "i" usually clash because the dot above the "i" collides with the top of the "f". They're normally replaced with a new character for the "fi" combination, called a ligature. There are common ligatures for fl, ffi, ffl, and so on, as well as more arcane ligatures in some fonts (for instance in Palatino Linotype, one of the core fonts in Windows) for combinations like "ct", "st", and "Qu".

Figure 1 "f" and "st" Ligatures in Palatino Linotype
Figure 1 "f" and "st" Ligatures in Palatino Linotype

When it comes to the screen, which is much lower resolution, you can't easily fit complex letter-shapes into its coarse pixel grid. Mapping the outline shape to the grid causes rounding errors; some pixels get turned on when they should be off, and vice-versa, and the text is awful to read. So the rasterizer, the code that turns outline data into pixels, needs a lot of help.

The solution is font hinting, a set of programs that manipulate the outlines so the right pixels get turned on. Hinting is really a hierarchy of programs; at the highest level you have global hints, which keep stem weights uniform across a font, or harmonize across a family, so the Bold weight of the font has a constant relationship to the Regular weight, going all the way down to very specific instructions (known as delta hints). These can be as specific as "in this font, at 10-point on a display of 100ppi, turn on this pixel in the lowercase 'a'..."

Many of the problems associated with displaying readable type on the screen are dramatically reduced by ClearType®, which Bert Keely, Greg Hitchcock, Mike Duggan, and I invented back in 1998. Font hinting has become simpler in many ways, and now needs a lighter touch. ClearType has evolved a lot over the years, and the latest and best version of it will ship as part of "Avalon," the code name for the presentation subsystem class libraries in WinFX. When the next version of Windows, code-named "Longhorn," ships, it will contain seven new fonts, specially designed to take maximum advantage of ClearType. There will be six new fonts for Latin-based languages—including a brand-new monospaced font, Consolas, designed to make your lines of code much easier to read. They've been created by some of the world's top font designers, and hinted to the highest standards possible.

There's a new Japanese font called Meiryo, which will offer the best on-screen Japanese ever. It contains no embedded bitmaps, so it's fully scalable. We used the TrueType hinting language to do the stroke-reduction.

I'm privileged to manage a group of folks at Microsoft who are passionate about creating the best type in the world for the screen. But the passion goes all the way to the top. I've actually had e-mail from BillG, at 10:30 P.M. on a Saturday night, asking me about ligature support! So please, next time you read on screen, remember you're using software technology with 5,500 years of history behind it. And never think of type, ever again, as "just fonts."

Bill Hill joined Microsoft in 1995 because he believed the company would lead the transition from reading on paper to reading on the screen. He is one of the inventors of the ClearType technology which dramatically improved the screen readability of text.