 MSDN Magazine September 2005

MSDN Magazine September 2005

Best Practices: Fast, Scalable, and Secure Session State Management for Your Web Applications

ASP.NET provides a number of ways to maintain user state, the most powerful of which is session state. This article takes an in-depth look at designing and deploying high-performance, scalable, secure session solutions, and presents best practices for both existing and new ASP.NET session state features straight from the ASP.NET feature team.

Mike Volodarsky ASP.NET 2.0: Personalize Your Portal with User Controls and Custom Web Parts

ASP.NET 2.0 introduces a Web Part control that is designed to deal with the serialization, storage, and retrieval of customization and personalization data behind the scenes. In this article, the authors explain how you can put the WebPart control to work in your ASP.NET 2.0 applications.

Ted Pattison and Fritz Onion Call MOM: Instrument and Monitor Your ASP.NET Apps Using WMI and MOM 2005

The current version of Windows Management Instrumentation (WMI) satisfies many current and future manageability requirements. In this article Michael Jurek demonstrates how WMI provides important system management capabilities and develops a WMI-aware monitoring solution you can use to instrument your ASP.NET applications. He then introduces the capabilities of MOM 2005 that allow you to monitor these instrumented applications.

Michael Jurek SQL Server 2005: Fuzzy Lookups and Groupings Provide Powerful Data Cleansing Capabilities

SQL Server 2005 offers a completely redesigned SQL Server Integration Services engine, formerly known as Data Transformation Services. It includes many new features such as built-in support for Fuzzy Lookups and Fuzzy Groupings, which help you build powerful data-cleansing solutions. This article provides an overview of fuzzy searching techniques and a dissection of the underlying fuzzy search technology implemented in SQL Server 2005.

Jay Nathan Spice It Up: Sprinkle Some Pizzazz on Your Plain Vanilla Windows Forms Apps

With the .NET Framework and GDI+, you can easily add elements of style to your applications. You can use transparency, irregularly shaped windows, notification icons, toast pop-ups, different color schemes, and lots more. Designed well, these techniques can create much more compelling interactions between your application and your users. This article explains how.

Bill Wagner Editor's Note: Becoming a Guru

Here in the pages ofMSDN Magazine, we cover a lot of topics that developers can use every day to further their careers. This month, however, we'll look at a more basic question: how does an average developer win fame and recognition as a guru on the programming scene?.

New Stuff: Resources for Your Developer Toolbox

From my first look at VG. net V2. 4b by Prodige Software Corporation, it seemed like just another typical graphics add-on, but as I discovered what a potent tool it is for Visual Studio® . NET graphical interface development, I became more and more impressed.

Marnie Hutcheson Web Q&A: Smart Navigation, ASP.NET Project Structure, and More

Edited by Nancy Michell Cutting Edge: ASP.NET Forms

Forms are an essential piece of ASP. NET—the ASP. NET Web programming model itself wouldn't be possible without forms. The use of forms is not constrained in pure HTML, but it is subject to some restrictions in ASP.

Dino Esposito Advanced Basics: Revisiting Operator Overloading

On the conference circuit recently, I was speaking about some of my favorite new features in the Microsoft® . NET Framework 2. 0, using the content of three recent columns in this series as fodder. In my talk I sped through generics, operator overloading, and the BackgroundWorker component, all in the space of an hour (for the columns, see Advanced Basics: Being Generic Ain't So Bad, Advanced Basics: Calling All Operators, and Advanced Basics: Doing Async the Easy Way).

Ken Getz Test Run: Low-Level UI Test Automation

There are several ways to test a Windows®-based application through its user interface. For example, in the January 2005 issue of MSDN®Magazine (Test Run: Lightweight UI Test Automation with . NET) I described a lightweight technique for testing .

James McCaffrey Design Patterns: Dependency Injection

Today there is a greater focus than ever on reusing existing components and wiring together disparate components to form a cohesive architecture. But this wiring can quickly become a daunting task because as application size and complexity increase, so do dependencies.

Griffin Caprio Security Briefs: Credentials and Delegation

I get loads of security questions from friends and former students, and recently I've gotten a number of questions about building secure data-driven Web sites for internal enterprise systems. I've decided to answer them here to hopefully save you some headaches in your own projects.

Keith Brown .NET Matters: Stream Decorator, Single-Instance Apps

Stephen Toub C++ at Work: Copy Constructors, Assignment Operators, and More

Paul DiLascia { End Bracket }: Hacking the Immune System

HIV is a major global health problem, with over 40 million people infected worldwide. Biomedical researchers are working to create a vaccine that would be effective against many different strains, taking into account all the individual differences in people's immune systems.

Nebojsa Jojic and David Heckerman


Best Practices: Fast, Scalable, and Secure Session State Management for Your Web Applications

Best Practices

Fast, Scalable, and Secure Session State Management for Your Web Applications

Michael Volodarsky
Parts of this article are based on a prerelease version of ASP.NET 2.0. Those sections are subject to change.
This article discusses:
  • ASP.NET 2.0 session state architecture
  • Improving state management performance and scalability
  • Addressing session state issues in Web farms
  • Securing your state management infrastructure
This article uses the following technologies:
ASP.NET
Contents
How Session State Works
Improving Performance
Disabling Session State
Reducing Network/Storage Overhead
Optimizing Serialization
In-Process Optimizations
Improving Scalability
ASP.NET 2.0 Session State Partitioning
Securing Session State
Configuring for Security
Conclusion

D
ue to the stateless nature of the HTTP protocol, Web applications have always shouldered the burden of user state management. Fortunately, ASP.NET provides a number of ways to maintain user state, the most powerful of which is session state. This feature provides a convenient programmatic interface for associating arbitrary application state with a user session, and takes care of back-end state storage and client session management for the application. This article takes an in-depth look at designing and deploying high-performance, scalable, secure session solutions, and presents best practices for both existing and new ASP.NET session state features straight from the ASP.NET feature team.

How Session State Works

ASP.NET session state lets you associate a server-side string or object dictionary containing state data with a particular HTTP client session. A session is defined as a series of requests issued by the same client within a certain period of time, and is managed by associating a session ID with each unique client. The ID is supplied by the client on each request, either in a cookie or as a special fragment of the request URL. The session data is stored on the server side in one of the supported session state stores, which include in-process memory, SQL Server™ database, and the ASP.NET State Server service. The latter two modes enable session state to be shared among multiple Web servers on a Web farm and do not require server affinity (that is, they don't require the session to be tied to one specific Web server).

Session state runtime operation is implemented by the SessionStateModule class which plugs into the request processing pipeline in the application as an IHttpModule. SessionStateModule executes once before handler execution in the AcquireRequestState pipeline stage, and once after handler execution in the ReleaseRequestState pipeline stage (see Figure 1). In the AcquireRequestState stage, SessionStateModule attempts to extract the session ID from the request and retrieve the session data for that session ID from the session state store provider. If the session ID is present and the state is retrieved successfully, the module builds the session state dictionary that can be used by the handler to inspect and change the session state.

Figure 1 Maintaining Session State
Figure 1 Maintaining Session State

In ASP.NET 2.0, session extraction is encapsulated in the session ID manager component, which can be replaced by a custom implementation in order to support custom session ID management schemes (for example, storing the session ID in a query string or in form fields). The default session ID manager supports both cookie-based and URL-based session IDs. The default ASP.NET 2.0 session ID manager also provides support for automatically detecting which session ID mode should be used per request based on device profile or runtime negotiation with the client.

If no session ID is present in the request, and the handler makes some changes to the session, a new session ID will be created in the ReleaseRequestState stage, and the new ID will be issued to the client by the session ID manager implementation. In this case, and also if changes were made to an existing session, SessionStateModule will use the state store provider to persist the changes made to session data for that session.

In ASP.NET 2.0, the session state store functionality is encapsulated in the session store provider component, which can be one of the built-in InProc, SQLServer, or StateServer providers, or a custom provider that implements the fetching of session data, the creation of sessions, and the saving of changes to existing sessions.

Improving Performance

Using session state in an ASP.NET application can add noticeable overhead to the application performance. This should be expected as you are executing more code during the processing of every request in the application and possibly making network requests to retrieve stored state. However, there are several techniques your application can take advantage of to reduce the performance impact of session state use while still maintaining the desired state management functionality.

The bulk of the session state processing that occurs on every request resides in SessionStateModule. This module executes once before the ASP.NET request handler to fetch the state for the session identified by the request information, and once after the request handler to create a new session or to save the changes made to the existing session during handler execution.

The InProc session state mode is the fastest of the built-in state storage modes. Its overhead is limited to extracting the session ID from the request, performing a cache lookup for the state dictionary stored in the application's memory space, and marking the session as accessed to prevent its expiration. Updates to the data contained in the session are made directly to objects in memory and do not require any additional work to be persisted for the next request. However, because the InProc mode lacks the application restart tolerance and does not work in Web farm scenarios, the application frequently has no choice but to use one of the other two out-of-process modes.

In the default out-of-process modes, SQLServer and StateServer, session state must be fetched from an external store and deserialized from its binary blob representation to the in-memory state dictionary form in the AcquireRequestState stage. In the ReleaseRequestState stage, the state dictionary needs to be serialized again and transferred to external storage. In addition, the session entry in the store must be updated to indicate the last access time to prevent its expiration. In these modes, serialization and deserialization of the state data, and its out-of-process transfer, are by far the most expensive operations introduced by session state to the request path.

SessionStateModule will perform a number of optimizations by default to avoid overhead wherever possible. These optimizations fall into four categories.

First, for handlers or pages that are not marked as requiring session state, no session state work will be performed except for marking the session as accessed in the store. For pages marked as requiring read-only session state access, only the last-accessed marking and initial data fetching will be done.

Second, until any data is actually saved into the session dictionary, no session will be started for requests that do not already specify a session ID.

Third, on requests where no session state accesses are made or where only read accesses are made to session variables that contain immutable primitive types, no state changes will be persisted to the provider at the end of the request. On requests where mutable session data was accessed or where the session was modified, only the accessed variables will be serialized and the rest will just be copied from their binary blob representations.

And fourth, primitive types are serialized directly, but object types are serialized using the relatively slower BinaryFormatter serialization method.

By taking advantage of these optimizations using the best practices illustrated in this article, you can reduce the performance impact of session state management. The strategies discussed here focus on the three most promising approaches to improving application performance when using session state, taking advantage of the session state performance optimizations:

  • Disabling session state when possible in order to avoid the overhead entirely.
  • Reducing the overhead of the serialization and deserialization of state data.
  • Reducing the overhead of out-of-process transfer of session state to and from the state store.

Disabling Session State

Not all pages in your application will need access to session state. For those pages that do not, you can indicate that session state is not needed and prevent session data from being fetched from the store in requests to these pages.

For pages that do not update session state, you can indicate that read-only access is required. This does not prevent the state data from being fetched, but it results in a read lock being taken on the database, which enables multiple read-only requests to access the session simultaneously and prevents lock contention when multiple requests are made to the server with the same session ID. More importantly, it always prevents the session state module from connecting to the store at the end of the request to update the state (there are other ways to avoid this also, which I'll describe later).

Here's how you disable session state:

<%@ Page EnableSessionState="False" %>
Similarly, this is how you indicate read-only session state:
<%@ Page EnableSessionState="ReadOnly" %>
In fact, most typical pages such as shopping carts will only update the state if the user performs a postback action, like adding or removing an item from the cart, but not when she simply views the shopping cart. You can tweak application performance in this case by separating the viewing of the cart into its own page with read-only session state that does not perform an update. Updating the shopping cart can be separated into another page to which the viewing page posts when an action is requested (cross posting is supported in ASP.NET 2.0).

You can also make the default behavior of the application be read-only or you can turn off session state by default by using the <pages> configuration element in your Web.config file. Then you can explicitly enable session state or enable write access in only the pages that need it by setting the page EnableSessionState attribute appropriately, as shown here:

<!--No session state by default-->
<configuration>
    <system.web>
        <pages enableSessionState="false" />
    </system.web>
</configuration>

<!--Read-only session state by default-->
<configuration>
    <system.web>
        <pages enableSessionState="ReadOnly" />
    </system.web>
</configuration>

Similarly, when you are building custom handlers for processing requests in your application, you can disable session state by default by not implementing the marker interface System.Web.State.IRequiresSessionState in your handler class. You can enable read-only mode by implementing the System.Web.State.IRequiresReadOnlySessionState marker interface instead of System.Web.State.IRequiresSessionState.

Note that if you turn off session state in your application, the HttpContext.Session property will throw an exception if the page or handler tries to access it. If you mark the page as read-only, updates to the session in the out-of-process modes will not be preserved across requests. In InProc mode, however, updates will be preserved because they are made to the live objects that stay memory-resident across requests.

Unfortunately, even when you turn off session state for a particular page or handler, SessionStateModule will still mark the session as accessed in the store, so an out-of-process connection to the store will still be made when not using InProc mode. This is normally desired to prevent session expiration as long as the user is actively making requests to the application, regardless of whether the requests are to resources that require session state. But you may want to disable this behavior for some requests for performance reasons, especially if you are using ASP.NET to serve images and page resources that are contained within a parent ASP.NET page that already has marked the session as active.

To optimize this behavior, you can take advantage of the ASP.NET 2.0 custom session ID generation features to hide the session ID for a request, thereby preventing any session state work for that request. You can do this by implementing a custom type that derives from System.Web.SessionState.SessionIDManager, and then implementing the GetSessionID method to return a null session ID for requests that do not require session state (see Figure 2). For all other requests, you can delegate to the default GetSessionID implementation of the SessionIDManager class, which provides the default cookie and cookieless session ID support in ASP.NET.

Figure 2 Custom Session ID Manager (Close Figure 2)
public class SessionDisablementIDManager : SessionIDManager
{
    // override the ID management behavior of the session ID manager
    public override String GetSessionID(HttpContext context)
    {
        // do not return the session ID if no session state is
        // desired for this request
        if (ShouldSkipSessionState(context))
            return null;

        // otherwise just delegate to the built-in session
        // ID manager to get the session ID from the request
        else
            return base.GetSessionID(context);
    }

    protected virtual bool ShouldSkipSessionState(HttpContext context)
    {
        // determine if session state should be skipped for this request
    }
}

There are some interesting ways to implement ShouldSkipSessionState that avoid tracking session state. For example, you could skip session state when the request is for one of the extensions specified in your custom configuration section. You could also ignore session state when the current request handler instance (available as HttpContext.Current.CurrentHandler) implements your own ISkipSessionState interface. You can make pages implement this interface by specifying your own page base class.

Another scenario is when the current request handler instance does not implement IRequiresSessionState and there was another recent request with this session ID on which the session was marked as accessed. (You can keep track of this by writing your own custom cookie, or by storing session ID/timestamp pairs in the ASP.NET cache.) For example, if the last request that marked the session as active was less then 30 seconds ago, do not mark it as active now. This achieves the objective and also does not seriously reduce the session activity window.

You can deploy the custom session ID manager code in the App_Code application directory, or you can compile it into an assembly and deploy it to the \Bin application directory or install it into the Global Assembly Cache (GAC). Then, to enable it, register it with session state configuration as follows:

<configuration>
    <system.web>
        <sessionState
             sessionIDManagerType="IndustryStrengthSessionState.
                 SessionDisablementIDManager"/>
    </system.web>
</configuration>

Note that if you employ this technique when using cookieless session IDs, some ASP.NET functionality that generates links or performs redirects that preserve the cookieless session ID will not work for the requests where you return a null ID.

Reducing Network/Storage Overhead

To reduce the network round-trip overhead, and also lessen the serialization and deserialization impact, you should avoid storing large amounts of data in your session. Design your application to store only a small set of information that doesn't require manipulation in the session, and build the rest of your session object model around that small set of information on every request. Unless recomposing this information is very expensive, you can achieve significantly smaller and simpler sessions that serialize, deserialize, and transfer faster.

To reduce the lock contention that occurs when multiple requests for the same session are made to the server, avoid using frames on your site, and avoid embedding downloadable resources such as images or stylesheets that are served by session-consuming handlers in your ASP.NET application. In both of these cases, the browser will make multiple simultaneous requests to the server with the same session ID, resulting in the session being accessed multiple times and time being wasted waiting for the session lock to be released by other requests. To avoid lock contention, you can also use the read-only session state technique described earlier to prompt the session state module to utilize the reader lock, which allows multiple readers at the same time.

Optimizing Serialization

Session state uses a custom serialization mechanism to convert the session dictionary and its contents to a binary blob before storing the data in an out-of-process store. The serialization mechanism has direct support for .NET Framework primitive types, including String, Boolean, DateTime, TimeSpan, Int16, Int32, Int64, Byte, Char, Single, Double, Decimal, SByte, UInt16, UInt32, UInt64, Guid, and IntPtr. These types are written directly to the blob, while object types are serialized with BinaryFormatter, which is slower. Deserialization follows the same rules. By optimizing the session contents, you can significantly reduce the overhead of serialization and deserialization of the state data.

When designing your session object model, avoid storing object types in the session. Instead, only store primitive types in the session dictionary and rebuild your business layer session objects on every request based on the session data. This avoids the overhead of using BinaryFormatter.

If you are storing a lot of different data items in the session, it's better to flatten your data into many separate session entries instead of grouping them into a single item by placing them in a class or single buffer. This enables more granular access to the needed items, and enables the application to take advantage of on-demand deserialization of needed data for session accesses. It also minimizes the number of data items that need to be serialized again if some are changed during request execution.

If you are using classes instead of primitive types, you can take control over the serialization performed by BinaryFormatter by implementing the System.Runtime.Serialization.ISerializable interface. This allows you to optimize the serialization process by only writing out required data when the object is serialized, and reconstructing the rest of the data when the object is deserialized. Alternatively, if you are simply marking your class with SerializableAttribute, you can use the NonSerializedAttribute to mark the object data members that you do not want to be serialized by the default serialization mechanism. (For more information about .NET Framework binary serialization, see Binary Serialization).

In your code, take advantage of the ASP.NET 2.0 partial deserialization mechanism by accessing only the session items that are needed. The rest of the items in the session will not incur the deserialization overhead, and will simply be copied into the outgoing session blob during the state store update instead of being serialized again. You can do even better by making sure that all of your read accesses on write-enabled session state pages are made to primitive immutable types, which, combined with lack of write accesses of the session state collection, will entirely avoid the round-trip to the server to update session data.

Now, let's see how you can apply these techniques in the real world. Figure 3 shows the initial business logic layer implementation of a session-based shopping cart (simple methods and property assessors have been abbreviated to save space).

Figure 3 A Session-Based Shopping Cart (Close Figure 3)
[Serializable]
public class ShoppingCartItem
{
    private int ID;
    private float price;
    private String name;
    private String description;

    public ShoppingCartItem(int ID, float price, String name,
        String description) {}

    public int ID { get; }
    public float Price { get; }
    public String Name { get; }
    public String Description { get; }

    public static ShoppingCartItem GetItem(int ID)
    {
        // implements database / cache lookup of product information
        // and returns an item instance.
    }
}

[Serializable]
public class ShoppingCartOrder
{
    private ShoppingCartItem item;
    private int quantity;

    public ShoppingCartOrder(ShoppingCartItem item, int quantity) {}

    public ShoppingCartItem Item { get; }
    public int Quantity { get; }
}

[Serializable]
public class ShoppingCart
{
    public const String ShoppingCartKey = "__ShoppingCart";
    private ArrayList orders;

    public ShoppingCart() { orders = new ArrayList(); }

    public ArrayList Orders { get; }

    public static void SaveShoppingCartToSession(ShoppingCart cart)
    {
        if (HttpContext.Current != null)
        {
            HttpContext.Current.Session[ShoppingCartKey] = cart;
        }
    }

    public static ShoppingCart GetShoppingCartFromSession()
    {
        if (HttpContext.Current != null)
        {
            Object shoppingCart =
                HttpContext.Current.Session[ShoppingCartKey];

            if (shoppingCart is ShoppingCart)
            {
                return (ShoppingCart)shoppingCart;
            }
        }

        return null;
    }
}

<%@ Page Language="c#" %>
<%@ Import Namespace="IndustryStrengthSessionState" %>
<script runat="server">
void Page_load(Object source, EventArgs e)
{
  // get the cart from session
  ShoppingCart cart = ShoppingCart.GetShoppingCartFromSession();

  ... // use the cart

  // update the cart
  ShoppingCart.SaveShoppingCartToSession();
}
</script>

This implementation simply defines the shopping cart object model, including a product item, an order containing a product item and quantity, and a shopping cart containing a collection of items. The classes are marked as serializable so that you can store the cart in the session simply by storing a reference to it, and having session state serialize and deserialize the object graph at run time.

Figure 4 is the optimized implementation that applies the techniques discussed earlier—converting the object model into primitive types, not storing any data that can be obtained separately from the session on each request, flattening a collection using primitive types, and optimizing item read/write access to avoid reserialization and updating whenever possible (methods unchanged from previous sample are abbreviated).

Figure 4 Optimized Shopping Cart (Close Figure 4)
[Serializable]
public class ShoppingCart
{
    public const String ShoppingCartKey = "__ShoppingCart";
    public const String ShoppingCartItemCountKey = "__ItemCount";
    public const String ShoppingCartItemIDKeyBase = "__ItemID_";
    public const String ShoppingCartOrderQuantityKeyBase =
        "__OrderQuantity_";
    private ArrayList orders;
    private bool isChanged;

    public ShoppingCart()
    {
        orders = new ArrayList();
    }

    public ShoppingCart(int size)
    {
        orders = new ArrayList(size);
        for (int i = 0; i < size; i++)
        {
            orders.Add(null);
        }
    }

    public void AddOrder(ShoppingCartOrder order)
    {
        orders.Add(order);
        isChanged = true;
    }

    public ShoppingCartOrder GetOrder(int i)
    {
        Object order = orders[i];
        if (order == null)
        {
            orders[i] = GetOrderFromSession(i);
        }
        return (ShoppingCartOrder)orders[i];
    }

    public void SetOrder(int i, ShoppingCartOrder order)
    {
        orders[i] = order;
        isChanged = true;
    }

    public int Count { get { return orders.Count; } }

    // Get the order from the session on demand
    private static ShoppingCartOrder GetOrderFromSession(int i)
    {
        HttpSessionState session = HttpContext.Current.Session;

        int ID = 0;
        int quantity = 0;

        // note: For simplicity session key strings are dynamically
        // created——for performance reasons they should be precreated.
        ID = (int)session[ShoppingCartItemIDKeyBase + i];
        quantity = (int)session[ShoppingCartOrderQuantityKeyBase + i];

        ShoppingCartItem item = ShoppingCartItem.GetItem(ID);
        return new ShoppingCartOrder(item, quantity);
    }

    // Store the order into the session
    private static void SaveOrderToSession(int i, ShoppingCartOrder order)
    {
        HttpSessionState session = HttpContext.Current.Session;

        // note: For simplicity session key strings are dynamically
        // created——for performance reasons they should be precreated.
        session[ShoppingCartItemIDKeyBase + i] = order.Item.ID;
        session[ShoppingCartOrderQuantityKeyBase + i] = order.Quantity;
    }

    public static void SaveShoppingCartToSession(ShoppingCart cart)
    {
        if (!cart.isChanged) return;

        HttpSessionState session = null;

        int oldCount = -1;
        int newCount = cart.orders.Count;

        if (HttpContext.Current != null)
        {
            session = HttpContext.Current.Session;
            object temp = session[ShoppingCartItemCountKey];

            if (temp is int) oldCount = (int)temp;

            for(int i = 0; i < newCount; i++)
            {
                // only set the items that changed or were added
                if (cart.orders[i] != null)
                {
                    SaveOrderToSession(i,
                        (ShoppingCartOrder)cart.orders[i]);
                }
            }

            // if the number of items in the cart changed, need to set
            // it in the session
            if (oldCount != newCount)
            {
                session[ShoppingCartItemCountKey] = newCount;
            }
        }
    }

    public static ShoppingCart GetShoppingCartFromSession()
    {
        HttpSessionState session = null;

        if (HttpContext.Current != null)
        {
            session = HttpContext.Current.Session;
            Object count = null;

            count = session[ShoppingCartItemCountKey];
            if (count is int) return new ShoppingCart((int)count);
        }

        return null;
    }

}

This implementation decouples the item information, which can be generated at run time, from the ID and quantity information, which need to be stored in the session. It also provides a mechanism to flatten the order collection into the session. This, in turn, enables reading and writing shopping cart orders independently of each other, taking advantage of the partial deserialization and serialization mechanisms. Finally, the implementation is careful to avoid the update round-trip when no changes to the cart are made, by avoiding any set calls and using only immutable primitive types for read accesses.

This implementation does not change any part of the object model, only how the cart is stored and retrieved within the session using the SaveShoppingCartToSession and GetShoppingCartFromSession methods. This isolates the required changes from most of the application. This example code does not implement the delete or get collection functionality for clarity reasons, but it is also possible to implement either or both without losing the benefits of optimization.

In-Process Optimizations

Although InProc mode is significantly faster than the out-of-process modes, there are a few guidelines to keep in mind to keep in-process performance optimal.

First, you should avoid storing single-threaded apartment (STA) COM objects in the session. This guidance has been valid since the ASP days. Since STA COM objects can only be accessed by the same thread, ASP.NET will serialize access to the COM object by ensuring that all requests for the session are processed by the same thread. Note that this will only happen when the AspCompat page attribute is set to true. Second, you should attempt to reduce session size altogether to avoid increased memory pressure that results in cache churn and negatively impacts the performance and capacity of the entire application.

Improving Scalability

ASP.NET session state enables horizontal scaling for ASP.NET applications by supporting out-of-process state storage, which allows multiple Web farm machines to process requests for the same session without losing the session data. However, while the capacity of the Web farm can be almost linearly increased by adding more Web server nodes, the single session state store becomes a bottleneck as session capacity increases. For InProc mode, which can be used outside of Web-farmed applications or in Web farms with proper affinity, the memory usage of sessions in Web server memory can also become prohibitive.

ASP.NET 2.0 Session State Partitioning

ASP.NET 2.0 provides a solution to the problems encountered when scaling up by enabling horizontal scale-out of session state stores through its state partitioning feature. State partitioning enables the session data and the associated processing load to be divided between multiple out-of-process state stores, allowing the session state load to scale as the Web farm grows and the number of concurrent sessions increases. It works by supplying a custom partitioning algorithm to SessionStateModule, which uses the algorithm to determine the state store connection string to be used for the current request based on the session ID. Both the SQLServer and the StateServer providers will then use the appropriate connection string to fetch and save the session.

You can implement a partitioning scheme by deriving a class from the System.Web.IPartitionResolver interface, and building the session ID-to-connection string mapping inside the ResolvePartition method. The basic implementation shown in Figure 5 creates an array of hardcoded connection strings corresponding to available state store partitions in the Initialize method. In the ResolvePartition method, the resolver hashes the session ID string into one of the buckets corresponding to one of the loaded connection strings, and selects the resulting connection string.

Figure 5 Session Partitioning (Close Figure 5)
public class PartitionResolver : System.Web.IPartitionResolver
{
    private String[] partitions;

    public void Initialize()
    {
        // create the partition connection string table
        partitions = new String[] {
            "tcpip=192.168.1.1:42424",
            "tcpip=192.168.1.2:42424",
            "tcpip=192.168.1.3:42424" };
    }

    public String ResolvePartition(Object key)
    {
        String sid = (String)key;

        // hash the incoming session ID into
        // one of the available partitions
        int partitionID = Math.Abs(sid.GetHashCode()) % partitions.Length;

        return partitions[partitionID];
    }
}

Ideally, you will want to implement either a configuration collection for specifying the available partitions that you will load in the Initialize method, or obtain the collection from a centralized location over the network on a Web farm. The simple uniform hashing implementation results in a relatively even distribution of sessions to stores over time because the session IDs are generated randomly. However, you may want to implement a load-balancing scheme where you dynamically determine the partition in which to place a given session based on current partition load. To do this, you will need to encode the partition ID into the session ID by using a custom SessionIDManager derivation together with the PartitionResolver to determine the partition for a new session, create the session ID with the partition ID encoded, and then determine the partition ID in future requests by pulling it out from the session ID in the partition resolver.

The partition resolver implementation can be deployed in the App_Code application directory, or it can be compiled into an assembly and deployed in the \Bin application directory or installed into the GAC. Finally, the resolver type has to be added to the session state configuration by specifying its fully qualified name in the partitionResolverType attribute:

<configuration>
    <system.web>
        <sessionState
            mode="StateServer"
            partitionResolverType=
                "IndustryStrengthSessionState.PartitionResolver" />
    </system.web>
</configuration>

Note that the partition resolver can only be used when session state is using the SQLServer or StateServer modes, and no connection string can be specified using the sqlConnectionString or stateConnectionString attributes.

Session state also provides an alternative approach for Web farm session state management, which allows the application to harness the speed of distributed InProc state storage (or out-of-process state storage for reliability purposes) provided that a session ID-encompassing affinity scheme can be used on the Web farm. The affinity scheme needs to ensure that all requests with a given session ID are passed to the same Web server, in which case each Web server can maintain its own session state store without sharing it with other Web servers.

The affinity scheme needs to be based on session IDs or other characteristics of the request that guarantee all requests containing a given session ID will be directed to the same Web server. Such schemes can be based on client IP network ranges (keeping in mind that clients may be coming from dynamically assigned Web proxies) or user agent headers. The problems with implementing such affinity schemes include the fact that they are not readily available on hardware load-balancing systems as they require HTTP-level routing of the requests (as opposed to the more common IP or TCP-level connection routing). In addition, these schemes prevent you from doing any real load balancing because routing needs to be deterministic with respect to session ID to state store mappings to preserve state.

Securing Session State

ASP.NET session state provides an important security advantage over client state management techniques in that the actual state is stored on the server side and not exposed on the client and other network entities along the HTTP request path. However, there are several important aspects of session state operation that need to be considered in order to maintain application security. Security best practices fall into three major categories: preventing session ID spoofing and injection, securing the state storage in the back-end, and ensuring session state deployment security in dedicated or shared environments.

Because session state uses client tickets to identify the server-side session, it may be susceptible to session ID spoofing and injection attacks. A session ID spoofing attack refers to a malicious entity providing the session ID of another user in its own request, thereby tricking the application into loading the session of that user. An injection attack is practically the opposite of a spoofing attack. In an injection attack, a malicious entity forces a legitimate user to make requests to the server with the attacker's session ID. Because the session ID is a random base64-encoded string of 62 characters, brute-force session ID searches are impractical and not considered a real security threat.

ASP.NET 2.0 session state has been hardened to help guard against session ID spoofing and injection. It takes advantage of the HTTP-only cookie feature (currently supported by Internet Explorer 6.0 with Microsoft® Internet Explorer 6.0 SP1 or Windows® XP SP2 installed), which prevents the session ID cookies from being available to client-side script and therefore to potential cross-site scripting exploits designed to steal session ID cookies. This feature is enabled automatically for supported browsers.

In addition, ASP.NET 2.0 also provides a session ID regeneration feature, which forces expired session IDs that are not found on the server to be replaced with a newly generated ID. This avoids the accidental reuse of session IDs that commonly occurs with cookieless session links (such as those indexed by search engine crawlers), as well as malicious session ID injection attacks through cookieless session link posting. This feature is on by default and is only supported for cookieless session IDs. You should keep this feature enabled if you are using cookieless session IDs in your application. Note that this feature does not prevent session spoofing if the attacker can discover the session ID while the original session is still active.

In previous versions, ASP.NET session state required the application to be configured to use either cookie-based session IDs or cookieless link-based session IDs. For applications that needed to support mobile clients or browsers that do not have cookies enabled, the only option was to use the cookieless mode for the entire application. This is discouraged from the security perspective because cookieless IDs lend themselves better to discovery and spoofing, and to injection by link posting or phishing attacks, but is necessary to support non-cookie browsers.

ASP.NET 2.0 enables two new session ID modes, UseDeviceProfile and AutoDetect, which can use the browser device profile or perform an autodetect of the cookie capability respectively to determine which type of session ID to use on each request. If you need to support non-cookie clients, use the UseDeviceProfile mode, which reduces the cookieless surface area by only using cookieless IDs for clients that require them.

An advanced technique that further minimizes the risk of spoofing attacks is to combine the random session ID with a hash of the client's user agent and the network part of the client IP address, and validate the ID to make sure these pieces of information match. Unfortunately this is not a perfect practice, since many clients may come from a single proxy server and may share the same user agent string, but it does provide an additional level of protection against spoofing attacks. You can use the custom SessionIDManager extensibility mechanism in ASP.NET 2.0 to implement this functionality, as described previously. You can find an example of implementing a similar technique in ASP.NET 1.x in Jeff Prosise's column in the August 2004 issue of MSDN®Magazine.

Reducing the session expiration timeout to the lowest usable value is a good security practice and can help avoid most of the session ID exploits. To provide an even higher level of protection, your app should make it easy for its users to log out and abandon the session before leaving the site, and use JavaScript to detect the browser window closing to force a session abandonment on the server.

Finally, secure sockets layer (SSL) should be used to prevent network-level sniffing of session IDs, authentication tickets, application cookies, and other request/response information. When using SSL, keep in mind that if you have any non-HTTPS URLs within the URL scope specified in the cookies (the entire domain by default), the cookies can be sniffed when they are sent by the browser during requests to those URLs. Because of this, it is recommended that you use SSL to protect your entire domain rather than only a subset of the URLs.

Configuring for Security

When using the SQLServer state provider, you typically deploy the session state schema to the SQL Server database by using the aspnet_regsql.exe utility included inside the framework directory for the appropriate version. This utility creates the needed session state tables, but does not grant any users access to the database objects. You need to explicitly grant a particular Windows or SQL Server account access to the database and required objects, depending on whether you want to use integrated Windows authentication or SQL Server authentication to connect to the server.

I recommend creating a SQL Server database role (let's call it "SessionStateFullAccess"), and grant the role permissions only to execute the session state stored procedures and not give any access to the tables themselves. Grant this role to the login account you will use for the Web server's session state configuration. This is a general least-privilege best practice for all ASP.NET data-driven features, many of which already come with these roles configured.

If you are using Windows integrated authentication in your SQL Server connection string, take advantage of the ASP.NET 2.0 configuration option to connect to the server under the hosting identity rather than request identity like in previous versions. This option is on by default and is configured by using the useHostingIdentity attribute in the <sessionState> configuration element. This simplifies the authentication management on an intranet, as you only grant database access to the ASP.NET worker process or application identity instead of granting access to the entire domain or a set of users.

If you are using SQL Server authentication in your connection string, you can place the connection string in the <connectionStrings> configuration section and just place the connection string name in the <sessionState> configuration. You can then use ASP.NET 2.0 configuration encryption to protect the login credentials by encrypting the <connectionStrings> section.

Also, make sure to protect the SQL Server instance from remote access by any machines other than the Web server. You can achieve this by using Windows Firewall or IPSec policies.

The State Server service does not provide any authentication capabilities, so anyone with network access to it can change the session data and cause undesired behavior. To help secure the State Server instance, you must make sure that it is protected from unauthorized access by machines other than your Web server using Windows Firewall or IPSec policies. You should also change the network port on which the server runs by setting the following registry key to a new port:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state
\Parameters\Port=PORT

Since the time of the .NET Framework 1.1 (along with .NET Framework 1.0 SP3 and .NET Framework 2.0), the state server will only listen for connections by default on the local loopback interface. You can configure the State Server to listen for remote connections from your Web server by setting the following registry key on the machine running the State Server instance:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state
\Parameters\AllowRemoteConnection=1

Conclusion

ASP.NET session state can provide a compelling mechanism for state management in single application and Web farm environments. ASP.NET 2.0 provides a number of new session state capabilities that enable applications to better leverage the power of session state in order to build rich, stateful experiences traditionally available only to client applications. By applying the design and deployment best practices described in this article, your Web applications can maintain session state with better performance, scalability, and security.

Michael Volodarsky is a technical Program Manager on the Web Platform and Tools Team at Microsoft. He owns the core server infrastructure for ASP.NET and IIS. He is now focusing on improving the Web application platform in the next generation Web server, IIS 7.0.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


ASP.NET 2.0: Personalize Your Portal with User Controls and Custom Web Parts

New information has been added to this article since publication.
Refer to the Editor's Update below.

ASP.NET 2.0

Personalize Your Portal with User Controls and Custom Web Parts

Ted Pattison and Fritz Onion
This article is based on a prerelease version of ASP.NET 2.0. All information contained herein is subject to change.
This article discusses:
  • Using Web Parts to create modular Web portal applications
  • Personalization and customization features
  • Using custom user controls as Web Parts
  • Creating a personalization provider
This article uses the following technologies:
ASP.NET 2.0
Code download available at: WebParts.exe (619KB)
Contents
Web Part Primer
Display Modes and Page Scope
Properties and Personalization
Web Part Catalogs
ASP.NET 2.0 Portal Development
User Controls as Web Parts
Web Parts and Personalization Providers
Changing the Personalization Data Store
Creating Your Own Personalization Provider
Taking it a Step Further

P
ortal applications are very popular today and the better ones share some common traits. Useful portals serve up generous portions of content through a user interface that is modular, consistent, and easy to navigate. More sophisticated portals go even further and allow site members to contribute content, upload documents, and personalize portal pages.

Microsoft added a scalable portal framework to the Windows Server™ 2003 platform with the release of Windows® SharePoint® Services, which provides the basic elements required by a portal framework, including support for site membership, content and document management, and the modular presentation of data through the use of Web Parts.

Web Parts provide the foundation for both customization and personalization. Depending on the site configuration, users can easily personalize or customize pages in a Windows SharePoint Services site by adding, reconfiguring, and removing Web Parts. Developing custom Web Parts provides an easy and powerful way to extend sites based on Windows SharePoint Services. When you create a custom Web Part that supports customization and personalization, you simply add properties to your Web Part class and apply a few special attributes. The Web Part infrastructure of Windows SharePoint Services does all the heavy lifting of serializing, storing, and retrieving the data associated with site customization and member personalization.

ASP.NET 2.0 introduces a Web Part control set that is similar to Windows SharePoint Services in the sense that it was designed to deal with the serialization, storage, and retrieval of customization and personalization data behind the scenes. It's different and more flexible in that it's not tightly coupled to either SQL Server™ or Active Directory®. That's great news for companies that want to build portal applications using forms-based authentication or don't want to be tied to a particular database solution.

Figure 1 Sample Portal Showing a Modular Web Part Design
Figure 1 Sample Portal Showing a Modular Web Part Design

This article walks you through a sample portal application written using Web Parts in ASP.NET 2.0. The primary goal is to teach you about the important design issues you will face when developing Web Parts for a portal application. We'll start by highlighting the fundamental concepts and control types involved with using the new Web Part control set in ASP.NET 2.0. For an example, see Figure 1.

Web Part Primer
Figure 2 Typical Layout for a Web Part Page
Figure 2 Typical Layout for a Web Part Page

A page that is designed to host Web Parts is known as a Web Part Page. A Web Part Page requires exactly one instance of the WebPartManager control and one or more WebPartZone controls, as shown in Figure 2. A Web Part Page may optionally include an EditorZone or CatalogZone control. Note that the tag for the WebPartManager control must be placed in an .aspx file before the tags for any of the other controls associated with the Web Part infrastructure, such as WebPartZone, EditorZone, and CatalogZone. For better control over the layout and appearance of the Web Part Page, you can also lay out the different zones within the .aspx file by using HTML tables.

Let's start with a simple example of a Web Part Page that contains WebPartManager and WebPartZone controls:

<asp:WebPartManager ID=" WebPartManager1" runat="server" />

<asp:WebPartZone ID="WebPartZone1" runat="server" HeaderText="Zone 1">
  <ZoneTemplate>
       <!-- time to add a Web Part -->
     </ZoneTemplate>
   </asp:WebPartZone>

Once you have a WebPartZone, you can use a Web Part definition to create a Web Part instance. There are two different approaches for creating a Web Part definition. First, you can create a custom class that inherits from the WebPart class. Second, you can create a user control. We'll take a more detailed look at the trade-offs between these two approaches later in this article. For now, let's begin by writing a simple class derived from WebPart (see Figure 3).

Figure 3 Class Derived from WebPart (Close Figure 3)
using System.Web.UI;
using System.Web.UI.WebControls.WebParts;

namespace WingtipWebParts {
  public class HelloWorld : WebPart  {

    public HelloWorld()  {
      this.Title = "Hello World";
      this.TitleIconImageUrl = @"img\WhaleBoy.gif";
    }

    protected override void RenderContents(HtmlTextWriter writer)  {
      writer.Write("Can I have your attention please");
    }
  }
}

Each Web Part instance exists on a specific page within a specific WebPartZone at a specific index. A WebPartZone can contain multiple Web Parts. For example, Figure 4 shows two Web Parts within the zone called WebPartZone1.

Figure 4 Web Parts in a Specific Zone
Figure 4 Web Parts in a Specific Zone

You can add a Web Part to a zone either programmatically or declaratively. A bit later you will also see a technique for adding Web Parts to a catalog of parts. Once you have created a CatalogPart, your users will be able to add new Web Parts into WebPartZones at run time.

The way to add Web Parts to WebPartZones with code varies depending on the type of Web Part you are dealing with. If you have a class that inherits from WebPart, you can create an instance of the class programmatically and call the AddWebPart method of the WebPartManager class. When you call AddWebPart, you must pass parameters to specify the Web Part instance, the target WebPartZone, and the index at which the Web Part will appear within the target zone:

// create Web Part instance from WebPart-derived class
WebPart wp1 = new WingtipWebParts.HelloWorld();
WebPartManager1.AddWebPart(wp1, WebPartZone1, 0);

That's the programmatic way of adding a Web Part to a WebPartZone. The declarative approach uses control tags within the .aspx file of the Web Part Page. If you want a Web Part to appear in a specific WebPartZone when the page is first retrieved by a user, you can add a ZoneTemplate within a WebPartZone:

<%@ Register Assembly="WingtipWebParts" Namespace="WingtipWebParts"
TagPrefix="Wingtip" %>

<asp:WebPartManager ID=" WebPartManager1" runat="server" />

<asp:WebPartZone ID="WebPartZone1" runat="server" HeaderText="Zone 1">
  <ZoneTemplate>
    <Wingtip:HelloWorld runat="server" id="HelloWorld" />
  </ZoneTemplate>
</asp:WebPartZone>

Keep in mind that your Web Parts are going to look pretty boring on a Web Part Page if you don't dress them up a bit. A common feature request for intranet portal applications is the ability to "skin" the site, changing the appearance based on user preferences. In the past this meant that you had to build your own infrastructure to support changing control rendering, and this involved a fair amount of work to get right. ASP.NET 2.0 introduces the concept of themes, which are collections of styles and control attributes that can be applied individually to controls, across an entire page, or even globally at the application level.

To make your portal application look polished and professional, customize the appearance of all WebPartZone, EditorZone, and CatalogZone controls. This exercise can be tedious the first time you go through it because it involves skinning and branding the visual appearance of the body, title bars, and dynamic menus for Web Parts, Editor Parts, and Catalog Parts. Fortunately, the new themes feature of ASP.NET 2.0 makes it possible to factor all this visual work out of your .aspx files and into reusable .skin files and .css files. The sample portal application for this article customizes Web Part appearance by using skins and themes, and is available on the MSDN® Magazine Web site.

Display Modes and Page Scope

A single instance of the WebPartManager control runs on each Web Part Page and is responsible for managing Web Part instances and how they relate to WebPartZone controls. The WebPartManager control also provides a programmatic interface, making it possible to switch the Web Part Page between browse, design, and edit display modes. For example, to programmatically switch the current page into design mode, you can simply add a link control with an event handler that sets the DisplayMode property to DesignDisplayMode:

WebPartManager1.DisplayMode = WebPartManager.DesignDisplayMode;

It's important to understand the differences between each of the display modes of a Web Part Page. By default, a Web Part Page operates in browse mode, which doesn't let the user make changes to any Web Parts. When the Web Part Page is switched to design mode, the user is able to move Web Parts within a WebPartZone or between zones. Figure 5 lists the available display modes.

Figure 5 Web Part Page Display Modes (Close Figure 5)

Display Mode | Description
BrowseDisplayMode | Standard view mode. No personalization or editing.
DesignDisplayMode | Enables drag and drop layout personalization or customization.
EditDisplayMode | Enables personalization or customization of Web Part properties to change appearance and behavior. Also permits the user to delete Web Parts that have been added to the page dynamically.
ConnectDisplayMode | Enables user to connect Web Parts at run time.
CatalogDisplayMode | Enables user to add Web Parts into a WebPartZone at run time.

The ASP.NET 2.0 Web Part controls produce all the required DHTML and JavaScript behind the scenes to enable drag and drop within the browser. On browsers that don't support the necessary DHTML and JavaScript features, all of the functionality other than drag and drop editing remains. This means that you don't have to worry about mandating that all clients use Microsoft® Internet Explorer 5.0 or higher. The ASP.NET 2.0 Web Part controls also take care of managing the storage and retrieval of personalization data to remember where users have placed Web Parts during previous sessions.

In addition to allowing your code to switch between display modes, WebPartManager also provides the means for changing a Web Part Page between user scope and shared scope. The scope of the page dictates whether a change to a Web Part is a customization or a personalization. Remember that customization changes are seen by all users, while personalization changes are seen only by the user who made them. Changing a Web Part Page is accomplished by calling the ToggleScope method of the WebPartManager control, as shown here:

WebPartManager1.Personalization.ToggleScope();

The default scope is user scope, in which changes made to Web Parts are recorded as personalizations that are only seen by the current user. A successful call to the ToggleScope method will place a Web Part Page in shared scope, in which changes are recorded as customizations. The purpose of shared scope is to allow an administrator or site designer to make sweeping customization changes to the Web Parts on a Web Part Page that are seen by all users. Any personalization changes made by users always take precedence over global customizations if there is any overlap.

It is not always possible for the current user to enter shared scope. By default, no user has the required permissions to enter shared scope. A user must be granted permission to do so in the Web.config file. Here's an example of granting permission for all users in the admin or site_designer roles to enter shared scope and modify global customizations:

<webParts>
  <personalization>
    <authorization>
      <allow roles="admin, site_designer" verbs="enterSharedScope" />
    </authorization>
  </personalization>
</webParts>

Properties and Personalization

Each Web Part object has a standard set of properties that can be customized or personalized. For example, each Web Part has a Title property that can be customized after it has been added to a WebPartZone. The EditorZone control and various Editor Parts allow users to make changes to Web Part properties.

When the user sets a Web Part Page to EditDisplayMode, the Web Part menu provides an Edit command. Invoking the Edit command on a particular Web Part brings up the EditorZone along with whatever Editor Parts you have placed into the zone in question. ASP.NET 2.0 ships with several built-in Editor Parts for modifying the standard Web Part appearance, behavior, and layout.

The Web Part personalization scheme is simple to extend by adding custom properties that can be personalized. You just add properties to a Web Part class definition and apply attributes such as Personalizable, WebBrowsable, and WebDisplayName. Once you have done this, the Web Part controls take care of storing and retrieving property values that have been customized or personalized.

When you create a personalization property, it usually makes sense to define a private field along with it, like so:

private bool _HR = true;

[Personalizable(PersonalizationScope.User),
 WebBrowsable, WebDisplayName("Show HR News"),
 WebDescription("Use this property to show/hide HR news")]
public bool HR
{
  get { return _HR; } set { _HR = value; }
}

If you want to allow your users to personalize a property like this, you only need to add a PropertyGridEditorPart into an EditorZone for the current page. Figure 6 shows the experience your users will have when you do this.

Figure 6 Editor Parts Allow Users to Personalize Web Parts
Figure 6 Editor Parts Allow Users to Personalize Web Parts

If you define a Web Part property based on a string or numeric type, the PropertyGridEditorPart will provide a textbox for the user to change the value. If you define a Web Part property based on a Boolean value, the PropertyGridEditorPart will provide a checkbox, as shown in Figure 6.

Figure 7 shows a nifty programming trick that carries over from Web Part development with Windows SharePoint Services: you can define a personalization property whose type is based on an enumeration type.

Figure 7 Custom Personalization Property (Close Figure 7)
public enum Timeframe
{
  Today,
  Week,
  Month
}
...
protected Timeframe _Timeframe = Timeframe.Month;

[Personalizable(PersonalizationScope.Shared),
 WebBrowsable, WebDisplayName("Select a news timeframe"),
 WebDescription("Use this property to set timeframe")]
public Timeframe Timeframe
{
    get { return _Timeframe; } set { _Timeframe = value; }
}

The real value of creating a WebBrowsable, personalization property based on an enumeration is that the PropertyGridEditorPart generates a dropdown list of available property settings for the user, as shown for the Timeframe property in Figure 7. This is a nice convenience for the user and it also helps to ensure that the user picks a valid property value.

There's one more thing to note about the Timeframe property definition. It has been defined with a Personalizable attribute that has a setting of PersonalizationScope.Shared. When a property is defined in this manner as a shared property it can be customized, but not personalized. Since a shared property is not available for personalization, it does not show up in the PropertyGridEditorPart when the current Web Part Page is in user scope. It will only appear when the page is in shared scope.

Web Part Catalogs

You've seen Web Part Pages that prepopulate Web Parts into WebPartZones. You can complement this technique with another approach in which you allow users to add new Web Parts at run time. This is made possible through the use of a CatalogZone and CatalogParts such as PageCatalogPart and DeclarativeCatalogPart (see Figure 8).

Figure 8 Adding New Web Parts To Page (Close Figure 8)
<asp:CatalogZone ID="CatalogZone1" runat="server" >
  <ZoneTemplate>

    <!-- allows users to re-add Web Parts to page after being closed -->
    <asp:PageCatalogPart ID="PageCatalogPart1"
                         runat="server" Title="Local Page Catalog" />

    <!-- allows users to add new Web Parts to page -->
    <asp:DeclarativeCatalogPart ID="DeclarativeCatalogPart1"
                                runat="server"
                                Title="Wingtip Web Parts">
      <WebPartsTemplate>
        <cc1:WeatherWebPart ID="WeatherWebPart1"
                            runat="server"
                            ZipCode="04090" />
      </WebPartsTemplate>
    </asp:DeclarativeCatalogPart>
  </ZoneTemplate>
</asp:CatalogZone>

Once you have added a CatalogZone and CatalogPart in this manner, users will be able to dynamically add Web Parts at run time through a user interface like the one shown in Figure 9.

Figure 9 CatalogZones Allow Users to Dynamically Add Web Parts
Figure 9 CatalogZones Allow Users to Dynamically Add Web Parts

First, you need to understand the purpose of the PageCatalogPart. When in design display mode or edit display mode, the user has the ability to invoke the Close command on a Web Part. When a user closes a Web Part, the Web Part and its personalization or customization settings are retained so the user can add that WebPart again at a later time. Therefore, the PageCatalogPart shows a list of all the Web Parts that have been closed and then can be added to the page.

The Close command is different from the Delete command. The Delete command is only available when in edit display mode. When a user deletes a Web Part, all information about the existence of the Web Part, including its customization and personalization data, is deleted from storage.

A DeclarativeCatalogPart can be declaratively populated with Web Parts. The code in Figure 8 shows how to populate this catalog with a custom-named WeatherWebPart. Using this approach, you have the ability to make all kinds of Web Parts available to your users.

To complement this article and to give you more detail on the nuts and bolts of building Web Part Pages, we recommend reading the article "Introducing the ASP.NET 2.0 Web Parts Framework," by Stephen Walther. It provides more detail and working examples of building out Web Part Pages using EditorZones and CatalogZones. Stephen also delves into some more advanced topics, including Verbs, Connections, and importing and exporting Web Parts.

ASP.NET 2.0 Portal Development

In addition to the WebPart infrastructure itself, there are several new features that make the development of intranet Portal sites appealing in ASP.NET 2.0. As we mentioned earlier in this article, the introduction of themes and skins makes it very straightforward to isolate style properties in your portal pages and to make sweeping style changes without touching individual pages. Perhaps even more compelling, however, is the introduction of Master Pages. With Master Pages it is possible to isolate all of your WebPartZones and the overall WebPartManager control into a single template page from which individual pages inherit their base appearance and functionality. An interesting technique that we use in the sample portal is to add ContentPlaceholder controls within the ZoneTemplate of each WebPartZone in the Master Page. This way, content pages that use this Master Page can add their own Web Parts using Content controls that map onto the respective ContentPlaceHolder control.

One of the things you need to decide in designing your Master Page for a portal site is how to make the customization features available to the user. As you have seen, depending on what zone types have been added to a page, there are several different customization modes that a user can choose from to alter the page.

One approach to displaying customization options to the user is to wrap both the WebPartManager control and a collection of buttons (typically LinkButtons) into a user control that is then dropped into the Master Page to provide customization options for all pages in the site. Encapsulating these controls into a User Control (let's call it the WebPartManagerPanel) is also useful if you end up having more than one Master Page for your Web site, providing alternative layouts for some subset of your pages. The menu bar in the portal application that you saw earlier in Figure 1 shows a sample user control that displays the current mode and scope of the WebPartManager, and it also provides LinkButtons that change the mode of the page to one of the editing modes supported by the WebPartManager (this is the control that the sample portal application uses).

The other useful feature to provide in your WebPartManagerPanel is to dynamically show and hide the display modes that are available based on the current user and the current page. You can query the supported display modes by looking in the WebPartManager's SupportedDisplayModes collection and using that information to turn on or off the menu item that exposes that mode. For example, to find out whether the current page supports CatalogDisplayMode, you would do the following:

if (WebPartManager1.SupportedDisplayModes.Contains(
    WebPartManager.CatalogDisplayMode)) {
  //enable catalog display mode LinkButton here...
}

You should also note that a call to ToggleScope will fail when executing within the context of a user without the proper permissions. It's a good idea to supply code that queries the CanEnterSharedScope property exposed by the WebPartManager controls Personalization property to determine whether to hide or show the user interface element that makes it possible for the user to enter shared scope:

if (WebPartManager1.Personalization.CanEnterSharedScope)  {
  // display UI element that allows user to enter shared scope
}

The WebPartManagerPanel user control in the sample accompanying this article contains a complete implementation of a panel that dynamically adjusts its display based on the current user and page capabilities.

User Controls as Web Parts

One of the frustrating things about building Web Parts in Windows SharePoint Services is that you have to programmatically create the entire user interface of your control with no help from the designer. Because a lot of Web Parts are collections of server-side controls that interoperate, it is unfortunate that you don't have the Visual Studio® designer at your disposal when creating Web Parts. The obvious solution to this problem is to allow developers to create ASP.NET user controls and use them as Web Parts. (A third-party tool called SmartPart provides support for employing user controls as Web Parts in Windows SharePoint Services.)

ASP.NET 2.0 Web Parts address this issue by allowing any control to be used directly as a Web Part, with no modifications or wrapping needed. This is not only useful for incorporating user controls into your Web Part collection, but is also an easy way to integrate any custom controls that you may have built for your current ASP.NET pages.

The way this works internally is if a standard control (non-Web Part) is added to a WebPartZone, an implicit call to WebPartManager.CreateWebPart is made, which allocates an instance of the GenericWebPart class and initializes it with the control that was added. The GenericWebPart class derives from the WebPart base class, providing implementations of the core Web Part properties. When GenericWebPart is constructed, it adds the control it was initialized with as a child control. During the rendering process, GenericWebPart renders nothing to the response buffer itself, and simply delegates rendering to its child control, as do most composite controls. The end result is that you can add any control you would like to a WebPartZone on a page and it works. For example, the following page defines a WebPartZone with a user control and a standard Calendar control, both of which will be implicitly wrapped by the GenericWebPart class at the time of creation:

<%@ Register Src="webparts/CustomerList.ascx"
    TagName="CustomerList" TagPrefix="Wingtip" %>

<asp:WebPartManager ID=" WebPartManager1" runat="server" />

<asp:WebPartZone ID="WebPartZone1" runat="server" HeaderText="Zone 1">
  <ZoneTemplate>
    <Wingtip:CustomerList runat="server" id="CustomerList" />
    <asp:Calendar runat="server" id="CustomerCalendar" />
  </ZoneTemplate>
</asp:WebPartZone>

As with standard Web Parts, it is possible to dynamically create controls wrapped by GenericWebPart. If it is a user control, you must dynamically load and create the user control instance with a call to Page.LoadControl. Second, you must explicitly assign a unique ID to the control. Third, you must call the CreateWebPart method of the WebPartManager object to create an instance of the GenericWebPart class, which then acts as a wrapper around the user control instance. Finally, you need to take the GenericWebPart reference returned from the call to CreateWebPart and pass it into a call to AddWebPart, specifying the WebPartZone into which it should be added:

// create Web Part instance from User Control file
Control uc = this.LoadControl(@"webparts\CompanyNews.ascx");
uc.ID = "wp2";
GenericWebPart wp2 = WebPartManager1.CreateWebPart(uc);
WebPartManager1.AddWebPart(wp2, WebPartZone1, 1);

Figure 10 GenericWebPart
Figure 10 GenericWebPart

The only disadvantage to this technique is that you lose the opportunity to control the Web Part-specific features of the control because the GenericWebPart class is the one that inherits from WebPart, not your control. This will become obvious as soon as you run a page with controls wrapped by GenericWebPart, as they default to Untitled for their titles and have no icons or descriptions associated with them as most Web Parts do. Figure 10 shows a sample user control wrapped by GenericWebParts with the default title (Untitled) and icons.

One way that you can work around this problem is to add a handler for the Init event of your user control. If your control is wrapped by GenericWebPart (which you can determine by querying the type of your Parent property), then you should set the attributes of the GenericWebPart class like this:

void Page_Init(object src, EventArgs e) {
  GenericWebPart gwp = Parent as GenericWebPart;
  if (gwp != null)  {
    gwp.Title = "My custom user control";
    gwp.TitleIconImageUrl = @"~\img\ALLUSR.GIF";
    gwp.CatalogIconImageUrl = @"~\img\ALLUSR.GIF";
  }
}

When you run the page again, as long as the user control is being wrapped by GenericWebPart, the changes you made to the properties of the GenericWebPart parent will be reflected in the rendering of the Web Part containing your control. The newly attributed user control is shown in Figure 11. Note the title and icon.

Figure 11 Title and Icon
Figure 11 Title and Icon

Another compelling solution is to implement the IWebPart interface directly on your user control class. This doesn't seem like it would help, since the user control is never queried directly for Web Part properties as those details are handled by the GenericWebPart class. Fortunately, the designers of the GenericWebPart class anticipated this need and implemented the properties in the GenericWebPart class to automatically delegate to the wrapped control if that control implements the IWebPart interface.

So customizing the Web Part features of a user control is just a matter of implementing the IWebPart interface and filling out the seven properties that it defines. The code in Figure 12 shows an example of the codebehind class for a user control that achieves the same results as we did before by dynamically altering the GenericWebPart properties.

Figure 12 User Control with Dynamic Properties (Close Figure 12)
public partial class HelloWorld2 : UserControl, IWebPart {

    protected string _title = "My custom user control";
    public string Title {
        get { return _title; }
        set { _title = value; }
    }

    private string _titleIconImageUrl = "~/img/ALLUSR.GIF";
    public string TitleIconImageUrl {
        get { return _titleIconImageUrl; }
        set { _titleIconImageUrl = value; }
    }

    private string _catalogIconImageUrl = "~/img/ALLUSR.GIF";
    public string CatalogIconImageUrl {
        get { return _catalogIconImageUrl; }
        set { _catalogIconImageUrl = value; }
    }

    // Remaining properties not shown...
}

You might even consider creating an alternative base class for your user controls. This class would derive from UserControl and implement IWebPart, and can then be inherited by all of the user controls in your portal. This is what we do in the sample portal solution accompanying this article. With this solution your user controls can initialize the properties they care about in their constructors, and the rest takes care of itself. Figure 13 shows an alternative base class for user controls that implements IWebPart, and a corresponding codebehind class for a user control that makes use of the class to set its title and icon properties.

Figure 13 IWebPart (Close Figure 13)
public class WebPartBase : UserControl, IWebPart {

  protected string _title = "[Generic Title]";
  public string Title {
    get { return _title; }
    set { _title = value; }
  }

  private string _titleIconImageUrl = "~/img/star.GIF";
  public string TitleIconImageUrl {
    get { return _titleIconImageUrl; }
    set { _titleIconImageUrl = value; }
  }

  private string _catalogIconImageUrl = "~/img/star.GIF";
  public string CatalogIconImageUrl {
    get { return _catalogIconImageUrl; }
    set { _catalogIconImageUrl = value; }
  }

  // Remaining properties not shown...
}

// Code behind class for CustomerList.ascx User Control
public partial class CustomerList : WebPartBase {
  public CustomerList() {
    this.Title = "Wingtip Customer List";
    this.TitleIconImageUrl = @"~\img\ALLUSR.GIF";
    this.CatalogIconImageUrl = @"~\img\ALLUSR.GIF";
  }
}

Now that you have so much flexibility with user controls, you may very well ask why would you ever want to create a custom control when you can have designer support with user controls and still customize the WebPart features? There are several reasons, actually, starting with the fact that you cannot add custom verbs to a user control. For that you must resort to deriving directly from WebPart and overriding the Verbs property. Alternatively, you can consider implementing IWebEditable in your control.

The other reason is that user controls are intrinsically scoped to the application directory—it is not possible to share a user control implementation across multiple Web applications without physically copying the .ascx file from one project to another. Custom Web Parts that derive from the WebPart class, on the other hand, can be compiled into a reusable DLL and deployed globally in the Global Assembly Cache (GAC). Also, with a custom Web Part, you have the option of writing a custom designer for your control to change its default appearance within Visual Studio, and you can create an icon to associate with the Web Part when it is dropped onto the toolbox. Figure 14 provides a comparison of features for deciding between custom Web Parts and user controls.

Figure 14 Custom Web Part or User Control? (Close Figure 14)

Feature | WebPart-Inherited Class | User Control
Can be built in separate DLL and installed in the GAC to reuse across applications | Yes | No
Can be added to Visual Studio toolbox with a fancy icon | Yes | No
Can contained extended verbs | Yes | No
Designer support available | No | Yes

Web Parts and Personalization Providers

Providers are a new feature of ASP.NET 2.0 that are one of the main reasons you see so many fully functional controls available in this release that require little or no coding to get up and running. The basic idea behind a provider is to define a common set of data-related tasks for a particular feature, and aggregate those tasks into an abstract class declaration that inherits from a common ProviderBase class. In the personalization features we are exploring in this article, the obvious data-related tasks that must be available include:

  • Saving the Web Part properties and layout for a particular page and user.
  • Loading the Web Part properties and layout for a particular page and user.
  • Saving the general Web Part properties and layout (for general customization) for a particular page.
  • Loading the general Web Part properties and layout (for general customization) for a particular page.
  • Resetting the Web Part properties and layout to their defaults for a particular page and user.
  • Resetting the Web Part properties and layout to their defaults (for general customization) for a particular page.

There are some other ancillary features that are part of the personalization infrastructure that need persistence capabilities too, but it basically boils down to these six capabilities. If we assume that there is a class capable of performing these six actions and successfully saving and restoring the data, then the WebPartManager on each page can use that class to save and restore all personalization and customization data as the site runs. The abstract class that defines these methods is called PersonalizationProvider, and the one concrete derivative of this class that is used by default is the SqlPersonalizationProvider. The three methods that represent the six pieces of functionality we identified are shown in Figure 15. Note that each method is capable of working with either user personalization or shared customization based on whether the incoming userName parameter is null or not.

Figure 15 PersonalizationProvider (Close Figure 15)
public abstract class PersonalizationProvider : ProviderBase {
    protected abstract void LoadPersonalizationBlobs(
        WebPartManager webPartManager,
        string path, string userName,
        ref byte[] sharedDataBlob,
        ref byte[] userDataBlob);

    protected abstract void ResetPersonalizationBlob(
       WebPartManager webPartManager,
       string path,
       string userName);

    protected abstract void SavePersonalizationBlob(
       WebPartManager webPartManager,
       string path,
       string userName,
       byte[] dataBlob);

    ... // remaining methods not shown
}
Figure 16 Interaction
Figure 16 Interaction

Note that all of the personalization data is stored as straight binary data (byte[]), which in the default SqlPersonalizationProvider is written to an image field in the database. Since ASP.NET 2.0 knows that there is a class that provides these methods, it can build much more logic into its base set of controls than was possible before. In our case, the WebPartManager in each page that uses Web Parts is responsible for making the right calls to the current PersonalizationProvider class to serialize and restore the personalization settings for each page. The diagram in Figure 16 shows the interaction between the EditorZone control and the default SqlPersonalizationProvider.

The more you work with ASP.NET 2.0, the more you will see examples of this provider architecture. There are providers for membership, role management, site map data, and health monitoring, among others, all of which define similar core sets of methods for controls to interact with.

Changing the Personalization Data Store

As with most of the providers in ASP.NET 2.0, the default provider for personalization is implemented to target a SQL Server back end. If you make no changes to the configuration files, the default SqlPersonalizationProvider uses a connection string for SQL Server 2005 Express Edition, which supports a local file-based database. The connection string looks like this:

data source=.\SQLEXPRESS; Integrated Security=SSPI;
AttachDBFilename=|DataDirectory|aspnetdb.mdf; User Instance=true

The advantage to using a SQL Server 2005 Express Edition file-based database is that it can be created on the fly without any additional setup by the user. This means you can create a brand new site, start using the personalization features without setting up a database, and it just works! When you first interact with the site, it will generate a new aspnetdb.mdf file in your site's App_Data directory, and initialize it with the tables and stored procedures necessary to support all of the default providers.

This is great for small sites that don't need to scale or support many concurrent users, but for enterprise systems, it will be necessary to store the data in a fully administered dedicated database server. Fortunately, changing the database used by the SqlPersonalizationProvider is straightforward. The SqlPersonalizationProvider configuration initializes the connection string to LocalSqlServer, which means that it looks for an entry in the <connectionStrings> section of the configuration file with the name LocalSqlServer, and uses the associated connection string to open a connection to the database. By default, this string is the connection string you saw a few paragraphs ago, meaning it will write to a local SQL Server 2005 Express Edition .mdf file. To change this, you must first clear the LocalSqlServer connection string collection and then reassign a new connection string value in your Web.config file (or alternatively, you could change this in your machine-wide Machine.config file to affect all sites on that machine). Here's an example Web.config file that will change the provider database to point to a local SQL Server 2000 instance:

<configuration xmlns="http://schemas.microsoft.com/.NetConfiguration/
   V.2.0">
  <connectionStrings>
    <clear />
    <add name="LocalSqlServer" connectionString=
        "server=.;integrated security=sspi;database=aspnetdb"/>
  </connectionStrings>
  ...
</configuration>

Before this change will work, there must be a database named aspnetdb on the local server with the necessary tables and stored procedures required by the SqlPersonalizationProvider. To create this database, there is a utility that ships with ASP.NET 2.0 called aspnet_regsql.exe. When run with the default settings, it will create a database locally called aspnetdb with the necessary tables for all of the providers. Alternatively, you can choose to install the tables and stored procedures into an existing database. The table and stored procedure names are all prefixed with "aspnet" so it is unlikely they will clash with any existing tables.

As with all providers in ASP.NET 2.0, this level of indirection creates a very flexible architecture where the back-end data store can be completely changed without any modification to the pages or Web parts contained within.

Creating Your Own Personalization Provider

The ability to change the connection string for the personalization provider gives you a certain amount of flexibility, but under the hood the SqlPersonalizationProvider uses the System.Data.Sql.Client namespace to perform its data retrieval. This means that it is limited to a SqlServer database. If you need to store your personalization in a different database, or perhaps in a completely different data store altogether, you'll have to take the next step and build your own custom personalization provider. Fortunately, most of the hard work is already done for you and is easily leveraged. As an example of writing personalization data to a different data store, the sample portal site accompanying this article has a complete implementation of a custom personalization provider called FileBasedPersonalizationProvider that persists all personalization and customization data to local binary files in the application's App_Data directory. The binary file names are generated uniquely for each user and path, and there is also one file per unique path for general user settings.

To build a custom personalization provider you must first create a new class that inherits from the PersonalizationProvider base class, and override all of the abstract methods inherited from the base class. The class declaration that is shown in Figure 17 demonstrates how to do this.

Figure 17 Custom Personalization Provider Class (Close Figure 17)
namespace Wingtip.Providers {
  public class FileBasedPersonalizationProvider : PersonalizationProvider
  {
        public override string ApplicationName {
            get { ... } set { ... }
        }

        public override void Initialize(
            string name, NameValueCollection configSettings) { ... }

        public override int GetCountOfState(PersonalizationScope scope,
            PersonalizationStateQuery query) { ... }

        public override int ResetUserState(string path,
            DateTime userInactiveSinceDate) { ... }

        protected override void LoadPersonalizationBlobs(
            WebPartManager webPartManager, string path,
            string userName, ref byte[] sharedDataBlob,
            ref byte[] userDataBlob) { ... }

        protected override void ResetPersonalizationBlob(
            WebPartManager webPartManager, string path,
            string userName) { ... }

        public override int ResetState(PersonalizationScope scope,
            string[] paths, string[] usernames) { ... }

        protected override void SavePersonalizationBlob(
            WebPartManager webPartManager, string path,
            string userName, byte[] dataBlob) { ... }

        public override PersonalizationStateInfoCollection FindState(
            PersonalizationScope scope,
            PersonalizationStateQuery query, int pageIndex,
            int pageSize, out int totalRecords) { ... }
   }
}

There are really only two significant methods that must be implemented for your personalization provider to begin working: LoadPersonalizationBlobs and SavePersonalizationBlob. These two methods represent the binary serialization of personalization data and are called by the personalization infrastructure to retrieve data when a page is loading, and to write the data back out (typically on behalf of a particular user) when data is changed in edit, catalog, or design view on a page with Web Parts.

In the sample code download, the implementation of SavePersonalizationBlob writes the dataBlob parameter to a uniquely named file based on the userName and path that are passed in. Similarly, the LoadPersonalizationBlobs implementation searches for the file (using the same naming scheme) and returns either a user data blob or a shared data blob. Each of these two methods defaults to saving or loading shared data if the incoming userName parameter is null; otherwise they save or load user data. Figure 18 shows the implementation of each of these methods in the sample FileBasedPersonalizationProvider, along with a pair of helper methods to generate unique file names based on both user names and path information. [ Editor's Update - 4/26/2006: Figure 18 has been updated.]

Figure 18 LoadPersonalizationBlobs and SavePersonalizationBlob (Close Figure 18)
protected override void LoadPersonalizationBlobs(
  WebPartManager webPartManager, string path, string userName,
  ref byte[] sharedDataBlob, ref byte[] userDataBlob)
{
  string allUsersFileName = HttpContext.Current.Server.MapPath(
    ConstructAllUsersDataFileName(path));
  string userFileName = HttpContext.Current.Server.MapPath(
    ConstructUserDataFileName(userName, path));
  allUsersFileName = string.Intern(allUsersFileName);
  userFileName     = string.Intern(userFileName);

  try
  {
    // Read in shared user data first.
    // Lock on the file name in case two clients try accessing
    // the same file concurrently - note we lock on the Interned
    // file name string, which will always return the same object
    // for identical strings
    if (Monitor.TryEnter(allUsersFileName, 5000) &&
        File.Exists(allUsersFileName))
      sharedDataBlob = File.ReadAllBytes(allUsersFileName);
    else throw new ApplicationException("Monitor timed out");

    // Next read in user specific data (if there is any)
    if (Monitor.TryEnter(userFileName, 5000) &&
        File.Exists(userFileName))
      userDataBlob = File.ReadAllBytes(userFileName);
    else throw new ApplicationException("Monitor timed out");
  }
  finally
  {
    Monitor.Exit(allUsersFileName);
    Monitor.Exit(userFileName);
  }
}

protected override void SavePersonalizationBlob(
  WebPartManager webPartManager, string path,
  string userName, byte[] dataBlob)
{
  string fileName = HttpContext.Current.Server.MapPath(
    string.IsNullOrEmpty(userName) ?
      ConstructAllUsersDataFileName(path) :
      ConstructUserDataFileName(userName, path));

  fileName = string.Intern(fileName);
  try
  {
    // lock on the file name in case two clients try accessing
    // the same file concurrently
    if (Monitor.TryEnter(fileName, 5000))
    {
      File.WriteAllBytes(HttpContext.Current.Server.MapPath(
        fileName), dataBlob);
    }
    else throw new ApplicationException("Monitor timed out");
  }
  finally { Monitor.Exit(fileName); }
}

// Helper function for creating a unique file name for all users
// based on a path
private string ConstructAllUsersDataFileName(string path)
{
  string fileName =
    path.Replace('/', '_').Replace('~', '_').Replace('.', '_');
  return "~/App_Data/allusers" + fileName + ".bin";
}

// Helper function for creating a unique file name for a particular
// user based on a path
private string ConstructUserDataFileName(string user, string path)
{
  string fileName =
    path.Replace('/', '_').Replace('~', '_').Replace('.', '_');
  return "~/App_Data/" + user + fileName + ".bin";
}

Once the provider is fully implemented, you use the providers section of the personalization configuration section to add it as a registered personalization provider. To actually begin using it, you must specify it as the default provider for personalization in your Web.config file. Here is an example of wiring up our custom file-based provider to be the default provider in our application:

<webParts>
  <personalization defaultProvider="FileBasedPersonalizationProvider">
    <providers>
      <add name="FileBasedPersonalizationProvider"
           type="Wingtip.Providers.FileBasedPersonalizationProvider" />
    </providers>
  </personalization>
</webParts>

If we run our site again, all personalization data will now be stored in local binary files. Obviously not the most scalable solution, but the sample should give you an idea of how to implement your own personalization provider on whatever back end you like. Figure 19 shows our new provider plugged into the whole Web Part infrastructure.

Figure 19 Using FileBasedPersonalizationProvider
Figure 19 Using FileBasedPersonalizationProvider

Taking it a Step Further

Now you've seen how ASP.NET 2.0 and its new Web Part control set make it relatively painless to create rich portal applications that support customization and personalization. Probably the most significant features of this infrastructure is its plug-ability. Instead of being tied to a particular serialization implementation and data store, the provider architecture makes it relatively easy to write the personalization data to any back-end data store that makes sense for your site. So, go ahead and start creating customizable sites today with ASP.NET 2.0 Web Parts.

If you have enjoyed this article, there is plenty more to learn when it comes to building on Web Parts in an ASP.NET 2.0 portal application. Be sure to download the accompanying sample from the MSDN Magazine Web site. There are a number of samples you can work through on the ASP.NET 2.0 QuickStart tutorials. We also recommend checking out Fredrik Normén's blog, which has several interesting examples of working with Web Parts in ASP.NET 2.0.

Ted Pattison is an author and trainer who delivers hands-on training classes through Pluralsight, provides consulting through Ted Pattison Group, and is the author of several books.

Fritz Onion is a co-founder of Pluralsight, an education and content-creation company, where he focuses on Web development with ASP.NET. Fritz is the author of Essential ASP.NET (Addison Wesley 2003). Read his blog at www.pluralsight.com/fritz.


Call MOM: Instrument and Monitor Your ASP.NET Apps Using WMI and MOM 2005

Call MOM

Instrument and Monitor Your ASP.NET Apps Using WMI and MOM 2005

Michael Jurek
This article discusses:
  • The architecture of WMI
  • Building an ASP.NET module that generates WMI events
  • Creating a MOM management pack that monitors instrumented ASP.NET applications
This article uses the following technologies:
ASP.NET, WMI, MOM
Code download available at: WMI.exe (195KB)
Contents
Understanding WMI
Instrumenting an ASP.NET Application
Plugging into ASP.NET
Developing the Module
Deploying the Module
Testing the WMI Module
Using Microsoft Operations Manager
Creating a Custom Management Pack
Measuring Performance Rules
Watching the Results

T
he Microsoft® Dynamic Systems Initiative (DSI) provides a roadmap for simplifying enterprise software development, deployment, and management. Current Microsoft products and technologies that support the DSI model for system management include Automated Deployment Services, Microsoft Operations Manager (MOM) 2005, Systems Management Server (SMS) 2003, Virtual Server 2005, and Windows Server™ Update Services. Future technologies expected to extend the DSI model include Visual Studio® 2005 Team System and the System Definition Model (SDM) planned for a future version of Windows Server. (For more information about DSI, see Dynamic Systems Initiative.)

With these key DSI and SDM deliverables planned for the future, developers need manageable apps right now. Fortunately, the current version of Windows® Management Instrumentation (WMI) is able to satisfy many manageability requirements. In this article I'll demonstrate how WMI provides a first step towards DSI-style system management. I will develop a WMI-aware monitoring solution you can use to instrument your ASP.NET applications, and I'll introduce you to the capabilities of MOM 2005 that allow you to then monitor these instrumented applications.

Understanding WMI

WMI is the Microsoft implementation of Web-Based Enterprise Management (WBEM), and it has been an integral component of the Windows operating system since Windows 2000 (see Alan Boshier's MSDN®Magazine article on WMI at Windows Management Instrumentation: A Simple, Powerful Tool for Scripting Windows Management). WMI is an object-oriented repository that provides data about managed hardware and software. Central to WMI is the Common Information Model Object Manager (CIMOM) and a storage area called the CIM repository. CIMOM offers a unified programming interface for accessing data stored in the repository. CIMOM also offers a plug-in model for providers that expose objects to be managed. The operating system includes a number of providers allowing access to and management of hardware drivers, event logs, registry keys, performance counters, network settings, threads, processes, and so on (see Figure 1). Also, many Windows Server System technologies expose their management interfaces through WMI, including Application Center, BizTalk® Server, Operations Manager, SQL Server™, and Systems Management Server.

Figure 1 WMI Architecture
Figure 1 WMI Architecture

Objects in the CIM repository are organized in logical containers called namespaces. Core operating system classes use the default namespace root\cimv2. Other managed products create their own namespaces. For example, BizTalk Server uses root\MicrosoftBizTalkServer. Each namespace owns metadata information in the schema, defining object classes and their properties, methods, and relationships. For example, every process in the operating system is represented by an instance of the Win32_Process class in the root\cimv2 namespace. The Win32_Process class has many properties, such as Name, Caption, ExecutablePath, and MinimumWorkingSetSize. It has also methods, such as Terminate, GetOwner, and SetPriority.

From the perspective of a database developer, a namespace might look like a database, since classes are table definitions, instances of a class are rows, object properties are columns, and relationships are joins between tables. WMI Query Language (WQL), a SQL-like language, lets you query objects in the CIM repository.

The CIM repository supports three kinds of events: extrinsic, intrinsic, and timer. Extrinsic events are explicitly fired by the application; examples include a power state change and the start of the shutdown process. Intrinsic events resemble database triggers; they are fired whenever managed objects are created, modified, or deleted. And timer events fire when a timer goes off.

WMI provides COM-based interfaces that allow access to all the data it holds, and these COM interfaces are frequently used in scripting environments to manage computers through automated scripts. The Microsoft .NET Framework provides a more convenient option. The System.Management namespace offers a layer on top of that COM functionality and makes heavy use of .NET Framework constructs such as delegates, iterators, and collections (see Zina Posen's article on WMI and .NET at WMI and .NET: System.Management Lets You Take Advantage of WMI APIs within Managed Code).

Most often, you will use the ManagementObject class in this namespace to access a specific object in the repository. The ManagementObjectSearcher class has facilities for querying repository data and iterating through collection of results. And finally, the ManagementEventWatcher class can listen to WMI events and fire .NET Framework events in response. If you want your application to act as a WMI provider, offering management data and firing events, you will use objects from the System.Management.Instrumentation namespace. Management objects and events are created by inheriting from Instance and BaseEvent classes, respectively, or by decorating class definitions with InstrumentationClassAttribute.

Instrumenting an ASP.NET Application

There is a wealth of information about an ASP.NET application that you might want to instrument and monitor. My solution will take advantage of some of that data in order to implement the following list of functionalities:

  • Fire an event whenever an unhandled exception occurs during request processing.
  • Fire an event whenever request processing takes longer than a predefined period of time.
  • Provide the current number of available worker and I/O threads in the thread pool (which can be used as one factor in evaluating an application's health).
  • Provide the current number of active anonymous and authenticated users (an active user is defined here as a request from a unique IP address occurring within the last five minutes).

This is just a random selection of the types of information that can be obtained from the current HttpContext information and other sources, so you're really only limited by your imagination in terms of the quality and quantity of data you expose. For example, I could measure the number of unique users based on unique cookie values, the number of requests made using legacy browsers, the frequency of requests coming from the most active IP address (a possible indication of Denial of Service attacks), the number of requests coming from specific network subnets, the number of requests exceeding predefined content length, and so on.

All of my instrumentation requirements can be fulfilled with WMI. I'll define ASPNET_ErrorEvent and ASPNET_ExcessivelyLongRequestEvent event classes that will be fired whenever the corresponding condition occurs. In addition, I will create an ASPNET_Counters class that contains my ASP.NET performance counters and statistics data.

Plugging into ASP.NET

The ASP.NET architecture allows for very modular systems, making it easy to integrate WMI monitoring into ASP.NET applications. Before processing an incoming request, the ASP.NET runtime creates an instance of the HttpContext class, which provides access to the incoming HttpRequest, to the outbound HttpResponse, and to a variety of other important objects. This context is passed through a series of pipeline stages, the most important being the actual processing endpoint—requests are routed to objects whose classes implement the IHttpHandler interface (see the diagram in Figure 2). For example, any requests for ASP.NET pages are processed by the System.Web.UI.PageHandlerFactory class, which given an HttpContext returns an appropriate IHttpHandler. The IHttpHandler then processes the request through its ProcessRequest method. There are other important stages in the request pipeline: BeginRequest, AuthenticateRequest, AuthorizeRequest, and EndRequest, to name a few.

Figure 2 ASP.NET Processing Pipeline
Figure 2 ASP.NET Processing Pipeline

ASP.NET is extensible through modules, special components implementing the IHttpModule interface. These modules are configured in a dedicated section of the Web.config file for every ASP.NET application, and in the Machine.config file for the entire computer. Before processing an incoming request, modules are instantiated and given a chance to register hooks for pipeline processing stages. This mechanism is used by a handful of ASP.NET modules in the .NET Framework (see George Shepherd's ASP Column where he presents a list of modules and a description of their interaction).

IHttpModule is the basis for many ASP.NET infrastructure extensibility projects, including my WMI instrumentation module. The module registers event handlers for the BeginRequest, EndRequest, and Error events. In this way, the module is notified of every unhandled exception and can also measure the time interval between BeginRequest and EndRequest events, as well as updating statistics as necessary.

This approach has a couple of advantages. The module will be absolutely non-intrusive and orthogonal to any ASP.NET applications and extensions. Its installation is a simple two-step process. The first step is to copy the compiled module to the instrumented server; it either can be copied to the bin folder of every application or can be installed into the Global Assembly Cache (GAC) for sharing between applications. The second step is to register the module in a configuration file—either application-wide in a Web.config file or system-wide in a Machine.config file. Optionally, you can specify maximum tolerable request duration in either file. As a result, you can start monitoring of an ASP.NET application without any downtime; you won't even need to recompile it. For information specific to your applications, you can of course customize this solution as appropriate.

Developing the Module

To start writing the code, first specify the WMI namespace for the managed classes by adding a single InstrumentedAttribute to the assembly metadata (typically done in AssemblyInfo.cs):

using System.Management.Instrumentation;
[assembly:Instrumented(@"root\MsdnMagazine\InstrumentedHttpModule")]

Creating a WMI-managed class is trivial; see Figure 3 for an example of a WMI-managed class called ASPNET_ExcessivelyLongRequestEvent. There are only two remarkable things in the code. First, there is an optional ManagedNameAttribute attached to the class, which defines class name in WMI. If you omit this attribute, the WMI class name will be the same as a .NET Framework class name. As there are slightly different naming conventions in each environment, you should rename the classes.

Figure 3 ASPNET_ExcessivelyLongRequestEvent (Close Figure 3)
using System;
using System.Management.Instrumentation;

namespace MsdnMagazine.InstrumentedHttpModule
{
    [ManagedName("ASPNET_ExcessivelyLongRequestEvent")]
    public class AspNetExcessivelyLongRequestEvent : BaseEvent
    {
        private string applicationNameValue;
        public string ApplicationName
        {
            get { return applicationNameValue; }
        }

        private int durationValue;
        public int Duration { get { return durationValue; } }

        private string requestPathValue;
        public string RequestPath { get { return requestPathValue; } }

        public AspNetExcessivelyLongRequestEvent(
            string applicationName, int duration, string requestPath)
        {
            applicationNameValue = applicationName;
            durationValue = duration;
            requestPathValue = requestPath;
        }
    }
}

The second thing to note in the code is that the class inherits from a BaseEvent class defined in the System.Management.Instrumentation namespace, making the class WMI-manageable. As I mentioned earlier in the article, this is one of two ways to define WMI event classes. I have chosen the inheritance approach over the attribute approach because it leads to more elegant code. The other two managed classes, ASPNET_ErrorEvent and ASPNET_Counters, have very similar definitions and can be found in the code sample download accompanying this article.

The core module is shown in the code in Figure 4. There are six static fields: the name of the ASP.NET application, counters for anonymous and authenticated users, the threshold for excessively long requests, a managed class for the ASP.NET counters (not to be confused with Windows NT® performance counters), and a timer for refreshing counters.

Figure 4 Core Code of InstrumentedHttpModule (Close Figure 4)
using System;
using System.Web;
using System.Threading;
using System.Configuration;
using System.Management.Instrumentation;

namespace MsdnMagazine.InstrumentedHttpModule
{
    public class HttpModule: IHttpModule
    {
        private static string applicationName;
        private static ExpiringUniqueStringCounter anonymousUsers;
        private static ExpiringUniqueStringCounter authenticatedUsers;
        private static int excessivelyLongRequestThreshold=2000;
        private static AspNetCounters counters;
        private static Timer refreshTimer;

        static HttpModule()
        {
            applicationName = HttpContext.Current.
                Request.ApplicationPath.Replace("/","");
            anonymousUsers = new ExpiringUniqueStringCounter(
                TimeSpan.FromMinutes(5));
            authenticatedUsers = new ExpiringUniqueStringCounter(
                TimeSpan.FromMinutes(5));

            // If present, read long request threshold from config file
            if (ConfigurationSettings.AppSettings[
                "ExcessivelyLongRequestThreshold"] != null)
                excessivelyLongRequestThreshold = int.Parse(
                    ConfigurationSettings.AppSettings[
                        "ExcessivelyLongRequestThreshold"]);

            // Initialize and publish counters
            counters = new AspNetCounters(applicationName);
            RefreshCounters(null);
            counters.Published=true;

            // Start periodic refresh of counters
            refreshTimer = new Timer(new TimerCallback(RefreshCounters),
                null, 1000, 1000);
        }

        static void RefreshCounters(object state)
        {
            int availableWorkerThreads, availableIOThreads;
            ThreadPool.GetAvailableThreads(out availableWorkerThreads,
                out availableIOThreads);
            counters.Update(authenticatedUsers.Count,
                anonymousUsers.Count, availableWorkerThreads,
                availableIOThreads);
        }

        public void Init(System.Web.HttpApplication context)
        {
            context.Error += new EventHandler(context_Error);
            context.BeginRequest +=new EventHandler(context_BeginRequest);
            context.EndRequest += new EventHandler(context_EndRequest);
        }

        public void Dispose() {}

        private void context_Error(Object sender, EventArgs e)
        {
            Exception ex = HttpContext.Current.Server.
                GetLastError().InnerException;
            string typeName = ex.GetType().FullName;
            string message = ex.Message;
            string requestPath = HttpContext.Current.Request.Path;
            string details = ex.ToString();
            new AspNetErrorEvent(applicationName, typeName, message,
                requestPath,details).Fire();
        }

        private DateTime startTime;

        private void context_BeginRequest(object sender, EventArgs e)
        {
            startTime = DateTime.Now;
        }

        private void context_EndRequest(object sender, EventArgs e)
        {
            // Increment corresponding counter
            string ipAddress = HttpContext.Current.Request.
                ServerVariables["REMOTE_ADDR"];
            if (HttpContext.Current.Request.IsAuthenticated)
                authenticatedUsers.Add(ipAddress);
            else
                anonymousUsers.Add(ipAddress);
            // Fire excessively long request event if necessary
            int duration = (int) DateTime.Now.Subtract(
                startTime).TotalMilliseconds;
            if (duration > excessivelyLongRequestThreshold)
            {
                string requestPath=HttpContext.Current.Request.Path;
                new AspNetExcessivelyLongRequestEvent(applicationName,
                    duration,requestPath).Fire();
            }
        }
    }
}

User counters are instances of the ExpiringUniqueStringCounter class (see the accompanying code sample download). This class holds a private Hashtable of strings and the time of their last insertion. Upon retrieving the Count property, entries older than the predefined interval are removed and the count of remaining table entries is returned.

ASPNET_Counters class is a static member of the module, and therefore you will have one instance of this class per application domain. It is initialized and published to WMI in the module's static constructor that also starts a timer which updates these counters once per minute using the output value of ThreadPool.GetAvailableThreads and the Count property of the ExpiringUniqueStringCounter instances.

The module has to implement Init and Dispose methods defined in the IHttpModule interface. While Dispose has no role in this implementation (there aren't any unmanaged instance resources to release), Init is essential for the functioning of the module. Init is called at the beginning of every request and is the place where the module registers for the BeginRequest, EndRequest, and Error events of HttpContext.

BeginRequest's only task is to record the start time of the request processing. EndRequest has more things to do. It retrieves the request IP address from the HttpContext and passes it to either the anonymous or authenticated counter. EndRequest also calculates the time passed from start time and, if the interval is longer than the threshold, it collects the necessary data and fires ASPNET_ExcessivelyLongRequestEvent. The Error event handler has an expected role; it collects any necessary data about the exception and the environment and fires ASPNET_ErrorEvent.

Deploying the Module

As I mentioned earlier, the module either has to be copied into the /bin folder of every ASP.NET application using that module or it must be placed in the GAC. Since this module typically will be shared by many ASP.NET applications, I prefer to use the GAC, registering the assembly using gacutil.exe. like so:

gacutil /i MsdnMagazine.InstrumentedHttpModule.dll

The WMI schemas of the three instrumented object classes must be registered with CIMOM. When the instrumented class is first published or an instrumented event is first fired, the corresponding schema is automatically published by the .NET Framework runtime, if the process hosting the runtime has local administrative privileges. However, ASP.NET applications often run under a Network Service account that does not have local admin privileges; therefore you will need to register these schemas manually. Luckily, the .NET Framework contains the DefaultManagementProjectInstaller class, capable of registering WMI schemas. To use it, add a class to the assembly that inherits from the DefaultManagementProjectInstaller and is attributed with the RunInstallerAttribute, as shown in the following code:

using System.ComponentModel;
using System.Management.Instrumentation;

namespace MsdnMagazine.InstrumentedHttpModule
{
    [RunInstaller(true)]
    public class WmiSchemaInstaller : DefaultManagementProjectInstaller {}
}
The easiest way to run the installer is through the InstallUtil.exe utility, which uses reflection to iterate through all classes, running the Install method on all classes derived from the System.Configuration.Install.Installer class and properly attributed with RunInstaller. (Note that in the previous example, WmiSchemaInstaller derives from DefaultManagementProjectInstaller which in turn derives from Installer.) This utility is easy to use:
installutil MsdnMagazine.InstrumentedHttpModule.dll

Finally, the following XML fragment, when added to Web.config, will register the module for ASP.NET pipeline processing and will set up the configurable threshold:

<appSettings>
    <add key="ExcessivelyLongRequestThreshold" value="2500"/>
</appSettings>
<system.web>
    <httpModules>
        <add type="MsdnMagazine.InstrumentedHttpModule.HttpModule,
            MsdnMagazine.InstrumentedHttpModule, Version=1.0.0.0,
            Culture=neutral, PublicKeyToken=8ea5a67752f3a4dd"
            name="InstrumentedHttpModule" />
    </httpModules>
</system.web>
You can add this fragment to the Web.config file of every ASP.NET application to be instrumented, or you can enable the functionality for the whole machine by adding the configuration entries to the Machine.config file.

Testing the WMI Module

You can use many tools based on WMI technology, such as the WMI Administrative Tools or the WMI Server Explorer Extension, which is integrated into Visual Studio .NET. However, you can learn more by creating a custom monitoring console that tests the ASP.NET application instrumentation created earlier. Subscribing to the management events raised by the instrumentation is simple:

   ManagementEventWatcher errorWatcher = new
      ManagementEventWatcher(
       @"\\.\root\MsdnMagazine\InstrumentedHttpModule",
       "SELECT * FROM ASPNET_ErrorEvent");
   errorWatcher.EventArrived += new
      EventArrivedEventHandler(DumpEvent);
   errorWatcher.Start();

All you have to do is to create an instance of ManagementEventWatcher, then set up the name of the monitored computer (the dot in the path string indicates a local computer) combined with the WMI namespace and WQL query selecting monitored events, and finally wire up the EventArrived event of the watcher. The event handler is rather simple—it just dumps names and values of all properties of the arrived event to the console output stream:

Console.WriteLine(managedObject.SystemProperties["__CLASS"].Value);
foreach(PropertyData property in managedObject.Properties)
{
    Console.Write("\t" + property.Name + " = " + property.Value);
}

The code for listing all the instances of ASPNET_Counters is only a little bit more complicated—just create an instance of ManagementObjectSearcher, get its enumerator, iterate through the collection of returned objects, and dump their properties using the same code as I just showed:

using(ManagementObjectSearcher searcher = new ManagementObjectSearcher(
    @"\\.\root\MsdnMagazine\InstrumentedHttpModule",
    "SELECT * FROM ASPNET_Counters"))
using(ManagementObjectCollection c = searcher.Get())
using(ManagementObjectCollection.ManagementObjectEnumerator enumerator =
    c.GetEnumerator())
{
    while(enumerator.MoveNext())
    {
        DumpManagementObject(enumerator.Current);
    }
}

Figure 5 Simple ASP.NET Testing Application
Figure 5 Simple ASP.NET Testing Application

I will simulate ASP.NET request activity using a simple ASP.NET application consisting of a single page (see Figure 5). The yellow buttons simulate long-running requests by calling Thread.Sleep, and the red buttons produce two types of unhandled exceptions. These error events immediately appear in the monitoring console together with the values of ASPNET_Counters. Spend some time playing with this testing application and watch the results in the console output. This type of monitoring can be taken even further by deploying MOM.

Using Microsoft Operations Manager

MOM 2005 provides operational monitoring of server environments that support WMI. MOM is a scalable, distributed monitoring solution (see Figure 6). Agents can be deployed automatically or manually onto managed computers in order to collect data, watch for events, and execute response actions. Agents retrieve data, including event logs and performance counters, through various providers. Collected data is forwarded to the MOM server for storage in the central SQL Server database. Administrators have at their disposal an MMC console for defining rules, events, and actions. System operators can use a Windows Forms-based application for browsing through collected data and resolving or escalating alerts.

Figure 6 MOM 2005 Architecture
Figure 6 MOM 2005 Architecture

An important part of MOM is a data store called the System Center Data Warehouse (SCDW). This is a classic star-schema database that may be periodically loaded with data from an operational database using SQL Server 2000 Data Transformation Services (DTS) technology. Collected data may be accessed and visualized using a variety of tools, such as SQL Server 2000 Reporting Services. (Note that MOM 2005 Workgroup Edition does not contain the SCDW.)

MOM management packs are carefully packaged expertise for a specific product or technology, such as Active Directory®, Exchange Server, SQL Server, and many others. Lists of management packs available from Microsoft and other providers can be found at Management Pack and Product Connector Catalog. From a technical point of view, a management pack is nothing more than a collection of rules, scripts, operator console views, and data warehouse reports. The real value of a management pack is its contained knowledge. The majority of management packs are created by the developers of the particular product. According to the Common Engineering Criteria, every new product in the Microsoft Windows Server System family will be accompanied by a management pack at launch time.

Creating a Custom Management Pack

While preconfigured management packs will be available for many commercial enterprise services, MOM also provides functionality that enables you to customize these solutions for your specific needs. In fact, you can even create customized management packs to support specific business needs or your own applications.

The next task will be creating a prototype management pack for the ASP.NET monitoring module developed earlier. This part is not intended to be a step-by-step guide. It contains only basic guidelines and offers you a lot of space for experimentation. This management pack should be regarded as a rather simplistic prototype; you could certainly go further and fine-tune it. (A complete prototype of a management pack is part of code download accompanying this article.) See Monitor Your .NET Applications with Microsoft Operations Manager 2005 for a more detailed discussion of management pack authoring.

First, open the MOM Administrator Console, which is shown in Figure 7, and start creating rules. The first step is creating a Computer Group that will serve as a target for rules in Management Packs/Computer Groups console node. This group can be created either by using a formula based on computer attributes or by explicit inclusion of the computer in the group.

Figure 7 Event and Alert Rules in MMC Console (Close Figure 7)

Formulas require registry-based computer attributes, which are not handy for this scenario. Instead, create a computer group named ASP.NET Instrumented Module and explicitly add all computers with instrumented ASP.NET applications to this group. Now you can create a new rule group, also named ASP.NET Instrumented Module, in the Management Packs/Rule Groups console node. This new rule group will be associated with the computer group of the same name and serves as a container for all the rules that you'll create later.

Next, create two rules for collecting instances of ASPNET_ErrorEvent and ASPNET_ExcessivelyLongRequestEvent, using the wizard provided. Specify the rule type as Collect, and leave all the values at their defaults with the exception of instructing MOM to store all event parameters. The only tricky part is the configuration of a data provider. You have to create a new one, using the WMI Events type. Use the following query:

SELECT ApplicationName, Duration, RequestPath
FROM ASPNET_ExcessivelyLongRequestEvent

Do the same for ASPNET_ErrorEvent. After committing configuration changes in the MMC console and waiting for the changes to propagate, MOM should start collecting the custom events. This can be checked by raising these events in the ASP.NET test apps and seeing them in Events view of the Operator Console.

To help raise problem issues to operators, MOM has rules which create "Alerts." These rules are labeled Alert On or Respond To Event. Defining these rules is pretty straightforward as you can reuse data providers from the previous step. Only the alert page requires modification. Just select Generate Alert checkbox and set alert severity to red (Critical Error) for ASPNET_ErrorEvent and yellow (Warning) for ASPNET_ExcessivelyLongRequestEvent. Note that this type of rule will also collect events, so if you create an alerting rule, no collection rule is required.

After committing the configuration, MOM should raise alerts when the events occur. Individual events leading to a specific alert can be seen in Events tab of alert details.

If you have a working SMTP messaging infrastructure, you can e-mail a specific group of administrators about critical problems. First you must create a notification group called ASP.NET Instrumented Module Administrators, then create a new operator and add this operator to the group. Create an alert rule that responds to all alerts of Critical Error severity by sending an e-mail message to this notification group. After committing and propagating changes, every new alert of Critical Error severity should result in an e-mail delivered to the operator's mailbox.

Measuring Performance Rules

To collect performance data from monitored computers, performance rules must be created in the Administrator Console. The only non-trivial part is defining a data provider (see Figure 8 for an example). Create four rules for collecting your counters (AvailableWorkerThreads, AvailableIOThreads, UniqueAnonymousUsers, and UniqueAuthenticatedUsers). The prototype samples data at one-minute intervals, generating a huge amount of information. In a production environment, it's probably better to increase this interval to 15 minutes or so. Shortly after these changes are made, you'll be able to review the performance values in the Operator Console.

Figure 8 Numeric Provider Properties
Figure 8 Numeric Provider Properties

Performance data values that are too high or too low may indicate problems in the monitored systems. It is not surprising that MOM offers alerts based on these data values. In the prototype, I created two threshold performance rules that generate yellow warning alerts when the value of the AvailableIOThreads or AvailableWorkerThreads counter drops below 5. You can reuse data providers from the previous step, set up thresholds, generate alerts of warning severity, and commit changes. To test these rules, create a testing page that synchronously waits for some external operation and overload the application using a stress tool, such as Application Center Test.

Watching the Results

At this point you have created a basic set of rules for collecting and reacting to data produced by the instrumented module. The Operator Console offers many interesting views, but most of them are too general to effectively monitor your module. You will have to create some customized views for ASP.NET application operators.

You can create filtered views for events generated by the module. In the Operator Console, create a new folder named ASP.NET Instrumented Module and then create a new alerts view named Error Events. Events should be filtered using an event source- and description-matching regular expression, as shown in the following:

(Source='WMI') AND (Description LIKE '__CLASS=ASPNET_ErrorEvent*')
You can create a similar view for Excessively Long Request Events.

Next, create alert views for the module. For example, you can create an All Unresolved Alerts view displaying all unresolved alerts in your computer group regardless of the management pack they originate from. This view is very important as it also tracks lower-level problems of managed ASP.NET computers such as low disk space, processor over-utilization, long kernel-mode HTTP request queues, and others, depending on the management packs you have installed and enabled. Other views might display only unresolved alerts triggered by ASPNET_ErrorEvent and ASPNET_ExcessivelyLongRequestEvent events, respectively. Filter expressions for these three views can be expressed like this:

(ComputerGroup='ASP.NET Instrumented Module') AND
(ResolutionState<>'Resolved')

(Source='WMI') AND (ResolutionState<>'Resolved') AND
(Description LIKE '__CLASS=ASPNET_ErrorEvent*')

(Source='WMI') AND (ResolutionState<>'Resolved') AND
(Description LIKE '__CLASS=ASPNET_ExcessivelyLongRequestEvent*')

A third view type is the performance view. Here I've created three views of Performance Data View type that are very handy for a quick view of performance data in either tabular form or a chart (see Figure 9). The first view shows unique users of both anonymous and authenticated type for every computer and every ASP.NET application. The second view does the same for the number of available worker and I/O threads. The third view displays processor utilization for every computer in the computer group (this view relies on data collected by rules from Windows Base Operating System management pack). Symbolic filter expressions for all three views are as follows:

(Object='root\MsdnMagazine\Instrumented
  HttpModule:ASPNET_Counters') AND (Counter
    LIKE 'Unique*')

(Object='root\MsdnMagazine\Instrumented
  HttpModule:ASPNET_Counters') AND (Counter
    LIKE 'Available*')

(Object='Processor') AND (Instance='_Total')
  AND (Counter='% Processor Time') AND
    (ComputerGroup='ASP.NET Instrumented Module')

Figure 9 Numeric Performance Data in Operator Console (Close Figure 9)

As I mentioned earlier in this article, data from the MOM operational database is regularly moved to a star-schema data warehouse for analysis and reporting. Data is transferred using SQL Server DTS technology, with all the dirty work done by the MOM.Datawarehousing.DTSPackageGenerator.exe utility. This utility has a handful of command-line switches for creating and running DTS packages. If all the databases physically reside on a MOM computer and have their default names, you may run this utility without command-line parameters—the utility just transfers the data and exits.

Now you can build online analytical processing (OLAP) cubes from collected data or create reports using SQL Server 2000 Reporting Services. Predefined reports that come with management packs are based on SQL Reporting Services and can be easily accessed through the Web-based Reporting Console. Custom reports are designed in Visual Studio .NET. For an excellent introduction to report building, see the article by John C. Hancock at Reporting: Deliver User-Friendly Reports from Your Application with SQL Server Reporting Services or see SQL Server 2000 Reporting Services for general information. I also recommend taking a look through the Report Definition Language (RDL) for predefined reports. Figure 10 provides an example of a custom parameterized report showing the number of unique users over a period of time.

Figure 10 Charting Collected Data in Reporting Console (Close Figure 10)

An exported management pack and report project can be found in the code download. What I've discussed here is far from a complete solution for monitoring ASP.NET applications, so feel free to take this prototype, extend it, and fine-tune it to suit your needs.

Michael Jurek is a Software Architect for Microsoft in the Czech Republic, Prague, with a specialization in distributed systems architecture and integration. Before joining Microsoft, he worked as a scientist and a trainer of Microsoft server-side technologies. Michael can be contacted at mjurek@microsoft.com.


SQL Server 2005: Fuzzy Lookups and Groupings Provide Powerful Data Cleansing Capabilities

SQL Server 2005

Fuzzy Lookups and Groupings Provide Powerful Data Cleansing Capabilities

Jay Nathan
This article is based on a prerelease version of SQL Server 2005. All information contained herein is subject to change.
This article discusses:
  • SQL Server Integration Services
  • SQL Server 2005 Fuzzy Lookup and Grouping technology
  • Data-cleansing techniques
This article uses the following technologies:
SQL Server 2005
Contents
Data Cleansing
Traditional Approaches
Error Tolerant Index
SQL Server Integration Services Basics
A Data Warehousing Scenario
Fuzzy Search Architecture in SSIS
Designing Fuzzy Search Operations in SSIS
Field Selection and Weighting
Conclusion

D
ata cleansing is an important task for data warehouse specialists, database administrators, and developers alike. Deduplication, validation, and householding techniques can be applied whether you are populating data warehouse dimensions, integrating new data into an existing transactional system, or supporting real time dedupe efforts within a transactional system. The goal is a high level of data accuracy and reliability that translates into better customer service, lower costs, and peace of mind. Data is a valuable organizational asset that should be cultivated and refined to realize its full benefit.

To help with this, the SQL Server™ 2005 database platform offers a completely redesigned SQL Server Integration Services (SSIS) engine. Formerly known as Data Transformation Services (DTS), SSIS includes many new features to aid data architects and extract, transform, and load (ETL) developers. One noteworthy new feature is built-in support for Fuzzy Lookups and Fuzzy Groupings. This is a powerful data-cleansing solution. I will provide an overview of fuzzy searching techniques and a dissection of the underlying fuzzy search technology implemented in SQL Server 2005. I'll place a particular emphasis on real-world scenarios.

Data Cleansing

Data-cleansing techniques come in several forms including deduplication, validation, and householding. Because of limitations in the way many transactional systems gather and store data, these practices become a necessary part of providing accurate information back to the business users.

Deduplication ensures that one accurate record exists for each business entity represented in a corporate transactional or analytic database. Validation ensures that each attribute maintained for a particular record is correct. Addresses are a good candidate for validation procedures where cleanup and conformation procedures are performed. Householding is the technique of grouping individual customers by the household or organization of which they are a member. This technique has some interesting marketing implications, and can also support cost-saving measures of direct advertising.

Cleansing data before it is stored in a reporting database is necessary to provide value to consumers of business intelligence applications. The cleansing process usually includes deduping processes that prevent duplicate records from being reported by the system.

Suppose that I am a customer of a life insurance company that has a sister company from whom I also buy disability insurance. That I am an individual who buys life and disability insurance through related companies should be an interesting fact to either company's actuarial department. Since the sister companies are separate legal entities, they do not necessarily share transactional systems and most likely do not even realize that my information exists in both of their databases.

Many months after I buy my policies, a business intelligence consulting firm is hired to create a data warehouse that combines key customer and policy data from both companies' systems into one unified data store. The consultants quickly determine that customer information is commonly duplicated on the systems. They must come up with a way to combine this information to leave only one conformed customer record in the database for each customer across both companies. The resulting customer record must represent the latest and greatest information as it was captured by either system. And to make matters more confusing, each sister company's transactional system was implemented independently with varying levels of data consistency validation and enforcement.

Common data inconsistencies include misspellings, inconsistent abbreviation usage, and other free-form text input anomalies. Consider the duplicate records shown in Figure 1. While these records describe the same customer to human eyes, automating a custom process to recognize this fact is much more difficult.

Figure 1 Duplicate Customer Records (Close Figure 1)

First Name | Last Name | Address1 | Address2 | City | State | ZIP Code
John | Doe | 112 Sunny Vale Ln. | Apt. #23 | Anytown | NC | 28227
John | Do | 112 Sunny Vail Lane | Apt 23 | Anytowne | NC | 28227-5410

Traditional Approaches

Any number of techniques and tools can be employed to handle these kinds of situations. Specialized structured query language constructs such as the T-SQL LIKE and CONTAINS clauses can be used for basic wildcard searches. But LIKE queries are limited in their ability to handle misspellings, and CONTAINS queries are used in conjunction with SQL Server Full Text indexing.

Fuzzy search databases can be amassed that compile common misspellings (or variants) of specific words which can then be substituted during the cleansing process. This technique works better for applications that check one word at a time, like Microsoft® Word, which employs a similar technique for making spelling corrections on the fly.

Phonetic matching algorithms, implemented in SQL Server as SOUNDEX queries, also detect similarities between single words by matching prominent phonetic characteristics that are then scored numerically for comparison. Key drawbacks to SOUNDEX are that the input string must be contiguous with no spaces, and if the first character of the input string is not correct, the probability of a match being made drops down to zero.

Error Tolerant Index

While all of these traditional approaches and techniques are certainly an option with SQL Server 2005, there is a key new tool in the data cleansing and ETL arsenal called the error tolerant index (ETI). The ETI is a decomposition of the field values contained within a reference table of values into smaller tokens. After an ETI has been generated (also known as a match index), the SSIS Fuzzy Lookup transformation consumes it along with input records that are being tested for similarity to the records in the reference table. Rather than comparing the fields of the reference and input records on a one-to-one or exact match basis, input records are scanned to determine whether they contain smaller subsets of reference table values known as tokens, providing a much finer comparison. For example, instead of searching for a street address that contains the value "112 Sunny Vail Ln.", smaller components of the reference value might be used, such as "sunn", "nyva", and "112".

The SSIS Fuzzy Lookup and Fuzzy Grouping transformations are based on a technology that extends complex string-matching functionality. Developers apply domain-specific logic to fuzzy searches in the form of a multi-column weighting scheme. As you can probably guess, this level of search granularity is not achieved without some performance tradeoffs, which I'll discuss later.

The indexing process of a Fuzzy Lookup transformation creates tokens by splitting a reference table's field values based on a set of various delimiters that include the following characters: ,.;:-"'&/\@!?()<>[]{}|#*^%, as well as the space, tab, line feed, and carriage return characters. These default delimiters can be adjusted by the developer if necessary. After individual tokens are extracted from the reference data, they are further decomposed into smaller tokens by the indexing process and stored to the match index. Figure 2 demonstrates how an address reference field value may be broken down into tokens during the indexing process.

Figure 2 Generating Sample Tokens Reference Field Value
Figure 2 Generating Sample Tokens Reference Field Value

The main difference between error-tolerant match index searches and other fuzzy search techniques is that each input record is evaluated against subsets of characters that make up the strings found in the database fields. Rather than a pass or fail inner-join test, the input record is assigned a score based on the overall set of matching tokens found in the reference table for each input record. Searching at the substring level in combination with a sophisticated character-distance-searching algorithm is what makes the match index fault-tolerant. As you can see, a mechanism exists that can power through the misspellings and other nonstandard data found in transactional systems. The similarity function also uses "term frequency/inverse document frequency" (TF-IDF) measures for assessing the importance of a token. For example, if the reference table contains A. Datum Corporation, WingTip Toys Corporation, and Adventure Works Corporation, then the token "Corporation" will have less importance than other tokens for matching purposes.

SQL Server Integration Services Basics

Before going any further, let's discuss the new features of the Business Intelligence Development Studio in SQL Server 2005. Anyone who has used Visual Studio® .NET will find this development environment fairly comfortable. Inside the Business Intelligence Development Studio, you create data transformation projects as well as Analysis Services and Reporting Services projects.

Data transformation projects contain SSIS packages, which are now individual .dtsx files. Each SSIS package that is part of a project will have its own .dtsx file under the SSIS Packages node of the Solution Explorer. All of the connections needed for use within the project are created under the Data Sources node as .ds files.

In SSIS, each SSIS Package is now divided into several main areas: the Control Flow tab represents the overall control logic of a SSIS package; the Data Flow tab actually contains the logic behind individual data flows within the package; Event Handlers allow the developer to define actions to be triggered from specific SSIS engine events raised during package execution (such as OnError, OnPreExecute, and OnTaskFailed); and the Package Explorer tab provides access to variables and metadata for the SSIS package. The Data Flow tab toolbox contains a robust new set of transformations, which is where the Fuzzy Lookup and Fuzzy Grouping tasks can be found.

A Data Warehousing Scenario
Figure 3 Multisource Data Warehouse Scenario
Figure 3 Multisource Data Warehouse Scenario

In a data warehousing scenario, the customer dimension typically contains the latest and greatest information for each customer, no matter how many databases the customer's data is stored in. Revisiting the insurance policy example, where customer information is held in two mutually exclusive databases, the data warehouse ETL developer must merge sources of customer data together at the parent-company level (see Figure 3). This means that initially two records will exist in the staging area (partitioned by a source identifier) for individuals who hold both disability and life insurance policies with the sister insurance companies.

Each time customers are loaded from the transactional source systems, a Fuzzy Lookup is used in tandem with a regular exact match Lookup transformation to ensure that data is merged for customers who already exist within the warehouse, and that only new customer records are inserted into the customer dimension. Since I consider the data warehouse dimensions a pristine, accurate, and conformed view of customers across corporate systems, this serves as the reference table for the Fuzzy Lookup operation to use. The diagram in Figure 4 depicts the typical flow of data through the Fuzzy Lookup transformation within an SSIS package flow.

Figure 4 Data Flow via a Lookup App
Figure 4 Data Flow via a Lookup App

Fuzzy Search Architecture in SSIS

The match index is a combination of two tables created by the fuzzy transformation task. One of the tables is simply a copy of the reference table with an additional column called RID attached. The RID, or reference ID, column is an integer key added to the copy of the reference table. This field is referenced by another table that stores the individual tokens generated by the Fuzzy Lookup processing task. Each row in this other table contains a string token value, a column index corresponding to a column in the reference table, and a column filled with a set of reference IDs that contain this token in the specified column.

Fuzzy search transformations come in two types, Fuzzy Lookup and Fuzzy Grouping. As you know, the Fuzzy Lookup transformation uses a reference table to create a match index against which input records are evaluated and scored. The fields chosen for Fuzzy Lookup transformations are only allowed to contain string data. The output of a Fuzzy Lookup transformation includes those pass-through fields that the developer requires for downstream data-flow operations, as well as additional metrics that provide metadata around the fuzzy searching operation that was conducted. Two extra columns are added to the Fuzzy Lookup output stream, _similarity and _confidence. These two fields can be used together to decide whether an input record might be a duplication of an existing reference record that should perhaps be updated, or a new record that should be inserted, becoming a new reference record.

The Fuzzy Grouping task performs the same operations as the Fuzzy Lookup task but instead of evaluating input records against an outside reference table, the input set becomes the reference. Input records are therefore evaluated against other records in the input set and evaluated for similarity and assigned to a group. Later in the article, I'll outline a scenario where the Fuzzy Grouping task can be applied.

Confidence and similarity are evaluated per input record. This is an important point, since it is possible to return multiple reference records for input record processed by a Fuzzy Lookup. Similarity is a score between 0 and 1 that describes the commonality between an input record and the possible match records found by a Fuzzy Lookup in the reference table. As the similarity score approaches 1, the match record becomes closer to an exact match of the input search record. A score of 1 indicates a perfect string match. Similarity is also output by Fuzzy Lookup at a column level.

Confidence is also a score between 0 and 1; it describes the likelihood that each possible match record returned by the Fuzzy Lookup is a match to the input record. This measure indicates the probability that each returned reference record for any one input record is actually a match. A high similarity factor does not necessarily mean that a definitive match has been found, but the combination of high similarity and confidence scores usually does.

Designing Fuzzy Search Operations in SSIS

When configuring a Fuzzy Lookup in an SSIS data flow, there are several key decisions to be made regarding match index reuse, reference table sources, similarity thresholds, and the input and output columns that determine a record's uniqueness.

First, you need to determine whether your match index will be reused for other Fuzzy Lookup operations. Match indexes can be reused as long as they are based upon the same set of columns. In other words, a match index configured to use the customer last name, address 1, city, state, and ZIP code fields for a deduplication process in one SSIS package can be reused by other packages only if the same set of fields can be used to perform the search. When a match index is reused, Fuzzy Lookup transformations generally run faster because they don't have to generate their own match index each time the SSIS package is run. For this reason, match index reuse is desirable if the reference table remains static.

For reference tables that do not remain static, the Fuzzy Lookup transformation can be configured to maintain the match index as the underlying reference table changes. The Fuzzy Lookup Table Maintenance feature allows the error-tolerant index to be updated each time the reference table changes and a Fuzzy Lookup task is executed against it. This feature involves adding a trigger to the reference table itself, so performance implications should be weighed carefully when considering this technique. As an alternative to both of these methods, the match index can simply be recreated each time the Fuzzy Lookup transformation is executed.

The next steps are setting the similarity threshold and maximum number of results per input record. The MinSimilarity custom property tells the Fuzzy Lookup transformation how similar a reference record must be to be returned by the search. Additionally, the transformation needs to know how many matches to output for each input record. The MaxOutputMatchesPerInput custom property sets this value. The maximum output matches per input setting and performance have an inverse relationship. The more records that the Fuzzy Lookup must locate and output, the slower the search will be. Likewise, the lower the similarity threshold setting, the longer the search duration, because the set of possible reference table matches increases.

Field Selection and Weighting

It is important for developers to understand that the fuzzy search technology available in SQL Server 2005 is powerful, but it's also domain-agnostic in its implementation. Therefore, the selection of fields that provide linkage information used to measure the similarity of input and reference data is more of an art than a science and will become the primary factor influencing a fuzzy search's accuracy. Ralph Kimball, noted as one of the fathers of modern data warehousing techniques, calls these attribute sets "survivorship blocks" in his book The Data Warehouse Toolkit (Wiley, 2002). Depending on the business domain and purpose of the operation being performed on input records, the survivorship block you define will vary, as will the weighting that each field in the block receives toward an overall match score. To illustrate the use of field selection and weighting for different matching scenarios, let's talk about deduplication and householding in more depth.

Figure 5 Customer
Figure 5 Customer

In a customer record deduplication scenario, such as that found in the sample project, the ultimate goal is to ensure that one accurate up-to-date record exists in the data warehouse's customer dimension for each unique customer across all source systems. Given the schema of the customer dimension shown in Figure 5, I must first decide which fields I might combine to form a natural key for each of my customers. For the purposes of this deduping effort, choosing CompanyName, ContactName, Address, City, Region, and PostalCode as my survivorship block should provide me with a unique customer reference. After adding these attributes to the Fuzzy Lookup transformation properties, setting my minimum similarity threshold and maximum results per input record, I am ready to begin testing input records for similarity to my reference table customer. By running subtly different input records through the ETL process, you'll get the output shown in Figure 6.

Figure 6 Sample Fuzzy Lookup Output (Close Figure 6)

CompanyName | ContactName | Address1 | City | State | ZIP Code | _similarity | _confidence
Great Lakes Food Market | Howard Snyder | 2732 Baker Blvd. | Eugene | OR | 97403 | .80674612 | .93456379
Great lakes Food Mart | Howard Synder | 2732 Baker Blvd. | Eugene | OR | 97403-1176 | .74123439 | .67434212

The fields chosen for each entity may differ significantly depending on your business rules. In addition to selecting multiple columns that represent similarity, you can also take advantage of the Minimum Similarity per Column feature, which allows match similarity and confidence scoring to be controlled at the field level.

Some columns should be valued more highly toward an overall similarity score than others. By using the advanced Fuzzy Lookup editor (right-click Fuzzy Lookup transformation task, Show Advanced Editor), match contribution values can be set higher for individual fields in the survivorship block that require a higher degree of accuracy to be considered a good match. The salutation and state fields in a customer table demonstrate why more or less emphasis should be placed on certain attributes. The salutations and state abbreviations for Alabama (AL), Alaska (AK), and Arizona (AZ) may appear relatively similar to the Fuzzy Lookup transformation, but business knowledge and understanding of these data elements suggest otherwise. Thoroughly studying the potential impact of each domain-specific attribute utilized in a Fuzzy Lookup will in essence allow you to train the operation to think in domain-specific terms. Any merging or conforming that occurs as the result of a Fuzzy Lookup's output should be logged to an audit table. Over time, analysis of audit records may indicate that the settings on the Fuzzy Lookup may need to be tweaked to be more or less inclusive in similarity evaluations.

When evaluating hierarchical data, such as a data warehouse dimension, the Minimum Similarity per Column feature can also be brought to bear. This setting instructs the Fuzzy Lookup transformation to automatically apply a heavier weighting to the attributes closer to the leaf level of the dimension hierarchy.

When designing a fuzzy search operation to aid in householding activity, the same concepts still apply, but the survivorship block will differ. Unlike in the deduping scenario, it is desirable within a householding process to search for matches only down to the individual customer household level, not the individual customer. The Address, City, Region, and PostalCode fields are utilized, leaving the name, social security, birth date, and other identifying attributes out of the survivorship block. In the end, customers who share the same address should be grouped by and assigned to a household identifier.

To achieve this with SSIS, use a two-step process. Initially, a grouping operation should be performed on the clean set of data representing the total customer population. For this task, use the Fuzzy Grouping transformation. The Fuzzy Grouping transformation accepts a set of input records and utilizes Fuzzy Lookup functionality under the covers to create a match index and assign a grouping key to sets of input records that meet the minimum similarity threshold.

The Fuzzy Grouping transformation uses a slightly different technique. It generates similarity and confidence scores based solely on the other records in the input set. Essentially, this transformation uses an input set against itself as the reference table. The Fuzzy Grouping generates a temporary match index (with the help of Fuzzy Lookup behind the scenes), which is used during the grouping operation. During the transformation process, three key fields are added to the output stream of the task: _key_in, _key_out and _score, as shown in Figure 7. The _key_in column is the reference key (RID) that is assigned to an input record when the underlying match index is generated. The _key_out column points to the reference key (_key_in) of another record in the input set considered by the fuzzy grouping to be a match according to the minimum similarity threshold setting that has been chosen. The _score field represents the similarity of the _key_in and _key_out records.

Figure 7 Additional Output Fields Generated by a Fuzzy Grouping Transformation (Close Figure 7)

_key_in | _key_out | _score
3 | 3 | 1
4 | 3 | 0.93498355
2 | 3 | 0.89746004
1 | 3 | 0.82209551

Based on a matched group of records, address information can be standardized, conformed, and stored as a household identifier. After the initial grouping utilizing the Fuzzy Grouping transformation has parsed out individual households, a Fuzzy Lookup transformation can be used to create household assignments as new customer records enter the customer dimension.

Conclusion

The fuzzy search capabilities provided by SQL Server 2005 Integration Services serve as a framework for building robust data cleansing solutions. The underlying technology utilizes state-of-the-art techniques for identifying similarities in error-prone data. This level of configurability comes with performance and storage tradeoffs, but the ever increasing processing power and near limitless storage capabilities of today's business servers allow robust processes like the Fuzzy Lookup and Grouping transformations to be executed with minimal effort.

Jay Nathan is a Business Intelligence Architect with Blackbaud, Inc., a global provider of software and related services in Charleston, SC. He has delivered solutions to the utilities, telecommunications, insurance and healthcare industries using Microsoft technologies.


Spice It Up: Sprinkle Some Pizzazz on Your Plain Vanilla Windows Forms Apps

Spice It Up

Sprinkle Some Pizzazz on Your Plain Vanilla Windows Forms Apps

Bill Wagner
This article discusses:
  • Designing eye-catching windows
  • Building "toast" pop-ups
  • Adding color and transparency to forms
  • How transparency and translucency work
This article uses the following technologies:
.NET, Windows Forms
Code download available at: WindowsForms.exe (864KB)
Contents
Spiff Up Your Apps
Removing that Main Window Dependency
Notifying Users
Using GDI+
Fade In and Fade Out
Skinning Your Application
Conclusion

T
he Microsoft® .NET Framework makes it very easy to create Windows®-based applications: You simply create a form, add controls, connect the form to your business logic, and you're finished. But such applications don't always give users the interaction they really want. For example, they typically don't notify users when important external events happen. And the windows all look the same—the same battleship grey form, the same standard controls, the same feel. Boring!

With the .NET Framework and GDI+, you can easily add some style to your applications. You can use transparency, irregularly shaped windows, notification icons, " toast" pop-ups, different color schemes, and many other design elements. Designed well, these techniques can create a much more compelling interaction between your application and your users.

Spiff Up Your Apps

There are a variety of simple techniques you can apply in order to spruce up your applications. One of these is using notification windows to alert users to important events, making them stand out, while using less intrusive methods to inform the user of lower-priority occurrences. A number of applications apply this technique effectively. For example, MSN® Messenger uses toast pop-ups above the tray area to tell you when a friend has logged in or is trying to contact you, and Outlook® 2003 provides desktop alert notifications for new e-mail messages. (These alerts also let the user delete, flag, or open the item when it arrives.) The addition of appealing forms and windows with which the user can interact also enhance the application experience. For example, Windows Media Player lets you apply your own skins, and allows for window shapes beyond the standard rectangle. Obviously, you shouldn't use all these techniques on every application, but by judiciously applying some of them, you can create applications that will enhance the user's experience.

I'm going to show you how and when to apply these techniques by modifying a simple event monitor application. The initial application shows the most recent weather data for a chosen city (see Figure 1). This typical .NET-based desktop application retrieves data from a Web service and displays the results on a form using standard Windows controls. The service I use and a number of other Web services can be found at www.xmethods.net.

Figure 1 Standard Application Window
Figure 1 Standard Application Window

This is not a particularly friendly application. You need to dedicate screen real estate to the weather report, and then you need to periodically press the Update button. I'll spend the rest of the article enhancing it to show you how to create a more compelling user experience. In the end, you'll have a weather alert center that you can leave running constantly. It will monitor particular cities and let you know when important weather changes occur. And it will break the mold of those dull grey applications. You'll modify the application to run as a Notify icon in the task bar, provide custom pop-up information when events occur, activate itself, and display its information using a custom window shape with a custom background.

Removing that Main Window Dependency

Your first task is to modify the application so that it no longer relies upon a main form, but rather creates a Notify icon when it starts. There's a reasonable amount of startup code, so it's best to create a new class to handle the application-level tasks. You'll be checking for multiple copies of the application, creating the Notify icon, adding menus and handlers, starting timers, starting the message loop, and handling some session events.

It's important to do all these small tasks when your app runs as a Notify icon because if you don't clean up properly, it's easy to end up with multiple copies of your Notify icon appearing in the notification area of the task bar. That makes it very hard to work with your app and will often confuse users trying to figure out which Notify icon is from which session of your application.

To alleviate that problem, begin by preventing multiple copies of your application from running in the first place. You typically do this by creating a global mutex that indicates your application is running (for a more detailed look at this problem, see the .NET Matters column in this issue of MSDN®Magazine). The first instance acquires a lock on the mutex and keeps it for the duration of the application. Subsequent copies attempt to acquire the lock, fail, and exit, as shown in the following:

Public Shared Sub Main()
    Dim appSingleton As New System.Threading.Mutex(False, _
        "SingleInstance WeatherAlert")
    If appSingleton.WaitOne(0, False) Then Application.Run()
    appSingleton.Close()
End Sub

Next, you're going to enable the Windows XP themes in your application. You've undoubtedly heard about EnableVisualStyles, but when using the .NET Framework 1.x you need to add a bit of extra code in order to enable backwards compatibility (this extra code is no longer necessary with the .NET Framework 2.0). Calling EnableVisualStyles while running on older operating systems will terminate your program, because your users will be missing uxtheme.dll. You need to test for the newer operating systems before applying themes, as shown here:

Private Shared Sub InitThemes()
  If (((Environment.OSVersion.Platform = PlatformID.Win32NT) _
    AndAlso (Environment.OSVersion.Version.Major >= 5)) _
    AndAlso (Environment.OSVersion.Version.Minor > 0)) Then
      If OSFeature.Feature.IsPresent(OSFeature.Themes) Then
        Application.EnableVisualStyles()
      End If
      Application.DoEvents()
    End If
End Sub

Your main application class also creates the Notify icon and runs your main message loop. Running an application without a main window is almost the same as running with a main window—there are only two changes you need to make. You need to change your call from Application.Run(FormName) to Application.Run() so that you start a message loop without any main form. If the user never creates the main window, you don't need to create it in an invisible state; just don't create it at all. The second change is to handle the SessionEnded event. If a user logs off Windows while your application is running, Windows will request that your application exit by raising the SessionEnded event. If you don't respond to this event, Windows will assume your application just crashed. Windows will tell your users that your application is not responding, and will request to terminate it. To fix this, just handle the event and close your application. Attach the handler in your main application like this:

AddHandler Microsoft.Win32.SystemEvents.SessionEnded, _
    New Microsoft.Win32.SessionEndedEventHandler(AddressOf _
    ApplicationMain.SystemEvents_SessionEnded)
And there clean up properly in that handler:
Private Shared Sub SystemEvents_SessionEnded(ByVal sender As Object, _
    ByVal e As Microsoft.Win32.SessionEndedEventArgs)
    If (Not _mainWindow Is Nothing) Then _mainWindow.Close()
    _icon.Visible = False
    Application.Exit()
End Sub

There are a couple of items in the exit code that are different from your standard Windows Forms application. First, you need to check if the main window was created before you close it. The user may not have even opened the main window. Second, you must hide the Notify icon. Have you ever had a program that ran as a tray icon go away and leave the ghostly tray icon around until you try and activate it? Some developer forgot to clean up the taskbar. You need to hide the Notify icon before your program exits. There is no Close method for the Notify icon; when you hide it, the icon will clean itself up.

Notifying Users

Now it's time to give feedback to your users when they run your app as an icon. You'll do that by creating a pop-up toast window. I don't like the standard balloon notification—I prefer something with style, such as the Outlook message notifications or the MSN Messenger contact sign-in notification. I want to control the notification to inform the user without interfering with regular tasks.

When Outlook 2003 is configured appropriately and you receive an email, a small window fades in at the bottom-right corner of your screen, showing you the sender and the subject of the message. From here you can click to read the message, delete the message immediately, or set the Follow-up flag. If you're busy and the message isn't time critical, you can ignore the pop-up, allowing it to fade out of view, leaving the message unread for later.

Creating a similar window for the weather alert application will make it easier for users to receive weather information updates. The notification window should display itself briefly, and then go away if the user ignores it. Don't create a form of notification that forces the user to dismiss it. If the user is away from his desk, or decides the message is not important enough to disrupt the current task, the notification should just disappear.

Figure 2 Notification Box
Figure 2 Notification Box

I'll place the window in the lower-right corner of the desktop, directly above the notification icons. The toast window should fade in, show itself for a few seconds, then fade out. To make it easier to use, if the user moves the mouse over the window, it will change to opaque immediately, and not fade out until the user leaves the window. If the user wants to open the main application, she can click on the toast, and your application should show the main window. Finally, to add a little style, you can draw the background using a gradient brush, and hide all the usual window borders (see Figure 2). GDI+ makes this fairly simple.Chroma Keying, Alpha Blending, and Transparency

Graphics programmers developed chroma-keyed images to implement the transparency used in cel animation. Cel animations are the classic hand-drawn cartoons in films. An animation studio artist would draw a background scene on a single page. Then an animator would draw each frame for each character on a cel, a transparent sheet of acetate or nitrate. Using a transparent medium saved the animators and artists significant time. Artists drew one background, rather than a separate background for each frame. Then animators could concentrate on drawing the characters without having to worry about their exact position on the background.

Graphics programmers wanted to emulate this technique on screen. Of course, there is no transparency on a computer monitor, so developers designated one solid color to stand in for the transparent background. When drawing these images, called sprites, any pixels in the designated transparent color were skipped. Whatever background was beneath the sprite would still be seen. Artists drew backgrounds the size of the screen or window. Animators drew an animated series of images using the transparent color as the background. Graphics programs would compose the scene by drawing the background image, and then drawing the sprites, using the transparent key.

This technique has a number of limitations. First, transparency was an all or nothing proposition: a pixel was either transparent or not. There was no way to do any kind of translucent effects. Artists had much more work to do in order to create windows, fog, or other similar visual effects. Secondly, the transparent algorithms needed an exact color match, not simply a visual match. A green pixel of (R=0, G=255, B=0) does not match a green pixel of (R=0, G=254, B=0). It was easy to get a sprite that appeared correct, but did not have an exact color match. The result was an ugly sprite with blotches in a color that should have been transparent.

As PC graphics capabilities became more sophisticated, this simple algorithm was enhanced to provide translucency as well as transparency. This technique is called alpha blending. The transparency of a pixel is called the alpha component. The richer pixel formats, such as 32-bit ARGB, now devote one byte to transparency information. The alpha component ranges from 0 for completely transparent to 255 for opaque.

The Microsoft .NET Framework supports both techniques in its drawing library and its Windows Forms library. The System.Drawing.Bitmap class contains support for ARGB images. The MakeTransparent method keys an entire bitmap by modifying the alpha component on each pixel based on a single transparency key color. All the overloads for the System.Drawing.Graphics.DrawImage method obey transparency keying using the alpha component. The System.Windows.Forms.Opacity property specifies how Windows uses alpha blending to draw the form on the screen.

Using GDI+

Creating a transparent, nonrectangular window requires a few simple steps. First, change the Form Border Style in the designer to None. That removes all the normal non-client area objects from your toast window. There's no close box, system menu, or sizing handles. You should also turn off the "Show In Taskbar" setting. You don't want your toast window to show up in the task bar.

You accomplish transparency using a technique known as chroma keying. Chroma-keyed images assign one color as the transparent color. When the window is drawn, any pixel that matches the transparent color is not drawn. It's important to pick a color that is unlikely to be used in a normal application. Obviously, white, black, and grays are not good choices since they're used so frequently. Lime green or chartreuse might be much better. The Windows Form Properties in Visual Studio® .NET lets you set the colors that will be transparent via the TransparencyKey property. See the "Chroma Keying, Alpha Blending, and Transparency" sidebar for more information on chroma keying.

Once you've set the transparency, you need to add the code to paint the background. To implement rounded rectangle functionality with GDI+, you can use a graphics path. In fact, with a graphics path, you can create any irregular shape you want. The code in Figure 3 shows a method used to create a rounded rectangle path.

Figure 3 Creating a Rounded Rectangle Path (Close Figure 3)
Public Shared Function CreateRoundedRectPath(
    ByVal rect As Rectangle, ByVal radius As Integer) As GraphicsPath

    Dim roundRect As New GraphicsPath

    ' top line
    roundRect.AddLine(rect.Left + radius, rect.Top, _
        rect.Right - radius, rect.Top)

     ' Upper-right corner
    roundRect.AddArc(rect.Right - 2 * radius, _
        rect.Top, radius * 2, radius * 2, 270, 90)

    ' right edge
    roundRect.AddLine(rect.Right, rect.Top + radius, _
        rect.Right, rect.Bottom - 10)

    ' bottom-right corner
    roundRect.AddArc(rect.Right - radius * 2, _
        rect.Bottom - radius * 2, radius * 2, radius * 2, 0, 90)

    ' bottom
    roundRect.AddLine(rect.Right - 2 * radius, rect.Bottom, _
        rect.Left + radius, rect.Bottom)

    ' bottom-left edge
    roundRect.AddArc(rect.Left, rect.Bottom - 2 * radius, _
        2 * radius, 2 * radius, 90, 90)

    ' left side
    roundRect.AddLine(rect.Left, rect.Bottom - radius, _
        rect.Left, rect.Top + radius)

    ' upper-left corner
    roundRect.AddArc(rect.Left, rect.Top, 2 * radius, _
        2 * radius, 180, 90)

    Return roundRect
End Function

The background makes use of another new GDI+ feature, gradient brushes. I used a linear gradient brush to create the desired background effect. First, I filled the entire background rectangle with the lime green transparency color, and then I filled the interior rounded rectangle using the linear gradient brush fading from steel blue to light blue. I did all this work in an override for the OnPaintBackground method. Using the overrides can be more efficient than attaching event handlers to relevant events.

To finish the drawing code, you need to add a paint handler that draws the text. It draws a message passed to the toast window in its constructor. Next, you want to set the font to be consistent with the system-defined link color and active link. When the user enters the toast window, change the window font to underlined. When the user leaves the window, change the font back. This gives the user the impression that the text in your toast window is a link.

Fade In and Fade Out

The next step is to add the timer and handlers to update the opacity of the toast window. In the designer, add a timer and set the initial Opacity of the toast window to 0 percent. When the timer event fires, increase the opacity (I choose to increase it by 0.05). When the opacity gets to 1, reset the timer to fire in 3 seconds so the user will have time to see the toast window. Then reset the timer to a small interval, lowering the opacity each time it fires. Once the opacity reaches 0, close the window.

You need to add handlers to stop the fade in/fade out process when the user moves the mouse into the toast window. And, of course, you restart the fade out process when the user leaves the window. There's only one caveat here. The close box (see Figure 2) is a child window. Your main window gets the MouseLeave message when the user moves the mouse over the close box.

In addition to mouse enter and mouse leave methods for the toast window, you'll also need to add event handlers for any child windows in your toast window. While you do get the enter and leave events when the user moves between the close box and the parent toast window, you will never get the MouseLeave message from one window before you get the MouseEnter event for the other. That means you cannot get the state wrong as you move from one window to the other. Your toast window won't go away while the user is interacting with it.

Before you leave the toast window implementation, you need to define how it should behave when you receive multiple updates. In this sample, I took the easy way out: if a second notification should happen while a toast window is being displayed, it's simply ignored. Commercial applications take a different approach. Outlook 2003 creates a queue of toast windows, one for each e-mail message. MSN Messenger stacks toast windows up the screen until the top of the desktop is reached. This sample just doesn't generate enough interesting notifications to make that worthwhile.

Skinning Your Application

Now it's time to apply some style to that main window (see Figure 4). To get the effect, you need to create a background image. Any location that you want to be transparent should be drawn in the transparent color. In my sample, I again chose lime green for the background. Set the background image to the new bitmap, set the form border style to None, and you're nearly finished. There is a problem in some graphics cards that prevents the transparency from working when the bitmap color depth does not match the monitor color depth (see the Knowledge Base article 822495 at BUG: The TransparencyKey Property Is Not Effective for Microsoft Windows Forms If the Color Depth of the Monitor Is Set to a Value That Is Greater Than 24-Bit for more information). To fix this, you need to make the background bitmap transparent. The OnLoad override shown in the following code does just that.

Protected Overrides Sub OnLoad(ByVal e As EventArgs)
    Dim bitmapBG As Bitmap = _CType
(BackgroundImage,Bitmap)
    bitmapBG.MakeTransparent(bitmapBG.GetPixel(0, 0))
    TransparencyKey = bitmapBG.GetPixel(0, 0)
    MyBase.OnLoad(e)
End Sub
After updating the image, you also need to update the transparency key in order to match the changed values of the pixels.

Figure 4 Adding Style to the Main Window
Figure 4 Adding Style to the Main Window

Next, you need to modify the controls so they are transparent and show up on the background better. I chose a yellow font to contrast with the background image. You can modify the background color either in the designer or code, setting it to System.Drawing.Color.Transparent. The background image will then show through your controls (see Figure 5).

Figure 5 Background Image Showing Through
Figure 5 Background Image Showing Through

Finally, it helps your users to add a mechanism to move the main window around the screen. Since there is no longer a title bar, you can let the user move the window by pressing the left mouse button anywhere in the visible form and dragging the window from place to place. The MouseMove method moves the window if the user is dragging it:

Protected Overrides Sub OnMouseMove(ByVal e As MouseEventArgs)
    If _isMoving Then
        Location = New Point(Location.X + e.X - XOffset,
            Location.Y + e.Y - YOffset)
    End If
    MyBase.OnMouseMove(e)
End Sub
That works quite well, unless the user happens to click the mouse in one of those child windows you've created to display the weather information. To make that work, you need to create a mouse down handler that initiates a move operation when the user clicks a mouse button when over one of your child windows. But it should not work on all your child windows—you still want the default behavior for the link buttons and the close box. The same method can handle all the MouseDown events for any of the other child controls, as shown in the following:
Private Sub Child_MouseDown(ByVal sender As Object, _
    ByVal e As System.Windows.Forms.MouseEventArgs) Handles _
        labelAirPressure.MouseDown, labelDewPoint.MouseDown, _
        labelHumidity.MouseDown, labelLocation.MouseDown, _
        labelSkyConditions.MouseDown, labelTemperature.MouseDown, _
        labelVisibility.MouseDown, labelWind.MouseDown
    If (e.Button = MouseButtons.Left) Then
       _isMoving = True
       XOffset = e.X
       YOffset = e.Y
    End If
End Sub
Add the close box and the link buttons to get the latest weather information and you're finished.

Conclusion

When you are creating applications that need to interact with the user in different ways, make them stand out, but don't overdo it. The standard look is familiar to users, so if you break that mold, do it with good reason. Too many applications take up space on the system tray, so don't do that unless it really is the best way for your application to interact with users. If it's not, use a different method to make your application's presence known.

Bill Wagner is a cofounder of SRT Solutions, Inc. and is Microsoft Regional Director for Michigan. Bill is the author of Effective C# (Addison-Wesley, 2004). Reach him at wwagner@srtsolutions.com, or his blog at www.srtsolutions.com/public/blog/20574.


Editor's Note: Becoming a Guru

Editor's Note
Becoming a Guru

Here in the pages of MSDN Magazine, we cover a lot of topics that developers can use every day to further their careers. This month, however, we'll look at a more basic question: how does an average developer win fame and recognition as a guru on the programming scene?

Twenty years ago, the field was a lot smaller and more concentrated. Right before Windows 3.0 was released, word was that the Windows 2.11 SDK had been received by about 40,000 programmers. We don't know whether this was actual fact, but we do know that when Windows 3.0 hit the shelves, the dev community blossomed. Soon there were all sorts of utilities for Windows popping up on BBSs. (Note to youngsters: before the Web, people would dial up to someone's computer and connect to a "bulletin board" to browse and download files.)

To make a long story short, today there are literally millions of people who are targeting Windows when they sit down to code. Still, there's a lot of opportunity for widespread recognition. Every author in this magazine started small, with their first submission.

If you want to make a name for yourself in the world of programming, the first commandment is to know exactly what you're doing. Nothing will make people ignore you faster than if they can't trust your work. One thing you'll find in the community of programmers is that people can smell B.S. a mile away. Software itself is really nothing more than the end product of a bunch of thoughts put together, and if the software isn't sound, it's a sure sign that the underlying ideas aren't either.

The second commandment is to have an idea. This sounds simple, but it's the toughest thing to do. If you want to demonstrate a concept, it should be unique and it should fill a need. We see submissions all the time from people who want to show how to create a form or how to write a simple Web service—these ideas are well-trod by now.

Most great ideas are born of necessity. People have written RAW file viewers for their cameras, mobile blogging tools, and all sorts of power toys for Visual Studio. If you get something started, you don't even need to perfect it yourself; put it up in a shared workspace on gotdotnet.com or elsewhere, and invite others to add to the project you own.

The third commandment is that you must express yourself clearly, both through your writing and your code. A bad cover letter, with spelling and grammatical errors, severely limits an editor's confidence in the care you take in the rest of your work.

Finally, the fourth commandment is to keep your options open. A magazine is one place to get started, but if you join the expanding community of dev bloggers, you won't have to wait for acceptance before you start sharing your ideas.

Think of your favorite source of development information. Is it a magazine? A blog? A column? A book? A chat list? What do you get from this source that's unique? There's no need to reinvent the wheel when you're getting started. If you have something to share, do it through your favorite source. Even if you're not an MVP, you can offer your expertise, creativity, and guidance.

There are other ways to get exposure as well. You can enter a coding challenge, like our recently completed Connected Developer competition. You might be invited to speak at a conference, although this requires that you not only have your thoughts in order but can present them well in front of a room full of people. It's both easier than it sounds (imagine them all wearing pizza-stained Tech•Ed 97 T-shirts) and harder (imagine them all walking out five minutes into your talk).

Of course, we can't guarantee that fame and recognition will follow you if you look for it, but more than in most professions, in software development, talent is rewarded because the word about who has the talent really gets around. If you want to be discovered, remember our four simple rules: know what you're doing, have a unique idea, express yourself clearly, and look for opportunities in a variety of places. Even if you really want to become known as a guru, it doesn't come overnight. You'll have to work hard like the rest of us gurus did!

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Joe Binder, Shawn Burke, Simon Calvert, Jonathan Caves, Andy Cheung, Manish Godse, Manish Jayaswal, Dale Koetke, Ronald Laeremans, Paul Leach, Gerrard Lindsay, Martyn Lovell, Brian McMaster, J.D. Meier, Shaun Miller, Vivek Narasayya, Jeff Peil, Andres Sanabria, Michael Stuart, Maura Van Der Linden, and Tyler Whitney.

Active Directory, ActiveX, BizTalk, JScript, Microsoft, MSDN, MSN, Outlook, SharePoint, Visual Basic, Visual C++, Visual Studio, Windows, Win32, Windows NT, Windows Media, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Marnie Hutcheson

Contents
Create Animated Graphical Controls
Create and Manage Icons
Print Source Code in Color
Encrypt Your Data Securely
The Bookshelf

Create Animated Graphical Controls

From my first look at VG.net V2.4b by Prodige Software Corporation, it seemed like just another typical graphics add-on, but as I discovered what a potent tool it is for Visual Studio® .NET graphical interface development, I became more and more impressed. The VG.net Graphical Designer integrates into the Visual Studio .NET ToolBox to create graphical components for your Windows® Forms applications. You also get a special drawing toolbar for manipulating graphics shapes. The free VG.net runtime, Prodige.Drawing.dll, powers these vector-based graphics objects and their animations. You just redistribute it with your programs.

A VG.net Picture is a .NET-based class containing vector graphics Elements. You can create several types of gauges or buttons and reuse them in different Pictures that appear in Canvases in different Windows Forms. Since these are vector-based graphics, they paint fast and are never pixelated, no matter how close you zoom in. Which brings me to the next cool thing that you can do with VG.net without writing any code—zoom.

VG.net uses floating point coordinates and transformations that allow you to zoom in as you create your graphics so you can see every little detail. It even lets your user interface scale and translate at run time. You just set the docking property for the Canvas to "Fill" the form, then set the AutoSizePicture property to True, and presto, your picture and all its sub-pictures automatically rescale to fit the current window size, no matter how big or small it becomes. Even the text scales perfectly and stays smooth.

VG.net graphical Elements support true transparency. You can change the transparency property for fills (most impressively for gradients), strokes, and text dynamically with no unfortunate surprises.

Events bubble up from Elements to groups, to sub-pictures, and so on. Graphics properties and styles cascade down, so you have lots of flexibility to animate as you process events. Properties in Styles and Elements are combined during the display pass. You can set properties on the child that are different from the parent; the child properties override the properties on parents.

You can change the fill of each graphic by modifying its Fill property, but it is easier to use Styles because you can change the appearance of the graphics at run time by swapping Style references, which is great for animating mouseovers and click events. A Style defined at any Picture level in a hierarchy is available to all descendents. An Element several levels deep can use a Style by setting the Element Style property equal to the Style Name. VG.net Designer also lets you create server controls that will render your graphic objects into bitmap images on the fly for your Web applications.

The VG.net help file integrates with Visual Studio .NET help and there are lots of samples both on the Web site and in the samples folder that is included with the installation. Getting started is as simple as adding a new item to your project and selecting VG.net Picture from the Add New Item list.

Price: $550 per single-user license.

www.vgdotnet.com

Create and Manage Icons

ArtIcons 4.15 by Aha-Soft is a Windows-based icon maker with features that let you create, import, and export great-looking icons, cursors, animated mouse cursors, and other small graphics. You can import existing icons (even from the Mac) and images in many formats. The drawing surface, the tools, and the preview pane are all conveniently located in the window.

The ArtIcons image pixel editor is stocked with some really nice tools and effects, like shadow, opacity, smoothness, negative, grayscale colorize, hue, saturation, and color replacement. I really liked drawing with the gradient fill. You can design multilayer icons either in standard or custom sizes with the color depth up to 16 million colors.

Once you have an icon, you can create an entire set of sizes and color depths in just a few mouse clicks (see the preview pane in the accompanying screen shot).

Price: $39.95 for professional single-user license.

www.aha-soft.com

Print Source Code in Color

VS.NETcodePrint 2003—Developer Suite by StarPrint makes your code the prettiest and most readable on the block. It lets you print Visual Basic®, C#, J#, and ASP.NET source code in color and export output to RTF, PDF, and HTML.

When you combine source code indention, line numbering, and lines that delimit language statement constructs with user-selectable color and font attributes, your code gets a lot more readable. This is great for maintenance and code reviews, too.

You can preview, print, and export a complete solution, selected projects, project items, namespaces, classes, modules, and procedures. You can customize just about everything—the font attributes and colors for comments, identifiers, keywords, line numbers, strings, procedure headings, table of contents, and page headers and footers.

The preview screens let you see the formatted source code with unlimited zooming, multi-page thumbnails, and side-by-side pages. You have control over page layout, including paper source, orientation, line spacing, borders, margins, headers, and footers. You can even print in multiple columns.

Price: $89 per single-user license.

www.starprint2000.com

Encrypt Your Data Securely

Private Disk 2.05 by Dekart creates encrypted partitions that show up as drive letters on your FDD, CD, CD/R, CD/RW, MO, MD, ZIP disks, flash drives, all types of flash memory cards, handhelds, and digital cameras. Files saved to these partitions are automatically encrypted using AES 256-bit encryption. You can work with this protected data on any computer without installing any additional software; Private Disk takes care of all the details.

You can manage multiple disks and work with your encrypted files in Windows Explorer. Private Disk 2.05 lets you launch specific programs automatically when the disk is mounted. And, in case you forget to lock your PC when you leave it, the disk will disconnect by itself if you leave it inactive for a predefined period of time. The disk can save and automatically restore network shares.

Price: $45 per single-user license.

www.dekart.com

The Bookshelf

Developing Business Intelligence Solutions Using Information Bridge and Visual Studio .NET by James Henry (BlueVision, 2005) introduces the Microsoft Information Bridge and walks you through the process of architecting, implementing, deploying, and securing Business Intelligence solutions. It's a must read for developers thinking about implementing an Information Bridge business solution.

Henry contends that a document should have live links to all the information the reader needs to make an informed decision. For example, if you receive a memo about a customer who is thinking about canceling their account, there could be a smart tag leading to their account number. When you click on it, you will see their latest metadata, which can include information like order history, payment history, customer summary, and so on.

Creating and linking to Web services that grab the data you want, defining the metadata and the actions that the users will be able to perform using the Visual Studio metadata designer plug-in, and programming the actions of the smart tags is an involved process. This book covers it all.

Price: $49.99, 376 pages.

All prices were confirmed at press time and are subject to change.

Send your New Stuff to newstuff@microsoft.com.

Marnie Hutcheson is president of Internet Development Associates (Ideva), a firm in Ocala, Florida, that specializes in Internet and intranet Web application design and development. She has published technical papers and books on various computing topics. You can reach her at marnie@ideva.com.


Web Q&amp;A: Smart Navigation, ASP.NET Project Structure, and More

Web Q&A
Smart Navigation, ASP.NET Project Structure, and More
Edited by Nancy Michell

Q I'm getting intermittent errors in Microsoft Internet Explorer, and they seem to be related to using ASP.NET 2.0 with SmartNavigation turned on. What exactly does SmartNavigation do and when should I use it?

A SmartNavigation is a property of the Page class in System.Web.UI. When a request comes in to Internet Explorer 5.5 or higher and SmartNavigation is turned on (set to true), the following actions are performed:

  • The flash caused by navigation is eliminated
  • The scroll position is persisted when moving from page to page
  • Element focus is persisted between navigations
  • Only the last page state in the browser's history is retained

However, according to the Microsoft® .NET Framework Class Library documentation, smart navigation is best used only with ASP.NET pages that require frequent postbacks but have visual content that does not change dramatically on return.

In most circumstances, you should not set this property in code. Set the SmartNavigation attribute to true in the @ Page directive in the .aspx file. When the page is requested, the dynamically generated class sets this property for you. Smart navigation is a feature that you must test in your particular scenarios to be sure the proper behavior is exhibited.

Q I am experiencing some problems adopting ASP.NET 2.0, mainly because I need to compartmentalize functionality within a large Web application. There will be number of feature compartments under the root virtual directory, like so:

Internet Bank (virtual directory)
 Accounts (folders)
 ApplicationForms (folders)
 Investments (folders)

The subdirectories cannot be virtual directories as all of the areas need to share session state which is scoped to virtual directories. With ASP.NET 2.0 all of these extraneous code files have to be placed in App_Code in order to be compiled, making it really hard to understand what files in App_Code are used by what functional areas. Can you suggest an easy solution to this?

A Your UI codebehind code lives with the .aspx or .ascx file—so it will live side-by-side with the actual content. It's typically recommended that you encapsulate all per-UI form code in these files (similar to how it's done today with Visual Studio® .NET 2003). App_Code is then used for shared code across the application.

You could organize the shared code using subfolders underneath the App_Code directory. Then use regular folders to cleanly organize your pages/user-controls across the project and keep all codebehind code with it (the default). See Figure 1 for an example.

Figure 1 Project Folders (Close Figure 1)
App_Code
     \Accounts
        MyAccounts.cs
    \ApplicationForms
        MyApplicationForm.cs
    \Investments
        MyInvestment.cs

\Accounts
    MyAccountsPage.aspx
    MyAccountsPage.aspx.cs

\ApplicationForms
    MyApplicationFormPage.aspx
    MyApplicationFormPage.aspx.cs

    \Investments
        MyInvestmentPage.aspx
        MyInvestmentPage.aspx.cs

Otherwise, rather than use App_Code and have a single place where all shared code is stored, partition it into three separate class-library projects that the Web project links to. These would then be deployed as DLLs in the \bin directory.

Q I am building a Web service in C# that deals with a global Hashtable that is accessed by many users simultaneously. Is it safe to have multiple people reading the Hashtable without a lock while other threads are writing to it? Do I need a lock every time I read something from the Hashtable?

A If a writer adds a value that causes the basic structure of the Hashtable to redistribute or that causes buckets to be added, you will run into trouble. You should use the ReaderWriterLock to achieve thread safety. Also note that if you intend to use IEnumerable to enumerate the items in the Hashtable, an exception will be thrown if the Hashtable changes between calls to MoveNext.

For the .NET Framework 2.0, the docs for Hashtable say it's safe for multiple readers and a single writer. The page at Hashtable.Synchronized Method states that "A System.Collections.Hashtable can support one writer and multiple readers concurrently."

Q I have found that when catching the oncontextmenu event to create a pop-up window, the pop-up is being blocked by the Internet Explorer pop-up blocker in Windows XP SP2, but only in the Internet zone. It was my understanding that any user-initiated pop-up should not be blocked. Is this by design? Here's the code I'm using:

<html><body>
<a oncontextmenu="showModalDialog('about:blank');return false">
    <b>Right</b> click here to invoke a modal dialog.
</a>
<a oncontextmenu="window.open('about:blank');    return false">
    <b>Right</b> click here to invoke a window.open.
</a>
</body></html>

A WM_CONTEXTMENU is not considered to be a user-initiated action. You could display a floating div or use window.createPopup instead of HTML dialogs or window.open. The reason this only occurs in the Internet zone is that the pop-up blocker is only enabled by default in Internet and untrusted zones.

The rules concerning pop-ups say that they are not blocked if they are:

  • Opened by a link which the user clicked.
  • Opened by applications that are running on the computer. These applications can include adware.
  • Opened, on user-initiated actions, by ActiveX® controls that are instantiated from a Web site.
  • Opened from the Trusted Sites or Local Intranet zones.
  • Launched from a Web site on a user's allow list. (Users can add Web sites to their allow list through the pop-up blocker settings.)
  • Launched with the window.createPopup method. Note that window restrictions allow only one pop-up window at a time to be open on a page, and that they restrict pop-up windows launched with this method to the Internet Explorer client window.

Whether pop-up windows appear in each of these cases is ultimately determined by the user. Users can configure the pop-up blocker to be more or less restrictive, or they can choose to turn it off altogether.

For more information, see "About the Pop-up Blocker".

Q Is there a setting to disable the address bar in Internet Explorer via Group Policy? What other browser settings can I enforce through policy?

A Yes, you can disable the address bar and set lots of other restrictions using Group Policy. The browser restrictions that are available with Internet Explorer 6.0 SP1 are shown in Figure 2. The instructions for setting the browser restrictions can be found in the Knowledge Base article located at The restrictions that are available to Internet Explorer 6.0 SP1.

Figure 2 Browser Restrictions (Close Figure 2)

Setting | Description
NoFileOpen | Turns off the Open command on the File menu.
NoFileNew | Turns off the New command on the File menu.
NoBrowserOptions | Turns off the Internet Options command on the Tools menu.
NoSelectDownloadDir | Turns off Save on the File Download dialog box.
NoBrowserClose | Turns off Close button, located in the upper-right corner of the window or dialog box. This registry value also turns off the Close command on the File menu.
NoBrowserSaveAs | Removes the Save As command on the File menu.
NoFavorites | Removes the Favorites menu and disables the Favorites bar.
NoHelpItemTipOfTheDay | Removes the Tip of the Day command on the Help menu.
NoHelpItemNetscapeHelp | Removes the For Netscape Users command on the Help menu.
NoHelpItemTutorial | Removes the Internet Explorer Tutorial command on the Help menu.
NoHelpItemSendFeedback | Removes the Send Feedback command on the Help menu.
NoPrinting | Removes the Print command on the File menu.
No_LaunchMediaBar | Removes the Media Bar command on the toolbar when the user plays online content.
NoTheaterMode | Disables the F11 key (full-screen mode).
NoBrowserContextMenu | Disables the right-click shortcut menu on a Web page.
NoFindFiles | Turns off the F3 command and the Search bar.
NoViewSource | Turns off the Source command on the View menu. However, to completely disable user's ability to view the source, the NoBrowserContextMenu value must be added.
NoOpeninNewWnd | Turns off the CTRL+N command and the Open command to open a new window.
AlwaysPromptWhenDownload | Makes the "Always ask before opening this type of file" checkbox on the Safe Open File dialog box unavailable so that it is always selected.
NoNavButtons | Disables the Back and Forward navigation buttons on the toolbar.
NoHelpMenu | Disables the Internet Explorer Help menu.
No_MediaBarOnlineContent | Disallows the playing of online content.
NoToolbarOptions | The user receives the error message when they try to click any command on the View menu except Customize and Lock the Toolbars.
NoToolBar | Removes the toolbar. However you can add standard buttons on the View Toolbar menu. To completely enable this setting, you must enable the NoToolbarOptions DWORD value.
NoAddressBar | Removes the Address bar.
NoLinksBar | Removes the Link bar. The user receives an error message when they try to click Links on the Toolbar menu.

Q I'm using an application (developed in Visual Basic) in a Citrix environment. Each time you open a Web link, it is supposed to open in the same Internet Explorer window. This works fine in the old production environment, but in our new Citrix farm a new Internet Explorer window opens each time. How can I work around this without touching the Visual Basic application?

A You should read Knowledge Base article 837247, named, appropriately enough, "A link that has a specified target always opens in a new window if Internet Explorer is the startup program in a Terminal Services session or the shell for the operating system". The solution to the problem involves downloading and installing the hotfix at You cannot gain access to linked Web pages from a Web site that you access through a desktop shortcut in Internet Explorer Service Pack 1 and editing the registry.

Q I have read that if you have a page that does not write to session state you should set the EnableSessionState attribute of the Page directive to ReadOnly for improved performance. Is this true and why? How can I view this performance impact?

A Consider the following two processes. You issue a GET request for a page with read/write access to session state. You grab the session state object, deserializing as necessary. Then you process the page. Next, you write back to the session state object, serializing it as necessary, and return the response to the client.

On the other hand, when you have read-only session state, you issue a GET request for the page, get the session state object, and deserialize it if necessary. Then you process the page and return the response to the client. Therefore, you are saving the writing back steps and also the reserializing steps.

There is also a lock created around the session object when you access it as read/write. This lock is not created with a read-only session. The lock is a session-state-level lock and is held for the entire time the page is processed and until the session state is written back to the server. There is normally not a lot of contention for this lock; however, if you utilize frames you will see the contention start to climb. Once a request detects that it cannot gain an exclusive lock on the session state it goes into a polling process which will poll for the lock every 500 ms. The lock is not used when multiple read-only requests are made for the same session state.

Send your questions and comments to webqa@microsoft.com.

Thanks to the following developers at Microsoft for their technical expertise: Mark Atherton, Earl Beaman, Ryan Byington, Todd Carter, Doug Cook, Jeff Davis, Mark Flick, Scott Guthrie, Brett Hill, Keith Homiski, Frank Isernhagen, Darren Jefford, Andreas Klein, Jamie Laflen, Francisco Javier Banos Lemoine, Ajay Mansata, Aditi Maheshwari, Ian Parramore, Gang Peng, Shahar Prish, Ahmad Safa, Mike Skovrinskie, Sean Sutherland, Niall Waller , Brad Wilson, and Radomir Zaric.


Cutting Edge: ASP.NET Forms

Cutting Edge
ASP.NET Forms
Dino Esposito

Contents
Rendering Forms
The HtmlForm Class
Managing Multiple Forms
Multiple <form> Tags
Cross-Page Posting
Redirecting Users to Another Page
Conclusion

Forms are an essential piece of ASP.NET—the ASP.NET Web programming model itself wouldn't be possible without forms. The use of forms is not constrained in pure HTML, but it is subject to some restrictions in ASP.NET. In ASP.NET pages, a single form can post to itself, and the model provides for generalized control state management and postback events. Writing ASP.NET apps is easy and effective because of the single form model.

The restrictions on forms in ASP.NET may sound weird and arbitrary at first, but actually the model is quite straightforward to work with. However, there is one real-world scenario that the ASP.NET 1.x form model doesn't address: having multiple, highly specialized forms in the same page. For example, you can't have a search box that posts to a different page.

In the May 2003 issue of MSDN®Magazine, I wrote a column about forms programming in ASP.NET 1.x (see Cutting Edge: Form-based Programming in ASP.NET. With the introduction of ASP.NET 2.0, a few things have changed that specifically address the theme of posting to different pages. This month I'll discuss forms programming in ASP.NET 2.0.

Rendering Forms

Let's start exploring the universe of forms in ASP.NET with some consideration of how forms (and contained controls) are actually rendered. In an ASP.NET page, the <form> tag can be the child of various container controls such as <table>, <div>, or <body>; however, in most pages it is simply the child of <body>. If a non-container control (TextBox, for example) is placed outside the form tag, an HttpException is thrown at run time (no check is made for this at compile time). Take a look at the following code, which is excerpted from the TextBox's AddAttributesToRender method:

protected override void AddAttributesToRender(HtmlTextWriter writer)
{
    if (this.Page != null)  this.Page.VerifyRenderingInServerForm(this);
    ...
}
A call to Page's VerifyRenderingInServerForm method does the job. (Be aware of this behavior when you write your own custom server controls.)

The HtmlForm Class

HtmlForm inherits from HtmlContainerControl, which provides the form with the capability of containing child controls. HtmlForm provides programmatic access to the HTML <form> element on the server through the set of properties listed in Figure 1. As you can see, the interface changes in HtmlForm between ASP.NET 1.x and ASP.NET 2.0 are limited to a few properties.

Figure 1 Properties of the HtmlForm Class (Close Figure 1)

Property | ASP.NET 1.x | ASP.NET 2.0 | Description
Attributes |  |  | Inherited from Control. Gets a name/value collection with all the attributes declared on the tag.
ClientID |  |  | Inherited from Control. Gets the value of UniqueID.
Controls |  |  | Inherited from Control. Gets a collection object that represents the child controls of the form.
DefaultButton |  |  | String property. Gets or sets the button control to display as the default button on the form.
DefaultFocus |  |  | String property. Gets or sets the button control to give input focus when the form is displayed.
Disabled |  |  | Gets or sets a value indicating whether the form is disabled. Matches the disabled HTML attribute.
EncType |  |  | Gets or sets the encoding type. Matches the enctype HTML attribute.
ID |  |  | Inherited from Control. Gets or sets the programmatic identifier of the form.
InnerHtml |  |  | Inherited from HtmlContainerControl. Gets or sets the markup content found between the opening and closing tags of the form.
InnerText |  |  | Inherited from HtmlContainerControl. Gets or sets the text between the opening and closing tags of the form.
Method |  |  | Gets or sets a value that indicates how a browser posts form data to the server. The default value is POST. Can be set to GET if needed.
Name |  |  | Gets the value of UniqueID.
Style |  |  | Gets a collection of all cascading style sheet (CSS) properties applied to the form.
SubmitDisabledControls |  |  | Indicates whether to force controls disabled on the client to submit their values, allowing them to preserve their values after the page posts back to the server. False by default.
TagName |  |  | Returns "form".
Target |  |  | Gets or sets the name of the frame or window to render the HTML generated for the page.
UniqueID |  |  | Inherited from Control. Gets the unique, fully qualified name of the form.
Visible |  |  | Gets or sets a value that indicates whether the form is rendered. If false, the form is not rendered to HTML.

A form must have a unique name. One is assigned automatically by ASP.NET if you don't specify one. You can set the form's identifier by using either the ID or Name property. If both are set, the ID attribute takes precedence. It is important to note, though, that any programmatic use of the Name attribute compromises the XHTML compliance of the page. In XHTML, elements are identified by ID and not by Name. So generally speaking, you're better off relying on the ID property.

The parent object of the form is the outer container control that has the runat attribute set. If such a control doesn't exist, the page object is set as the parent. Typical containers for the server form are <table> or <div> if they are marked as server-side objects.

Figure 2 lists some of the methods available on the HtmlForm class that you'll be using most often. All the methods listed in the table are inherited from the base System.Web.UI.Control class. Note that the FindControl method searches only among the form's direct children. Controls belonging to an inner container or a child of a form's child control are not found.

Figure 2 Methods of the HtmlForm Class (Close Figure 2)

Method | ASP.NET 1.x | ASP.NET 2.0 | Description
ApplyStyleSheetSkin |  |  | Applies the skin in the defined page theme in the manner defined by StyleSheetTheme.
DataBind |  |  | Calls the DataBind method on all child controls.
FindControl |  |  | Retrieves and returns the control that matches the specified ID.
Focus |  |  | Sets the input focus to a control.
HasControls |  |  | Indicates whether the form contains any child controls.
RenderControl |  |  | Outputs the HTML code for the form. If tracing is enabled, caches tracing information to be rendered later, at the end of the page.

Managing Multiple Forms

Generally, embracing the single-form model and renouncing system support for multiple forms is not a big sacrifice. Some pages, though, would have a more consistent and natural design if they could define multiple forms—at least, forms with logically related groups of input controls. For example, think of a page that provides some information to users, but also needs to supply an additional form containing a search or a login box.

You could incorporate search and login functionalities in ad hoc classes and call those classes from within the page displayed to the user. This is not necessarily the best way to factor out your code, however. If you're porting some old code to ASP.NET, you might find it easier to insulate login or search code within a dedicated page. But how do you force the page to post data to this page?

In the single-form model, each page always posts to itself and doesn't supply a hook for developers to set the final destination of the postback. In HTML and ASP programming, the Action property of the form is simply not exposed on the ASP.NET HtmlForm class. The single-form model is so closely integrated with the ASP.NET platform that you can only take or leave it—or, as an additional choice, code the old ASP way without server forms. As you'll see later, in ASP.NET 2.0, posting data to a different page is possible, but the implementation of the feature passes through some new capabilities of button controls. For now, let's see what is involved in using additional HTML non-server forms.

In ASP.NET, an exception is thrown if multiple HtmlForm controls are being rendered. When the first HtmlForm control in the page is rendered, a Boolean flag is flipped to true. The flag thus indicates whether any other HtmlForm control has been rendered. When another HtmlForm attempts to render, the flag will already be true causing an exception to be thrown.

Nothing bad happens if the Web Form contains one server form and any number of <form> tags devoid of the runat attribute. Without the runat attribute, any tags become pure and simple HTML markup and are rendered verbatim (see Figure 3).

Figure 3 Server and Client Forms in the Same Page (Close Figure 3)
<html>
<body>
    <table><tr><td>
        <form id="form1" runat="server">
        <h2>Ordinary contents for an ASP.NET page</h2>
        </form>
    </td>

    <td>
        <form method="post" action="search.aspx">
            <table><tr>
                <td>Keyword</td>
                <td><input type="text" id="Keyword" name="Keyword" /></td>
            </tr><tr>
                <td><input type="submit" id="Go" value="Search" /></td>
            </tr></table>
        </form>
    </td>
    </tr></table>
</body>
</html>

The page contains two forms, the second of which is an HTML form and lacks the runat="server" attribute. As such, it is completely ignored by ASP.NET. The markup served to the browser legally contains two <form> elements, each pointing to a different action URL.

Although functional, this code has a major drawback: you can't use the ASP.NET programming model to retrieve posted data in the action page of the client form. When writing search.aspx, the action page for the HTML client form, you can't rely on page controls that read and update their state from view state and posted values. (The apparent statefulness of ASP.NET server controls is obtained by making pages post to themselves.) To know what's been posted to search.aspx, you must resort to the old-fashioned yet effective ASP model of looking directly into the collection of posted data:

protected void Page_Load(object sender, EventArgs e)
{
    // Use the Request to retrieve posted data
    string textToSearch = Request.Form["Keyword"].ToString();
    ...
    // Use standard ASP.NET to populate the page UI
    KeywordBeingUsed.Text = textToSearch;
}

You use the protocol-specific collections of the HttpRequest object (available from Page.Request as well as HttpContext.Current.Request) to retrieve posted data—Form if POST is used, QueryString if GET is used, or Params if you want combined access to Form, QueryString, ServerVariables, and Cookies. The HttpRequest object is populated with data before the page is created, so any call to Page.Request works from any of the page's events. In self-posting ASP.NET pages, you don't need to use Request because you can rely on a strongly typed programming model, but the old, faithful HttpRequest object is still there for you to use if you need it.

It is also interesting to note that when the user clicks on the Search button, the search.aspx page is invoked, and it receives only the values posted through the HTML form. No view state is posted, and no extra data is passed. If you have to post to another page, the old-fashioned approach is still going to be the most effective performance-wise. As you'll see later in this column, the cross-page posting feature of ASP.NET 2.0 still moves around a fairly large, view-state-like field.

Multiple <form> Tags

If multiple server forms are declared in the same Web Form, an exception is thrown. Not so obvious, and not very well known, is that a Web Form can actually contain as many server-side forms as needed, so long as only one is visible and rendered at a time. For example, a page with three <form> tags marked runat="server" is allowed, but only one form's Visible property can be set to true. By playing with the Visible property of the HtmlForm class, you can change the active server form during the page's lifetime. This little trick doesn't really solve the problem of having multiple active forms, but it can still be helpful at times.

Let's consider the page in Figure 4. As you can see, all <form> tags are marked runat="server", but only the first is visible. Mutually exclusive forms are great for implementing wizards in ASP.NET 1.x. By toggling the form's visibility in button event handlers, you can obtain a wizard-like behavior, as depicted in Figure 5.

Figure 4 Markup for an ASP.NET 1.x Wizard-Like Page (Close Figure 4)
<html>
<body>
    <form id="step0" runat="server" visible="true">
        <h1>Welcome</h1>
        <asp:textbox runat="server" id="Textbox1" />
        <asp:button ID="Button1" runat="server" text="Step #1"
            OnClick="Button1_Click" />
     </form>

     <form id="step1" runat="server" visible="false">
        <h1>Step #1</h1>
        <asp:textbox runat="server" id="Textbox2" />
        <asp:button ID="Button2" runat="server" text="Previous step"
            OnClick="Button2_Click" />
        <asp:button ID="Button3" runat="server" text="Step #2"
            OnClick="Button3_Click" />
    </form>

    <form id="step2" runat="server" visible="false">
        <h1>Finalizing</h1>
        <asp:button ID="Button4" runat="server" text="Finish"
            OnClick="Button4_Click" />
    </form>
</body>
</html>
Figure 5 Wizard-Like Page in Action
Figure 5 Wizard-Like Page in Action

The trick is rather useless in ASP.NET 2.0 because you find two new controls, MultiView and Wizard, ready for the job. The MultiView control employs logic nearly identical to multiple exclusive forms, except that it relies on panels rather than full-fledged forms. MultiView allows you to define multiple and mutually exclusive HTML panels. The control provides an API for you to toggle the visibility of the various panels and ensure that exactly one is active and visible at a time. The MultiView control doesn't provide a built-in user interface. The Wizard control is just a MultiView plus some wizard-like predefined UI blocks. I covered the Wizard control in the November 2004 issue of MSDN Magazine (see Cutting Edge: The ASP.NET 2.0 Wizard Control).

Cross-Page Posting

ASP.NET 2.0 offers a new built-in mechanism to override the normal processing cycle and to allow a page to post back to another page. In general, postbacks occur in one of two ways: through a Submit button or through a script. Submissions through a button typically point automatically to the address that the posting form indicates. More flexibility is possible when the post occurs through a script. In ASP.NET 2.0, you can configure certain page controls—in particular, those that implement the new IButtonControl interface—to post to a different target page. This is referred to as cross-page posting.

Core controls that implement the IButtonControl interface are Button, ImageButton, and LinkButton. In general, by implementing IButtonControl, any custom control can act like a button on a form. The IButtonControl interface is a clear example of the refactoring process that ASP.NET went through in the transition from version 1.x to 2.0. The IButtonControl interface now groups a few properties that most button controls (including some HTML button controls) support since ASP.NET 1.x. In addition, a few new properties heralding new functionality have been added, like PostBackUrl and ValidationGroup. Figure 6 details the IButtonControl interface. The following code snippet shows how to proceed:

<form runat="server">
    <asp:textbox runat="server" id="Data" />
    <asp:button runat="server" id="buttonPost" Text="Click"
        PostBackUrl="target.aspx" />
</form>

Figure 6 IButtonControl Interface (Close Figure 6)

Property | Description
CausesValidation | Boolean value indicating whether validation is performed when the control is clicked.
CommandArgument | Gets or sets an optional parameter passed to the button's Command event along with the associated CommandName.
CommandName | Gets or sets the command name associated with the button that is passed to the Command event.
PostBackUrl | Indicates the URL that will handle the postback triggered through the button control.
Text | Gets or sets the caption of the button.
ValidationGroup | Gets or sets the name of the validation group to which the button belongs.
Visible | Boolean value indicating whether the button control is rendered.

When the PostBackUrl property is set, the ASP.NET runtime binds the corresponding HTML element of the button control to a new JavaScript function. Instead of the __doPostback function normally used, it uses the new WebForm_DoPostBackWithOptions function. The button defined previously renders the following markup:

<input type="submit" name="buttonPost" id="buttonPost" value="Click"
     onclick="javascript:WebForm_DoPostBackWithOptions(
         new WebForm_PostBackOptions("buttonPost", "",
         false, "", "target.aspx", false, false))" />

As a result, when the user clicks the button, the current form posts its content to the specified target page. What about the view state? When the page contains a control that does cross-page posting, a new hidden field named __PREVIOUSPAGE is created. The field contains information about the posting page. The target page uses this information in order to build a stateful reference to the calling page object.

In the target page, you use the PreviousPage property, a new property on the Page class, to reference the posting page and all of its controls. Here's the codebehind for a sample target page that retrieves the content of a TextBox defined in the form:

protected void Page_Load(object sender, EventArgs e)
{
    // Retrieves some posted data
    TextBox txt = (TextBox) PreviousPage.FindControl("TextBox1");
    ...
}

By using the PreviousPage property on the Page class, you can access any input control defined on the posting page. Access to input controls is weakly typed and occurs indirectly through the FindControl method. The problem here lies in the fact that the target page doesn't know anything about the type of the posting page. PreviousPage is declared as a property of type Page. As such, it can't provide access to members specific to a derived page class.

Furthermore, note that FindControl looks up controls only in the current container. If the control you are looking for lives inside another control (say, a template), you must first get a reference to the container and then search the container to find the control. To avoid using FindControl altogether, a different approach is required.

To retrieve values on the posting page, FindControl represents your only safe option if you don't know in advance which page will be invoking your target. However, when using cross-page posting in the context of an application, there's a good chance that you know exactly who will be calling the page and how. In this case, you can take advantage of the PreviousPageType directive to cause the target page's PreviousPage property to be typed to the source page class. In the target page, you add the following directive:

<%@ PreviousPageType VirtualPath="crosspostpage.aspx" %>

The directive can accept either of two mutually exclusive attributes—VirtualPath or TypeName. VirtualPath points to the URL of the posting page. TypeName indicates the type of the calling page. The PreviousPageType directive makes the PreviousPage property on the target page return an object of the same type as the page at the given path (or of the specified type if the TypeName attribute is used). This fact alone, though, is not sufficient to let you access input controls directly. In ASP.NET, each page class contains protected members that represent child controls. Unfortunately, you can't call a protected member of a class from an external class. In fact, only derived classes can access protected members of the parent class.

To work around this issue, in the caller page you must add public properties that expose any information you want posted pages to access. For example, imagine that crosspostpage.aspx contains a TextBox named _textBox1. To make it accessible from within a target page, you add the following code to the codebehind class:

public TextBox TextBox1
{
    get { return _textBox1; }
}

The new TextBox1 property on the page class wraps and exposes the internal TextBox control. In light of this code, the target page can now execute the following code:

Response.Write(PreviousPage.TextBox1.Text);
Being the potential target of a cross-page call doesn't automatically make a target page a different kind of page. There's always the possibility that the target page is invoked on its own, for example, through a hyperlink. When this happens, the PreviousPage property returns null and other postback-related properties (like IsPostBack) assume the usual values. If you have such a dual page, it's a good idea to add some extra code to discern the page behavior. The following code hosted in the Page_Load event makes the page work only through cross-page calls:
if (PreviousPage == null)
{
    Response.Write("Sorry, that's not the right way to invoke me.");
    Response.End();
    return;
}

Redirecting Users to Another Page

In addition to the PostBackUrl property of button controls, ASP.NET provides another mechanism for transferring control and values from one page to another: the Server.Transfer method. When you call this method, the URL of the new page is not reflected by the browser's address bar because the transfer takes place entirely on the server—no client redirection ever occurs. The following code shows how to use the method in order to direct a user to another page:

protected void Button1_Click(object sender, EventArgs e)
{
    Server.Transfer("targetpage.aspx");
}
Note that any code following the call to Transfer in the page is never executed. In the end, Transfer is just a page-redirect method. However, it is particularly efficient for two reasons. First, no round-trip to the client is requested as is the case, for example, with Response.Redirect. Second, the same HttpApplication that was serving the caller request is reused to serve the new page request, thus limiting the impact on the ASP.NET infrastructure.

In ASP.NET 1.x, the spawned page can access the page object representing its caller by using the Handler property of the HTTP context object, as shown here:

Page caller = (Page) Context.Handler;
Because the Handler property returns a valid instance of the referrer page object, the spawned page can access all of its public members. It cannot directly access the controls because of the protection level already discussed.

This programming model also works in ASP.NET 2.0. However, in ASP.NET 2.0, things are simplified and using Handler is no longer needed. You can use the same programming model of cross-page postings and rely on a non-null PreviousPage property and the @PreviousPageType directive for strongly typed access to input fields. How can a page detect whether it's being called through a server transfer or through a cross-page postback? In both of these cases, PreviousPage is not null, but the Page.IsCrossPagePostBack on the PreviousPage object is true for a cross-page posting and false in case of a server transfer.

Conclusion

Passing values from one page to another is a task that can be accomplished in a variety of ways—cross-page posting, server transfer, HTML forms, cookies, session-state, and query strings, among others. Which one is the most effective? In ASP.NET 2.0, cross-page posting and server transfer offer a familiar programming model, but potentially move a significant chunk of data through the view state. Whether this information is really needed depends on the characteristics of the target page. In many cases, the target page just needs to receive a few parameters to start working. If this is so, HTML client forms might be more effective in terms of moving data. HTML forms, though, require an ASP-like programming model.

ASP.NET 2.0 adds a few new properties to the HtmlForm class, but doesn't change its core behavior, so self-posting pages are still the primary method of writing pages in ASP.NET. You can still mix client and server forms and also host multiple server forms provided that only one is visible at a time.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Programming ASP.NET and the new book Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Advanced Basics: Revisiting Operator Overloading

Advanced Basics
Revisiting Operator Overloading
Ken Getz

Code download available at: AdvancedBasics0509.exe (369KB)
Contents
Introducing the RGB and HSV Classes
Conversion Operators
Overloading Comparison Operators
Performing Mathematical Operations

On the conference circuit recently, I was speaking about some of my favorite new features in the Microsoft® .NET Framework 2.0, using the content of three recent columns in this series as fodder. In my talk I sped through generics, operator overloading, and the BackgroundWorker component, all in the space of an hour (for the columns, see Advanced Basics: Being Generic Ain't So Bad, Advanced Basics: Calling All Operators, and Advanced Basics: Doing Async the Easy Way). This accomplishment requires speed talking, but over the years I've mastered that skill. Of course, a downloadable"fast conference speaker" codec would help listeners keep up, but alas, that may never happen.

After the session was over, one attendee came up to chat. He suggested that my example for operator overloading was a bit too academic. It showed how to create a class named Vector, which contained a one-dimensional array of integers. In that example, you can overload the +, =, <>, and CType operators in order to manipulate the contents of the vector instances. He then suggested an example that is also a bit esoteric, but it presented a perfect opportunity to revisit operator overloading: using the RGBA and HSV standards for handling colors.

.NET provides the System.Drawing.Color structure for dealing with colors. It's a wrapper around standard RGBA values. An RGBA value is a simple quadruple containing the color values representing the red, green, and blue components of a color, as well as an alpha value representing transparency. Although RGBA is convenient for developers, it's terrible for designers, so there's an alternative technique for describing colors, the HSL (slightly modified as HSV) standard, which stores hue, saturation, and brightness values instead. (A few years back, I delved into these concepts and you can read the results at GDI+: A Primer on Building a Color Picker User Control with GDI+ in Visual Basic .NET or C#.) The earlier article compared the RGB and HSV standards, and understanding their differences will help you here. Also, if you haven't investigated overloading operators in Visual Basic® 2005, please take a moment to review the previously mentioned column on the topic, so that you can understand the basics.

Figure 1 The Sample Form
Figure 1 The Sample Form

The sample application for this column includes a simple form, as shown in Figure 1. This form includes the same features as in my previous GDI article, and adds several new demonstrations that allow you to interact with the new overloaded operators.

Introducing the RGB and HSV Classes

The original GDI article's example included RGB and HSV structures, but here I've promoted them to classes. The RGB class contains three integer values (one each for the red, green, and blue components of the color), and each can take on values between 0 and 255. The HSV class contains three integer values (one each for hue, saturation, and value) and although the scale of these values is arbitrary, I chose to use values between 0 and 255 for consistency with the RGB values. Each class contains a shared method that converts an instance of the type into the other color representation. That is, the RGB class contains a ToHSV method, and the HSV class contains a ToRGB method. These methods (which aren't directly pertinent to this discussion) perform the mathematical conversion from one format to the other. You can investigate these methods in the sample code (both conversions are somewhat complex). The HSV class contains the simple declarations shown in Figure 2.

Figure 2 Portion of the HSV Class (Close Figure 2)
Public H As Integer
Public S As Integer
Public V As Integer

Public Sub New(ByVal H As Integer, ByVal S As Integer, ByVal V As Integer)
    Me.H = H
    Me.S = S
    Me.V = V
End Sub

Public Overrides Function ToString() As String
    Return String.Format("({0}, {1}, {2})", H, S, V)
End Function

Because this class doesn't need to perform any operations on its component values, it seems reasonable for this example and for simplicity to expose them as public integer fields, even though public fields are typically frowned upon. The RGB class provides similar functionality, but in a slightly more complex manner, as you'll see later. The public constructors and ToString method, however, work as you might expect (see Figure 3).

Figure 3 Portion of the RGB Class (Close Figure 3)
Public Sub New(ByVal R As Integer, ByVal G As Integer, ByVal B As Integer)
    Me.R = R
    Me.G = G
    Me.B = B
End Sub

Public Sub New(ByVal value As Color)
    R = value.R
    G = value.G
    B = value.B
End Sub

Public Overrides Function ToString() As String
    Return String.Format("({0}, {1}, {2})", R, G, B)
End Function

There's a point to this added complexity—once it's possible to add two RGB values together, or to add an integer to an RGB value (and both these operations are possible, once you've overloaded the + operator for use with this class), you may end up with values in the R, G, and/or B components that are outside the possible range of 0 to 255. For example, you may want to add 255 to a color to force it to be white (255, 255, 255). On the other hand, you may later want to be able to subtract 255 and retrieve the original color. By storing the actual R, G, and B values in private variables, and only converting these values within the 0 to 255 range when necessary for public consumption, it's possible to play tricks with the RGB contents. You'll see an example later.

Note also that the RGB class contains an overloaded constructor that allows you to pass in a System.Drawing.Color value. This addition makes perfect sense, because the RGB class is merely a wrapper around the three components that make up the System.Drawing.Color value. Creating this added constructor makes it easy to write code like this:

Dim myRGB as New RGB(colorLabel.BackColor)
This code copies the R, G, and B values from the System.Drawing.Color into the new RGB instance.

What might you want to do with these classes that would require operator overloading? If you investigate the previous GDI article, you'll find a lot of code that uses that article's ColorHandler class, which contains the HSV and RGB structures as well as methods to convert between the two. All conversions had to be performed calling methods of the ColorHandler class, like the code snippets in Figure 4, which compare the old and the new conversions.

Figure 4 Old and New Conversions (Close Figure 4)
hsv = ColorHandler.RGBtoHSV(rgb)
' becomes this:
hsv = CType(rgb, HSV)

SetRGB(ColorHandler.HSVtoRGB(hsv))
' becomes this:
SetRGB(CType(hsv, RGB))

selectedColor = ColorHandler.HSVtoColor(hsv)
' becomes this, using explicit conversion:
selectedColor = CType(hsv, Color)

fullColor = ColorHandler.HSVtoColor(hsv.H, hsv.S, 255)
' becomes this, using explicit conversion:
fullColor = CType(New HSV(hsv.H, hsv.S, 255), Color)

In addition, using operator overloading, you can easily add new features that weren't needed in the original application, including:

  • Overloading the = and <> (equal and not equal) operators, allowing you to compare HSV, RGB, and System.Drawing.Color values.
  • Overloading the + and – (addition and subtraction) operators for the RGB class, allowing you to add two RGB values together or add a single integer to an RGB instance.
  • Overloading the / (division) operator, allowing you to divide all the components of an RGB instance by a particular integer—this allows you to average two RGB colors, for example.

The remainder of this column discusses the code that is required to provide these overloaded operators, using sample code from the downloadable application in order to demonstrate how you might use the new operators.

Conversion Operators

You can overload the CType operator, creating either a widening (implicit) or narrowing (explicit) conversion operator. Normally, you use a widening conversion if you're guaranteed that the conversion cannot lose data, and a narrowing conversion if it's possible that data loss might occur. You can call the widening conversion operator without explicitly including the call to CType, but for the narrowing conversion, you must explicitly call CType.

In this case, because converting between RGB and System.Drawing.Color values maintains the same R, G, and B values, it made sense to use widening conversions for these, so that you can perform the conversions implicitly in code. (One could argue against this case, because the System.Drawing.Color structure maintains four components of each color—A, R, G, and B. Therefore, the conversion from System.Drawing.Color to RGB is technically a narrowing conversion, but I'll let that slide.) On the other hand, conversions between RGB and HSV, and similarly between HSV and System.Drawing.Color, could lose some precision during the conversion, and are therefore included as narrowing conversions (requiring an explicit call conversion using CType).

In the RGB class, the conversion operators convert to and from RGB and System.Drawing.Color, and from RGB to HSV (the corresponding HSV to RGB conversion could have been in this class, but it appears in the HSV class in the example in Figure 5).

Figure 5 RGB Conversion Operators (Close Figure 5)
' Allow implicit conversion from RGB to Color.
Public Overloads Shared Widening Operator _
        CType(ByVal value As RGB) As Drawing.Color
    If value Is Nothing Then
        Return Nothing
    Else
        Return Color.FromArgb(value.R, value.G, value.B)
    End If
End Operator

' Require explicit conversion (using CType) from RGB to HSV.
Public Overloads Shared Narrowing Operator _
        CType(ByVal value As RGB) As HSV
    If value Is Nothing Then
        Return Nothing
    Else
        Return ToHSV(value)
    End If
End Operator

' Allow implicit conversion from color to RGB.
Public Overloads Shared Widening Operator _
        CType(ByVal value As Color) As RGB
    Return New RGB(value)
End Operator

Using these operators, you can convert to and from RGB, as shown in the following lines of code:

' Explicit conversion from RGB to HSV:
Dim myHSV As HSV = CType(myRGB, HSV)

' Implicit conversion from RGB to Color:
myLabel.BackColor = myRGB
Note that when converting from HSV to Color, it's easiest to convert the HSV value to RGB first, and from there to Color. The HSV class contains similar overloaded conversion operators, converting to and from System.Drawing.Color, and to RGB, as shown in the code in Figure 6.

Figure 6 HSV Conversion Operators (Close Figure 6)
' Require explicit conversion from HSV to Color. Take advantage of
' implicit conversion from RGB to Color.
Public Overloads Shared Narrowing Operator CType( _
         ByVal value As HSV) As Color
    If value Is Nothing Then
        Return Nothing
    Else
        Return ToRGB(value)
    End If
End Operator

' Require explicit conversion from Color to HSV.
Public Overloads Shared Widening Operator CType(ByVal value As Color) As HSV
    Dim rgb As New RGB(value)
    Return CType(rgb, HSV)
End Operator

' Require explicit conversion (using CType) from HSV to RGB.
Public Overloads Shared Narrowing Operator CType(ByVal value As HSV) As RGB
    If value Is Nothing Then
        Return Nothing
    Else
        Return ToRGB(value)
  End If
End Operator

Overloading Comparison Operators

Because RGB, HSV, and System.Drawing.Color instances all contain essentially the same information, it makes sense to overload the = and <> operators, so you can easily compare two values, no matter how they're stored. The Visual Basic compiler will complain if you overload = without overloading <>, so you'll need to provide overloads for both of these. Although the compiler doesn't warn you if you don't, it just makes sense to also overload the Equals method of each class, as well as GetHashCode.

There's one small difficulty in comparing RGB and HSV values—although the RGB format stores black as 0, 0, 0 (that is, the absence of any red, green, or blue component), HSV can store black in a large number of different ways. In the HSV format, if the value component is 0, it doesn't matter what the hue or saturation components contain—the represented color is black. The simplest solution is to convert the HSV value to RGB format, and then compare two RGB values, and that's what the sample code does.

When overloading the = operator, you'll need to take into account some logic for handling null values. For example, the procedure declaration for overloading the = operator in the RGB class, comparing to RGB instances, looks like this:

Public Overloads Shared Operator =( _
    ByVal value1 As RGB, ByVal value2 As RGB) As Boolean

End Operator
The two values passed to this method might both be null, they might both be non-null, or one or the other might be null. What does it mean to compare two null RGB values, or to compare one null value with an RGB instance? (The same issues apply when working with HSV values.) Although you could decide that any comparison that involves a null reference returns False, I decided to handle colors as if they were value types, much like the String class. In other words, if two colors are both null, the = operator returns True. If one is null and the other isn't, the operator returns False. Otherwise, the operator compares the colors and returns the result. The code in Figure 7 shows the results from the RGB class.

Figure 7 Comparing RGB Values (Close Figure 7)
' Compare two RGB instances, either of which might be Nothing.
Public Overloads Shared Operator =( _
        ByVal value1 As RGB, ByVal value2 As RGB) As Boolean
    If value1 Is Nothing And value2 Is Nothing Then
        ' If both values are Nothing, return True—
        ' the colors are the same.
        Return True
    ElseIf value1 Is Nothing Then
        ' They're not both Nothing, but Value1 is Nothing.
        ' They can't be equal.
        Return False
    Else
        ' One or the other of the colors might be Nothing,
        ' otherwise compare their values.
        Return value1.Equals(value2)
    End If
End Operator

' Compare an RGB and a Color.
Public Overloads Shared Operator =( _
        ByVal value1 As RGB, ByVal value2 As Color) As Boolean
    If value1 Is Nothing Then
        ' A Color instance can't be null, so if the RGB
        ' is Nothing, return False.
        Return False
    Else
        Return value1.Equals(value2)
    End If
End Operator

' Compare an RGB and an HSV, either of which can be null.
Public Overloads Shared Operator =( _
        ByVal value1 As RGB, ByVal value2 As HSV) As Boolean
    If value1 Is Nothing And value2 Is Nothing Then
        Return True
    ElseIf value1 Is Nothing Then
        Return False
    Else
        Return value1.Equals(value2)
    End If
End Operator

Public Overloads Shared Operator <>( _
        ByVal value1 As RGB, ByVal value2 As Color) As Boolean
    Return Not (value1 = value2)
End Operator

Public Overloads Shared Operator <>( _
       ByVal value1 As RGB, ByVal value2 As RGB) As Boolean
  Return Not (value1 = value2)
End Operator

Public Overloads Shared Operator <>( _
        ByVal value1 As RGB, ByVal value2 As HSV) As Boolean
    Return Not (value1 = value2)
End Operator

Note that the overloaded versions of the <> operator simply return the opposite of the corresponding = overloads. This is standard behavior, but it's required by the compiler. The HSV class included the same set of comparison operators, so I'll let you investigate those in the sample project.

The overloaded = operator methods rely on the overloaded Equals method, which comes in three varieties in the RGB class (see Figure 8). If you overload the Equals method, you should also overload the GetHashCode method. The sample includes a simple implementation of this method in both the RGB and HSV classes—in each case, the code simply returns the hash code of the corresponding System.Drawing.Color value.

Figure 8 RGB Equals Overloads (Close Figure 8)
Public Overloads Function Equals(ByVal value As RGB) As Boolean
    If value Is Nothing Then
        ' If value is Nothing, the comparison must fail—the current
        ' instance cannot be null, so the two can't be equal.
        Return False
    Else
        Return (Me.R = value.R AndAlso _
            Me.G = value.G AndAlso Me.B = value.B)
    End If
End Function

Public Overloads Function Equals(ByVal value As Color) As Boolean
    ' The current instance can't be Nothing, and the Color
    ' can't be Nothing, so there's no need to check for
    ' null references. Use the Equals(RGB) overload, taking advantage
    ' of the implicit conversion from Color to RGB:
    Return Me.Equals(New RGB(value))
End Function

Public Overloads Function Equals(ByVal value As HSV) As Boolean
    If value Is Nothing Then
        ' If the HSV value is Nothing, then this comparison must fail.
        ' The current instance can't be Nothing.
        Return False
    Else
        Return Me.Equals(New RGB(value))
  End If
End Function

Public Overloads Function GetHashCode() As Integer
    ' Need to provide an arbitrary integer that represents this color.
    Return CType(Me, System.Drawing.Color).GetHashCode
End Function

The main form in the sample project includes a method named DisplayInfo that displays its parameters in the Listbox on the form:

Private Sub DisplayInfo(ByVal caption As String)
    lstResults.Items.Add(caption)
End Sub

Private Sub DisplayInfo(ByVal caption As String, _
        ByVal value As Object)
    DisplayInfo(String.Format("{0}: {1}", caption, value))
End Sub
Clicking the button labeled "Test Equality Operators" runs the code in Figure 9. Although these examples don't represent every possible combination of types in every possible combination of equality, they do demonstrate that colors that are equivalent compare correctly. (Note that rgb5 and hsv1 represent the same color, and that rgb1 contains the same color as Color.Red.) Without the use of overloaded operators for =, <>, and the overloaded Equals methods, it wouldn't be possible to write simple code like this that compares instances of the RGB and HSV classes.

Figure 9 Testing Equality Operators (Close Figure 9)
Dim rgb1 As New RGB(255, 0, 0)
Dim rgb2 As New RGB(255, 0, 0)
Dim rgb3 As New RGB(0, 255, 255)
Dim rgb4 As RGB
Dim rgb5 As New RGB(255, 109, 126)
Dim hsv1 As New HSV(250, 146, 255)
Dim hsv2 As New HSV(250, 146, 255)

' Display-related code removed here...

DisplayInfo("rgb1.Equals(rgb2)", rgb1.Equals(rgb2))
DisplayInfo("rgb1.Equals(rgb3)", rgb1.Equals(rgb3))
DisplayInfo("rgb1 = rgb2", rgb1 = rgb2)
DisplayInfo("rgb1 = Color.Red", rgb1 = Color.Red)
DisplayInfo("rgb1 = rgb4", rgb1 = rgb4)
DisplayInfo("rgb4 = Color.Red", rgb4 = Color.Red)
DisplayInfo("rgb5 = hsv1", rgb5 = hsv1)
DisplayInfo("hsv1 = hsv2", hsv1 = hsv2)
DisplayInfo("hsv1 = rgb5", hsv1 = rgb5)

Performing Mathematical Operations

Although you won't usually perform mathematical operations on colors, for RGB values it's easy enough to assign reasonable semantics to these operations:

  • Adding or subtracting an integer to an RGB increases or decreases each of the R, G, and B values by the same amount. (This action effectively changes the brightness of the color, so you would get a similar effect by adding or subtracting a value to the V component of an HSV value.)
  • Adding or subtracting two RGB values performs the appropriate operation on each of the properties.
  • Dividing an RGB value by an integer divides each of the members by the integer value.

Of course, you might find uses for other mathematical operations, but as you'll see, it's easy to create your own mathematical operator overloads so feel free to add your own.

As mentioned earlier, because legal RGB component values can only be between 0 and 255, you can get into trouble at the endpoints. For example, if you had an RGB value containing a particular color, and then added 255 to the color (or added Color.White to it), you couldn't simply subtract 255 to get back to the original color—that information would have been lost by adding 255. The sample uses one possible solution: it maintains private Integer values that represent the actual values of the R, G, and B components, but the R, G, and B property Get procedures ensure that the values are within the standard byte range (0 through 255) before returning the values.

For example, the B property procedure looks like this:

Private BValue As Integer

Public Property B() As Integer
    Get
        Return GetValueWithinRange(BValue)
    End Get
    Set(ByVal value As Integer)
        BValue = value
    End Set
End Property
The R and G property procedures work in a similar fashion. The GetValueWithinRange procedure guarantees that no matter what value is stored in the private backing field, the property returns a reasonable value:
Private Function GetValueWithinRange(value As Integer) As Integer
    Return Math.Max(0, Math.Min(255, value))
End Function

The RGB class contains the overloaded operator procedures shown in Figure 10, allowing you to add two RGB values together, or to add an integer to an RGB value. Note that each of these procedures returns Nothing if an RGB parameter is Nothing, and that each of the methods uses the private variables, rather than the public properties, to perform operations.

Figure 10 Addition with RGB Values (Close Figure 10)
' Add two RGB values together.
Public Overloads Shared Operator +( _
        ByVal value1 As RGB, ByVal value2 As RGB) As RGB
    If value1 Is Nothing OrElse value2 Is Nothing Then
        Return Nothing
    Else
        Dim newValue As New RGB( _
        value1.RValue + value2.RValue, _
        value1.GValue + value2.GValue, _
        value1.BValue + value2.BValue)
        Return newValue
    End If
End Operator

' Add a single value to an RGB value.
Public Overloads Shared Operator +( _
        ByVal RGBvalue As RGB, ByVal integerValue As Integer) As RGB
    If RGBvalue Is Nothing Then
        Return Nothing
    Else
        Return New RGB( _
            RGBvalue.RValue + integerValue, _
            RGBvalue.GValue + integerValue, _
            RGBvalue.BValue + integerValue)
    End If
End Operator

The RGB class also provides the overloaded \ (integer division) and / (division) operators shown in Figure 11.

Figure 11 Division with RGB Values (Close Figure 11)
' Divide an RGB value by an integer. The results will always be integers.
Public Overloads Shared Operator \( _
        ByVal value As RGB, ByVal divisor As Integer) As RGB
    If value Is Nothing Then
        Return Nothing
    ElseIf divisor = 0 Then
        Throw New DivideByZeroException()
    Else
        Return New RGB(value.RValue \ divisor, _
            value.GValue \ divisor, _
            value.BValue \ divisor)
    End If
End Operator

' Perform integer division.
Public Overloads Shared Operator /( _
        ByVal value As RGB, ByVal divisor As Integer) As RGB
    Return value \ divisor
End Operator

The sample form provides three demonstrations of these methods. The first, which you can demonstrate by sliding the track bar within the Add/Subtract Selected Color group box, allows you to add and subtract integers to an existing RGB value. When you click Select Color, the code allows you to select a color, and stores it into both the BackColor and ForeColor properties of the label. The button's click Event handler calls the procedure in Figure 12.

Figure 12 SetLabelColor (Close Figure 12)
Private Sub SetLabelColor(ByVal lbl As Label)
    Using frm As New ColorChooser2()
        frm.Color = lbl.BackColor
        If frm.ShowDialog(Me) = DialogResult.OK Then
            SetColor(lbl, frm.Color)
        End If
    End Using
End Sub

Private Sub SetColor(ByVal lbl As Label, ByVal clr As Color)
    lbl.BackColor = clr
    ' Store the color, so you can get it back later. A better solution
    ' would be to inherit from the Label control and add an OriginalColor
    ' property, but that's overkill for this demonstration.
    lbl.ForeColor = clr
End Sub

The track bar's Scroll event handler calls the following code, which simply adds the value of the trackbar, which contains values between -255 and +255, to the selected color:

Dim value As New RGB(colorLabelAddSubtract.ForeColor)

' Add in the value of the trackbar:
value += CInt(TrackBar1.Value)

' Set the label color and display the text:
colorLabelAddSubtract.BackColor = value
colorStringLabelAddSubtract.Text = value.ToString()

Clicking the button in the Blend Two Colors group box performs an even simpler operation.It averages the two colors you've selected, using the following code:

Dim rgb1 As New RGB(colorLabel1.BackColor)
Dim rgb2 As New RGB(colorLabel2.BackColor)
colorLabel3.BackColor = (rgb1 + rgb2) / 2
Try selecting some different pairs of colors to see how they average out. Red and blue average to violet; red and yellow average to orange, just as you might expect from your grade school experimentation with finger paints.

The final demonstration, in the group box labeled "Make a Color Black/White", shows what happens when you add 255 to a color and then subtract it back; or subtract 255 from a color and then add it back. Because the RGB class maintains the real values of its components rather than truncating them internally, it's possible to perform tricks like this. For the color red, the original R value is 255. Adding 255 to this value results in the private RValue member containing 510, but retrieving the R property returns only 255, as it should. Subtract 255 and the value returns to 255, as it should (rather than dropping to 0, as it would if the internal value matched the external value). The code isn't terribly interesting—it simply stores the selected color into a class-level RGB instance, and adds and subtracts 255 on demand. The interesting part is that the code correctly maintains the "out-of-bounds" values, so that you can undo changes that pushed the values out of bounds.

I'll end with a suggestion—don't use operator overloading as a solution in search of a problem. Make sure you're making the code simpler and easier to work with. I removed a lot of code from the original color chooser application, and I'm convinced that adding overloaded operators makes this sample application work better, and provides some new functionality that would otherwise have been more difficult to write.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies, and a courseware author for AppDev. He is coauthor of ASP .NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2002), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Test Run: Low-Level UI Test Automation

Test Run
Low-Level UI Test Automation
James McCaffrey

Code download available at: TestRun0509.exe (134KB)
Contents
The Application Under Test
The Test Automation Code
Finding Windows
Manipulating the App and Checking App State
Message Boxes and Menus
Extending the Test Harness

There are several ways to test a Windows®-based application through its user interface. For example, in the January 2005 issue of MSDN®Magazine (Test Run: Lightweight UI Test Automation with .NET) I described a lightweight technique for testing .NET-based applications using .NET reflection. Since then, many of you have asked how to perform lower-level tasks like clicking away a Message Box. This column answers all those questions by showing you how to write lightweight, low-level UI test automation for Windows-based applications. The technique I'll discuss involves calling Windows API functions such as FindWindow (exposed by user32.dll) and sending Windows messages like WM_LBUTTONDOWN to the application under test.

Now, let's take a look at Figure 1. It shows a dummy Windows color mixer application that allows a user to type a color into a textbox control, then type or select a color in a combobox control, and, after clicking on the button control, a message that represents the result of "mixing" the two colors is displayed in a listbox control. In Figure 1, red and blue produces purple according to the application. The UI test automation is a console application that launches the application under test, obtains window handles to the windows and controls on the app, and simulates a user entering information and clicking the button control. After automatically clicking away an error message window, the test automation checks the resulting state of the application under test, verifies that the listbox control contains the correct message, and prints a pass or fail result. I captured the screen shot in Figure 1 just before the test automation simulates a user clicking on the File | Exit menu item to close the application under test.

Figure 1 Form UI Test Automation
Figure 1 Form UI Test Automation

In the sections that follow I will briefly describe the sample application I am testing, explain how to launch it from the test automation program, and how to get references to the application's various windows and controls. I will also simulate user actions and check application state using Windows API functions and messages. Then I'll describe how you can extend and modify the test system to meet your own needs. I think you'll find the ability to quickly write lightweight low-level automated UI test automation a useful addition to your skill set.

The Application Under Test

Let's look at the application under test. The color mixer application is a simple Windows-based form. For simplicity I accepted the Visual Studio® .NET default control names of Form1, textBox1, comboBox1, button1, and listBox1. I added a single top-level menu item, File, which has a single submenu item, Exit. The heart of the application under test is the code in Figure 2.

Figure 2 Color Mixer Application Code (Close Figure 2)
private void button1_Click(object sender, System.EventArgs e)
{
    string tb = textBox1.Text;
    string cb = comboBox1.Text;

    if (tb == "<enter color>" || cb == "<pick>")
        MessageBox.Show("You need 2 colors", "Error");
    else if (tb == cb)
        listBox1.Items.Add("Result is " + tb);
    else if ((tb == "red" && cb == "blue") ||
             (tb == "blue" && cb =="red"))
        listBox1.Items.Add("Result is purple");
    else
        listBox1.Items.Add("Result is black");
}

When a user clicks on the button1 control, the application grabs the text values in the textBox1 and comboBox1 controls. If the user has not entered two colors (changing the contents of the controls from their default values), a Message Box is displayed with an error message (see Figure 3). As you'll see later, a Message Box is a top-level window which cannot be manipulated directly through the application that calls it. This means if you want to be able to click the Message Box away you'll have to work at the window level instead of the application level.

Figure 3 Error Message
Figure 3 Error Message

If the two color strings match, a message with that color is displayed. If the textbox and combobox controls contain "red" and "blue", a result message with "purple" is displayed. If any other color combinations are in the textbox and combobox controls, a result message with "black" is displayed. Obviously this is just a dummy app for demo purposes but it has most of the fundamental characteristics of Windows-based apps that are needed to demonstrate automated UI testing.

Manually testing even this minimal application through its user interface would be monotonous, error prone, time consuming, and inefficient. You'd have to enter some inputs, click the button control, visually verify the result message, and manually record the pass/fail result. A much better approach is to leverage the capabilities of the .NET environment to write lightweight test automation that simulates a user exercising the application and determining if the application has responded correctly. By automating tedious test cases you can free up time for more interesting and useful cases that need to be performed manually.

The Test Automation Code

The overall outline of the test code is shown in Figure 4. I decided to use C# but you can easily modify the code to Visual Basic® .NET or any other .NET-compliant language. I begin by adding and declaring a reference to a library named WindowControllerLib that contains all the low-level code and really does all the work (this library is available for download from the MSDN Magazine Web site). I'll explain that code in the next three sections of this column. I could have placed all the library code directly in the test harness, but creating a separate library DLL provides better organization and allows the library code to be reused more easily.

Figure 4 UI Test Automation Code Structure (Close Figure 4)
using System;
using System.Diagnostics;  // for Process.Start()
using System.Threading;    // for Thread.Sleep()
using WindowControllerLib; // low-level user-defined library

namespace RunTest {
  class Class1 {
    static void Main(string[] args) {
      try {
        Console.WriteLine("\nStart test scenario\n");
        Console.WriteLine("Launching app under test");
        using(Process p = Process.Start("C:\\LowLevelUIAutomation\\" +
            "WinApp\\bin\\Debug\\WinApp.exe"))
        {
          // find the app form window
          // find textBox1, comboBox1, button1, listBox1
          // click button, get error message box
          // find message box
          // get message box OK button, click it
          // type 'red' to textBox1 and 'blue' to comboBox1
          // click button
          // check listBox1 state, print 'pass' or 'fail'
          // get File->Exit, and click it
        }
        Console.WriteLine("\nEnd test scenario");
      }
      catch(Exception ex)
      {
        Console.WriteLine("Fatal error: " + ex.Message);
      }
    }
  }
}

After launching the application under test, the first step to automating is to get a reference to it:

Controller f = new Controller(p.MainWindowHandle);

My library has a class named Controller that represents a window. Here I declare a Controller object to represent the main application form. The handle to the main window of the application is available from a property on the Process object returned from the call to Process.Start.

Finding Windows

The key to my lightweight low-level UI automation is a library I named WindowControllerLib. It consists of a single class named Controller with a single data field:

private IntPtr ptrToWindow = IntPtr.Zero; // handle
The System.IntPtr data type is a platform-specific type that is used to represent a native pointer or handle.

I defined four overloaded Controller class constructors. The constructor I used previously simply accepts an IntPtr and stores that to the ptrToWindow private field. The primary constructor accepts a window class name and a window caption/name/title and calls the FindWindow Windows API function:

public Controller(string lpClassName,
   string lpWindowName)
{
   this.ptrToWindow = FindWindow(lpClassName,
                         lpWindowName);
}

The .NET environment makes it easy to call Windows API functions using the P/Invoke mechanism. All I have to do is import the System.Runtime.InteropServices namespace and then place a DllImport attribute inside my Controller class definition:

[DllImport("user32.dll", CharSet=CharSet.Auto)]
static extern IntPtr FindWindow(string lpClassName, string lpWindowName);
Here I am essentially saying I want to use the Windows API function FindWindow, located in user32.dll, and let .NET worry about Unicode and ASCII issues. Very nice.

I defined a third Controller constructor which is designed to obtain a reference to a child window of another window:

public Controller(IntPtr hwndParent, IntPtr hwndChildAfter,
                  string lpszClass, string lpszWindow)
{
  this.ptrToWindow = FindWindowEx(hwndParent, hwndChildAfter,
                                  lpszClass, lpszWindow);
}
This constructor is useful to get an object that is a control on a form because most controls are child windows. The constructor calls the FindWindowEx API function. The first parameter is a handle to the parent window. The second parameter tells the function where to start searching from and usually takes a null value, which means to search all child windows. Just like the FindWindow function, all I need to do to be able to call FindWindowEx is to insert an appropriate DLLImport attribute:
[DllImport("user32.dll", CharSet=CharSet.Auto)]
static extern IntPtr FindWindowEx(IntPtr hwndParent,
  IntPtr hwndChildAfter, string lpszClass, string lpszWindow);

The fourth constructor uses FindWindowEx to get a reference to a child window that does not have a name/title:

public Controller(IntPtr hwndParent, int index)
{
  int ct = 0;
  IntPtr result = IntPtr.Zero;
  do
  {
    result = FindWindowEx(hwndParent, result, null, null);
    if (result != IntPtr.Zero) ++ct;
  } while (ct < index && result != IntPtr.Zero);
  this.ptrToWindow = result; // return 1-based indexed child
}
This constructor is useful for test automation because there are many window controls that may not have a title. Examples include empty textbox and listbox controls. But every child window has a predecessor and successor window, so I can calculate a 1-based index value relative to its parent. I use a 1-based index so I can use an index of 0 to represent a self reference. So if you know the index of the child window you can use this constructor to get it whether it has a name and title or not. For instance, in the example application under test, the combobox child control of the main form window has an index of 1, the button is 2, the listbox is 3, and the textbox is 4. Note that these child window index values are not the same as the TabIndex property values for controls. There are a couple of ways you can determine the index value of a child control. One way is to examine the source code of the application under test. The index value of a child control is the position in which it is added with Controls.Add method. For example here is how the app controls are added to my dummy app:
this.Controls.Add(this.comboBox1);
this.Controls.Add(this.button1);
this.Controls.Add(this.listBox1);
this.Controls.Add(this.textBox1);

But in a testing environment you may not always have access to the application source code. A second way to determine the index value of a child control/window is to use the Spy++ tool that comes with Visual Studio .NET. You can use the tool to get detailed information about any window, including the predecessor and successor windows. Figure 5 shows the information for the button1 control. You can see the button1 control is a child of Form1 and its predecessor window has caption "<pick>" (which is the combobox control) and its successor window has an empty caption (which is the listbox control).

Figure 5 Getting Window Information Using Spy++
Figure 5 Getting Window Information Using Spy++

To summarize, I've defined four Controller constructors that are wrappers around the FindWindow and FindWindowEx Windows API functions. The first constructor can be used to get a reference to the main form, like so:

Controller f = new Controller(p.MainWindowHandle);
whereas the second constructor can be used to search for and get a handle to the same window using its window name:
Controller f = new Controller(null, "Form1"); // Form1

The third constructor can be used to get a reference to a child window/control if you know the child's name/title:

Controller butt = new Controller(f.PtrToWindow, IntPtr.Zero,
                                 null, "button1"); // button1

And finally the fourth constructor can be used to get a reference to a child window/control if you know its 1-based index relative to the parent, as shown here:

Controller lb = new Controller(f.PtrToWindow, 3); // listBox1

After calling a constructor I can check to see if I actually got a reference to a window by checking the IsValid property, which simply compares the stored IntPtr to IntPtr.Zero:

public bool IsValid { get { return this.ptrToWindow != IntPtr.Zero; } }

Manipulating the App and Checking App State

After adding code to my library that enables me to get a reference to any window/control on my application, I need to add code that allows me to manipulate the app. Exactly how you'll want to manipulate your application under test will vary according to your test scenario, but in this case I need to send characters to the textbox control. The key to most low-level window manipulation is to use the SendMessage API function to send a Windows message to the window/control. Here is code from the Controller class that will send a single character to a target window:

public void SendChar(char c) {
  uint WM_CHAR = 0x0102;
  SendMessage1(this.ptrToWindow, WM_CHAR, c, 0);
}

If you are new to calling the SendMessage function, the hard part is figuring out exactly which Windows message to send. Here I'm sending a WM_CHAR message. There's no easy way to know which message to use. What I like to do is just browse through a list of all Windows message constants in the WinUser.h file (usually located in the PlatformSDK\Include subfolder of your Visual Studio .NET installation) and then look up information in the Visual Studio .NET integrated help (another useful technique is to use the Spy++ tool mentioned earlier to monitor what messages are being sent to a control you're using).

With a little bit of experience you'll get a good feel for useful messages. The last two of the SendMessage function's four parameters are named wParam and lParam. These two parameters have variable data type and meaning depending on which message argument is passed to the second parameter. In the case of WM_CHAR, the wParam is the character code of a key press. (Notice that I perform an implicit type conversion from char to int.) The lParam is an integer whose bits specify the repeat count, scan code, extended-key flag, context code, previous key-state flag, and transition-state flag of the key press. In this case I just pass 0 (or in other words all 32 bits are 0) which is a normal key press.

To invoke the SendMessage function I added the following P/Invoke declaration:

[DllImport("user32.dll", EntryPoint="SendMessage")]
static extern bool SendMessage1(IntPtr hWnd, uint Msg,
                                int wParam, int lParam);
This attribute will create a "SendMessage1" alias for SendMessage. I do this because SendMessage can take several input parameter and return type signatures.

Once I have the ability to send a single character to a window control, I can easily create a method to send a string of characters:

public void SendChars(string s)
{
  foreach (char c in s) SendChar(c);
}

With these two routines in hand I can send a string to the textBox1 control:

Controller tb = new Controller(f.PtrToWindow, IntPtr.Zero,
                               null, "<enter color>");
Console.WriteLine("\nTyping 'red' to app");
tb.SendChars("red");
Nearly all UI automation needs the ability to simulate a button click. Here's how I implemented that functionality in my WindowControllerLib class:
public void ClickOn()
{
  uint WM_LBUTTONDOWN = 0x0201;
  uint WM_LBUTTONUP   = 0x0202;
  PostMessage(this.ptrToWindow, WM_LBUTTONDOWN, 0, 0); // button down
  PostMessage(this.ptrToWindow, WM_LBUTTONUP, 0, 0); // button up
}

I use the PostMessage API function instead of SendMessage. Usually you'll want to use SendMessage which calls the window procedure for the specified window and does not return until the procedure has processed the message. For a simulated button click I want to return without waiting for the thread to process the message. As before, I create a P/Invoke declaration for PostMessage:

[DllImport("user32.dll")] // used for button-down & button-up
static extern int PostMessage(IntPtr hWnd, uint Msg,
                              int wParam, int lParam);

When processing the WM_LBUTTONDOWN and WM_LBUTTONUP messages, the wParam value indicates whether various key-state masks like MK_SHIFT are down. The lParam value represents the x and y coordinates of the cursor relative to the upper-left corner of the window. Here I pass 0,0 to WM_LBUTTONDOWN and WM_LBUTTONUP in order to simulate mouse clicks on the upper-left corner of the target window without any keys pressed.

In a test automation scenario you must be able to check the state of windows/controls. In this example I need to check the listBox1 control to see if the expected message is there or not. Here is the method I added to my library to do that:

public int ListBoxFindString(string s)
{
  uint LB_FINDSTRING = 0x018F;
  int result = SendMessage4(this.ptrToWindow, LB_FINDSTRING, -1, s);
  return result; // -1 if not found, 0-based index otherwise
}

I use the SendMessage function with a LB_FINDSTRING Windows message argument. Notice that this use of SendMessage returns an int, and accepts int and string arguments for its wParam and lParam parameters. This signature is different than the one I used with the WM_CHAR message in my SendChar library method. Now because of overloading I could have just used a default SendMessage alias instead of creating SendMessage1 and SendMessage2; however, other Windows messages use the same parameter types but with different return types and so they can't be distinguished. I find it easier just to create multiple SendMessageX-type aliases. My method returns -1 if the target string is not found; otherwise it returns the 0-based index of the target's location in the associated listbox control.

With a ListBoxFindString method defined I can call it in my test automation code, like so:

Console.WriteLine("Checking listBox1 for 'purple'");
int result = lb.ListBoxFindString("Result is purple");
Console.WriteLine("Test scenario result = " +
  (result == -1 ? "*FAIL*" : "Pass");

Message Boxes and Menus

Dealing with message boxes and menus requires slightly different tricks than normal windows/controls. Recall that my dummy application under test will display an error message box if the user has not supplied two colors before clicking on the button control (see Figure 3). My test scenario exercises this functionality so I need to know how to click the error message box away. This is easy if you know that a message box window is not a child window of its calling window. So in order to click the OK button on a message box you get a reference to the message box just like you would any other top-level window and then get the child OK button control. The code in Figure 6 shows how I did this in my test scenario code.

Figure 6 Dealing with a Message Box (Close Figure 6)
// find msgBox
Console.WriteLine("Looking for Message Box");
Controller mb = null;
bool mbFound = false;
while (!mbFound)
{
  mb = new Controller(null, "Error");
  if (mb.PtrToWindow == IntPtr.Zero)
  {
    Console.WriteLine("Message Box window not found yet . . . ");
    Thread.Sleep(100);
  }
  else
  {
    Console.WriteLine("Message Box window found with ptr = " +
                      mb.PtrToWindow);
    mbFound = true;
  }
}

// get Message Box OK button
Console.WriteLine("Clicking away Message Box");
Controller okButt = new Controller(mb.PtrToWindow, 1);
okButt.ClickOn();

Dealing with menu items requires special handling because Windows menus are neither child controls nor top-level windows but rather separate objects. The trick to working with menus is to use the GetMenu, GetSubMenu, and GetMenuID Windows API functions with the WM_COMMAND message:

[DllImport("user32.dll")]
static extern IntPtr GetMenu(IntPtr hWnd);

[DllImport("user32.dll")]
static extern IntPtr GetSubMenu(IntPtr hMenu, int nPos);

[DllImport("user32.dll")]
static extern int GetMenuItemID(IntPtr hMenu, int nPos);

I coded a FormExitApp method in my library that simulates a user clicking on a specified item of a specified submenu:

public void FormExitApp(int subMenu, int exitItem)
{
  IntPtr pMenu = GetMenu(this.ptrToWindow); // get main Menu
  IntPtr pSubMenu = GetSubMenu(pMenu, subMenu);   // get "File" submenu
  int menuID = GetMenuItemID(pSubMenu, exitItem); // get "Exit"
  uint WM_COMMAND = 0x0111;
  SendMessage4(this.ptrToWindow, WM_COMMAND, menuID, null);
}
The code is fairly self-explanatory. GetMenu gets a reference to the main menu associated with a window/form. Then this reference can be used with GetSubMenu to get a reference to a particular submenu (such as "File", "Edit", "View"). And then that reference can be used to get a menu ID (such as "Copy", "Paste"). The WM_COMMAND message is sent when the user selects a command item from a menu, when a control sends a notification message to its parent window, or when an accelerator keystroke is translated. So sending WM_COMMAND simulates clicking on a menu item.

Extending the Test Harness

The code presented in this column gives you a foundation to build your own lightweight low-level UI test automation libraries. As you've seen, the steps determine what you want to do, determine which Windows message to use, write a small wrapper method around the SendMessage or PostMessage API functions, and then call the wrapper method in your test code.

Let's walk through an example. Recall that my example test automation supplies a color string to the comboBox1 control by calling a SendChars helper method, simulating typing into the combobox. Suppose you want to supply a color string by simulating a user selecting one of the comboBox1 items instead of simulated typing. By browsing through the WinUser.h file you'll find a CB_SELECTSTRING message. After looking this message up in the Visual Studio .NET integrated help you'd find this is just what you need and also get details of how to call SendMessage. In this case, the return value is CB_ERR = -1 if the selection is unsuccessful or the 0-based index of the selected item if successful. The wParam parameter specifies where to begin searching (-1 means search all), and the lParam parameter is the string to search for. Next you add a new P/Invoke definition if necessary. In this case, none of the library SendMessage signatures I've defined thus far is an exact match, so you'd add a new alias (the typical approach is to create a single P/Invoke definition for a target function and change how you marshal the parameters, but I find this approach easier):

[DllImport("user32.dll", EntryPoint="SendMessage",
  CharSet=CharSet.Auto)]
static extern bool SendMessage2(IntPtr hWnd, uint Msg,
                                int wParam, string lParam);
Then you'd code a wrapper method:
public void ComboBoxSelectString(string s)
{
  uint CB_SELECTSTRING = 0x014D;
  SendMessage2(this.ptrToWindow, CB_SELECTSTRING, -1, s);
}
And finally you'd call the new library method in your test code:
Controller cb = new Controller(f.PtrToWindow, 1); // comboBox1
cb.ComboBoxSelectString("blue");

Notice that because this column is primarily instructional I have left out most error checking, but in a production environment you'll want to add plenty. For example, in the ComboBoxSelectString helper method I didn't check my input argument and I ignored the SendMessage return value. Because your test automation will typically be running unattended and you expect to find errors, you'll want to liberally add error-checking code in both your library and your test harness.

The automated UI test technique I've presented here has been used successfully on many products. Because it is very quick and easy to implement it can be used early in the product cycle when the system under test is highly unstable. The disadvantage of this UI test automation technique is that it is relatively lightweight so it's difficult to handle all possible UI testing situations. Additionally, the code in this column is essentially traditional Win32® code that has been ported to managed code. I have not taken advantage of some of the new features of Windows Forms, such as support for the WM_GETCONTROLNAME message which allows you to easily identify controls that have no caption. For more information, see Brian McMaster's article "Automating Windows Forms".

The next generation of Windows, code-named "Longhorn," will use a new graphics subsystem called "Avalon," which promises to take the UI test automation to a new level by enabling direct and consistent access to all Windows UI elements. Even so, there will still be situations where low-level automation techniques, such as the ones I've presented here, will be useful.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Design Patterns: Dependency Injection

Design Patterns
Dependency Injection
Griffin Caprio

Code download available at: DesignPatterns0509.exe (121KB)
Contents
Factory Patterns Refresher
DI Implementation Using Factories
Abstracting DI Using Containers
Containers vs. Factories
Containers Are Not New
Lightweight Containers
Spring.NET
Factory Example
A Spring.NET Implementation
Enhancing Extensibility
Dependency Resolution
Conclusion

Today there is a greater focus than ever on reusing existing components and wiring together disparate components to form a cohesive architecture. But this wiring can quickly become a daunting task because as application size and complexity increase, so do dependencies. One way to mitigate the proliferation of dependencies is by using Dependency Injection (DI), which allows you to inject objects into a class, rather than relying on the class to create the object itself.

The use of a factory class is one common way to implement DI. When a component creates a private instance of another class, it internalizes the initialization logic within the component. This initialization logic is rarely reusable outside of the creating component, and therefore must be duplicated for any other class that requires an instance of the created class. For example, if class Foo creates an instance of class Bar and instances of Bar require several initialization steps, different for each instance of Bar, other classes that create instances of Bar will have to reproduce the same initialization logic found within Foo.

Developers like to automate monotonous and menial tasks, and yet most developers still perform functions such as object construction and dependency resolution by hand. Dependency resolution can be described as the resolving of defined dependencies of a type or object. Dependency Injection, on the other hand, aims to reduce the amount of boilerplate wiring and infrastructure code that you must write.

Containers provide a layer of abstraction in which to house components. DI containers, in particular, reduce the kind of dependency coupling I just described by providing generic factory classes that instantiate instances of classes. These instances are then configured by the container, allowing construction logic to be reused on a broader level.

Before diving into DI containers, let's first review a core pattern used through DI containers, the Abstract Factory pattern.

Factory Patterns Refresher

In Design Patterns (Addison-Wesley, 1995), authors Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides describe the intent of the Abstract Factory pattern like this: "To construct and instantiate a set of related objects without specifying their concrete objects." Utilizing the Abstract Factory pattern in your applications allows you to define abstract classes to create families of objects. By encapsulating the instantiation and construction logic, you retain control over the dependencies and the allowable states of your objects.

Frequently, objects need to be instantiated in a coordinated fashion, usually because of certain dependencies or other requirements. For example, when creating an instance of System.Xml.XmlValidatingReader in client code, an XmlSchemaCollection object is frequently populated with the relevant schemas for use when validating the XmlValidatingReader object. This is an example of needing to not only create an instance of a class, but also to configure it after creation and before it can be used.

Another type of factory pattern is called the factory method. A factory method is simply a method, usually defined as static, whose sole purpose is to return an instance of a class. Sometimes, in the case of facilitating polymorphism, a flag will be passed to the factory method to indicate the specific interface implementation or subclass to be returned. For example, the Create method of WebRequest takes in either a string or Uri instance, and returns a new instance of a class derived from WebRequest.

From this point forward, I will simply use the word "factories" to mean both the Abstract Factory pattern as well as the factory method implementation.

DI Implementation Using Factories

Factories allow for an application to wire together objects and components without exposing too much information about how the components fit together or what dependencies each component might have. Instead of spreading complex creation code around an application, factories allow for that code to be housed in a central location, thereby facilitating reuse throughout the application. Client code then calls creation methods on the factory, with the factory returning complete instances of the requested classes. Encapsulation is preserved, and the client is effectively decoupled from any sort of plumbing required to create or configure the object instance.

Figure 1 Factory Functions
Figure 1 Factory Functions

Factories can do more than simply create objects and assemble their dependencies. They can also serve as a central configuration area for applying services or constraints uniformly across all instances of an object (see Figure 1). For example, instead of returning an instance of an object, a factory can return a proxy to the real instance of the object, thereby enabling distributed method calls. Since the client application is unaware that the object it is being handed is, in fact, a proxy, as opposed to the real instance of the object, no changes to the client code need to occur. An example of this type of service can be found within the .NET remoting infrastructure. Distributed objects can be configured declaratively, with a .NET configuration file, and the client application can simply create an instance of the class using "new". This is the same for local and distributed objects, as well as for client-activated objects and server-activated objects. All of this configuration and management takes place without the client application knowing about .NET remoting.

However, factories are not without drawbacks. While a factory implementation can be quite valuable for a certain application, most of the time, it is not reusable across other applications. Frequently all of the available creation options are hardcoded into the factory implementation, making the factory itself non-extensible. Also, most of the time the class calling the factories' creation methods must know which subclass of the factory to create.

Secondly, all dependencies for an object that is created using a factory are known to the factory at compile time. Leaving .NET reflection out of the picture for a moment, at run time there is no way to insert or alter the manner in which objects are created, or which dependencies are populated. This all must happen at design time, or at least require a recompile. For example, if a factory is creating instances of class Foo and instances of class Foo require an instance of class Bar, then the factory must know how to retrieve an instance of the Bar class. The factory could create a new instance, or even make a call to another factory.

Third, since factories are custom to each individual implementation, there can be a significant level of cross-cutting infrastructure that is held captive inside a particular factory. Take my example of a factory dynamically substituting a proxy object for a real object. That is an example of a piece of infrastructure, namely the wrapping of simple objects for deployment over a distributed wire, that is completely encapsulated inside that particular factory. If another object needs to be altered in a similar manner, the logic to do so is hidden inside a factory, and would have to be repeated for the other object. Once this functionality is desired outside of the original application, the problem now becomes how to reuse such functionality while still maintaining the existing factory concept.

Lastly, factories rely on well-defined interfaces to achieve polymorphism. In order for a factory implementation to be able to dynamically create different concrete subclass instances, there must be a common base class or shared interface implemented by all classes that the factory will create. Interfaces decouple the construction of the object from the specific implementation of the interface. The dilemma that arises now is how you can accomplish this decoupling without being forced to create an interface for everything.

These are just some of the problems facing DI implemented using conventional factory implementations. However, as you will see shortly, another viable option exists. Also, DI is not based solely around the factory pattern and in fact has many correlations with many other patterns, including the Builder, Assembly, and Visitor patterns. For more information on these useful patterns, the Design Patterns book (already mentioned) is the seminal reference.

Abstracting DI Using Containers

Many of the previous shortcomings to DI can be solved by using a container. A container is a compartment that houses some sort of abstraction within its walls. Typically, responsibility for object management is taken over by whatever container is being used to manage those objects. However, containers can also take over instantiations, configuration, as well as the application of container-specific services to objects.

Containers allow for objects to be configured by the container, as opposed to being configured by the client application. This allows for the container to serve a wide range of functions, such as object lifecycle management and dependency resolution. In addition, containers can apply cross-cutting services to whatever construct is being hosted inside the container. A cross-cutting service is defined as a service that is generic enough to be applicable across different contexts, while providing specific benefits. An example of a cross-cutting service is a logging framework that logs all method calls within an application.

Containers vs. Factories

There are several reasons to use containers in your application development. Containers provide the ability to wrap vanilla objects with a wealth of other services. This allows the objects to remain ignorant about certain infrastructure and plumbing details, like transactionality and role-based security. Oftentimes, the client code does not need to be aware of the container, so there is no real dependency on the container itself.

These services can be configured declaratively, meaning they can be configured via some external means, including GUIs, XML files, property files, or vanilla .NET-based attributes.

Containers that have cross-cutting services are also reusable across applications. One container can be used to configure objects across various applications within an enterprise. Many services that can be applied across an enterprise are low-level infrastructure and plumbing services. These services can be used across an enterprise without the need to deeply embed container-specific code or logic within an application.

Containers Are Not New

Containers have been around in some form or another for many years. As a matter of fact, containers were used back when Microsoft® Transaction Server (MTS) was released as part of the Windows NT® 4.0 option pack.

Containers are still an active part of Microsoft enterprise development strategy today. In fact, if you're writing .NET-based code, you're already using a container to deploy your application: the .NET common language runtime (CLR). The CLR performs a wide variety of important tasks at run time, including memory management, automatic bounds checking and overflow protection, and method call security, to name a few.

The next version of MTS, dubbed COM+, was a major evolution. The .NET equivalent, Enterprise Services, is still the recommended approach to constructing distributed enterprise applications. COM+ and Enterprise Services offer a wealth of services beyond what Microsoft Transaction Server originally offered. In the current version, this includes object messaging, object pooling, declarative automatic transactions, loosely coupled events, and role-based security.

The problem with some containers is that they can be costly. Despite being built upon a fairly stable architecture, the current container technology available to developers using .NET has some drawbacks. They require container-specific constructs to be introduced into domain code. Container infrastructure can adversely impact performance for many operations, even if only minimally.

An example of requiring container-specific constructs can be found in the Microsoft .NET Framework 1.x Enterprise Services requirement that any object that is under its control must derive from the ServicedComponent class. Since .NET does not support multiple inheritance, this constraint limits where Enterprise Services can be utilized.

Since heavyweight containers impact performance and increase the complexity of the client application, they are usually employed in only the largest distributed applications.

Microsoft also offers built-in support for a lightweight version of Dependency Injection, with the System.ComponentModel namespace. Unlike EnterpriseServices, it does not provide any extra services or functionality; it merely provides service injection. However, like Enterprise Services, in order to use the classes within the System.ComponentModel namespace, your classes must become container-aware. This is accomplished by implementing certain container-specific interfaces.

Lightweight Containers

There are a great number of applications that would benefit from many of the features of the containers I described, but their needs don't justify the use of a heavyweight container. At the opposite end of the containers spectrum, lightweight containers provide many of the same benefits that the heavyweight containers do, but without all of the overhead of COM+ and Enterprise Services. Many organizations still choose to use Enterprise Services in spite of the existence of lightweight containers, but this situation is changing. Many of these lightweight containers offer services in addition to simple DI. The containers can often be configured to add other valuable services to your objects.

Spring.NET

You can build your own lightweight DI container, though a few implementations of such systems already exist which you might consider taking advantage of. One such solution is Spring.NET, which I'll be using for the rest of this column to demonstrate some of the ideas discussed thus far. Spring.NET offers a lightweight DI container built around the concept of factories. It not only provides DI by allowing users to use pre-built factories within their code, but it also provides a suite of services that can be applied to any object under Spring.NET's control. And since Spring.NET is built using standard .NET-based code, applications that utilize Spring.NET have no additional dependency on COM, COM+, or Enterprise Services.

Factory Example

The following code shippet is a simple interface, IDomainObjectInterface, which my objects will implement. The interface contains one property, which returns a string representation of the name of my object:

public interface IDomainObjectInterface
{
    string Name{ get; }
}

The code in Figure 2 contains two classes that implement the interface. As you can see, the Name property simply returns the name of the class, depending on which concrete class is used. I will use these two classes, as well as the interface they implement, as the basis of the example I will present.

Figure 2 Implementation Classes (Close Figure 2)
public class ImplementationClass1 : IDomainObjectInterface {
    public ImplementationClass1(){}
    public string Name
    {
        get { return "Implementation Class 1"; }
    }
}

public class ImplementationClass2 : IDomainObjectInterface {
    public ImplementationClass2(){}
    public string Name
    {
        get { return "Implementation Class 2"; }
    }
}

Typically, either of these two classes would be created by a factory class, similar to the one in Figure 3. In addition to the factory class, ImplementationClassFactory, Figure 3 also contains one enumeration, ImplementationClassType. The factory class has one method, GetImplementationClass, which accepts one of the enumeration values. Based on the value of the enumeration, one of the two IDomainObjectInterface implementations will be returned. The client class is responsible for choosing which implementation it would like to use.

Figure 3 Factory Class (Close Figure 3)
public enum ImplementationClassType
{
    ImplementationClass1, ImplementationClass2
}

public class ImplementationClassFactory
{
    public static IDomainObjectInterface GetImplementationClass(
        ImplementationClassType implementationClassType )
    {
        switch ( implementationClassType )
        {
            case ImplementationClassType.ImplementationClass1:
                return new ImplementationClass1();
            case ImplementationClassType.ImplementationClass2:
                return new ImplementationClass2();
            default:
                throw new ArgumentException("Class " +
                    implementationClassType + " not supported." );
        }
    }
}

Now, there are several drawbacks to this factory method. The first is that the number of implementation classes is hardwired into the factory method. Therefore, even though there is an interface for the implementation objects, it's impossible for the factory method to return an implementation class that it does not know about. This limits extensibility, especially in the case of a public API and application frameworks, where the ability to dynamically introduce new implementation classes is not only desired, but often expected to achieve a certain degree of flexibility.

Secondly, even if the ability to dynamically introduce new implementations existed, the client application would still need to know which class to ask for. This eliminates some of the flexibility that the factory class was supposed to provide.

The ConsoleRunner class in Figure 4 illustrates how a client would use the factory class to create an instance of the desired implementation class. Notice how the client code has to explicitly ask for the desired implementation class. At this point many of the benefits of the factory class have been negated.

Figure 4 Using the Factory Class (Close Figure 4)
using System;
using SpringDIExample;

class ConsoleRunner
{
    static void Main()
    {
        IDomainObjectInterface domainObjectInterface =
            ImplementationClassFactory.GetImplementationClass(
                ImplementationClassType.ImplementationClass1);
        Console.WriteLine("My name is " + domainObjectInterface.Name);
    }
}

A Spring.NET Implementation

Now that you have seen the typical factory pattern, let's take a look at how a DI container not only achieves many of the same goals, but also adds a significant amount of flexibility and functionality to your application.

Figure 5 contains an updated version of the ConsoleRunner class. For this example I'm using the Spring.NET DI container, which requires a bit of initial setup. First, you must create an instance of the factory, using a config.xml file as the source of your object definitions. Next, replace a call to the custom factory class with a call to the Spring.NET factory class. Notice that since the generic factory doesn't know anything about third-party interfaces, everything coming back from the factory is downcast to object. So you must upcast the object instances returned by the factory to the interface that you expect. Finally, the last line of the ConsoleRunner class remains unaffected, even though you have changed the source of the object and how it's instantiated.

Figure 5 ConsoleRunner Using Spring.NET (Close Figure 5)
using System;
using System.IO;
using Spring.Objects.Factory.Xml;
using SpringDIExample;

class ConsoleRunner
{
    static void Main()
    {
        // 1. Open the configuration file and create a new
        //    factory, reading in the object definitions
        using (Stream stream = File.OpenRead("config.xml"))
         {
            // 2. Create a new object factory
            XmlObjectFactory xmlObjectFactory =
                new XmlObjectFactory(stream);

            // 3. Call my factory class with generic label for the object
            //    that is requested.
            IDomainObjectInterface domainObjectInterface =
                (IDomainObjectInterface)xmlObjectFactory.
                GetObject("DomainObjectImplementationClass");

            // 4. Use the object just like any other concrete class.
            Console.WriteLine("My name is " + domainObjectInterface.Name);
        }
    }
}

Now, let's take a look at the make up of the config.xml file, which drives the factory class. Here is the full config.xml file:

<?xml version="1.0" encoding="utf-8" ?>
<objects xmlns="http://www.springframework.net">
    <object name="DomainObjectImplementationClass"
            singleton="false"
            type="ImplementationClass1, SpringDIExample" />
</objects>
You will notice that the configuration file is not very large, being comprised of only two elements, <objects>, which contain all of the object definitions, and the individual <object> definitions. From the configuration file shown, you can see that there is one object definition. The object definition contains three basic attributes that define what object is created as well as how it's created.

The name attribute defines the name that my ConsoleRunner class uses when requesting an object from the factory. In this case, it is "DomainObjectImplementationClass". This name is just used to reference the definitions contained in the configuration file from the client code.

Next, the singleton attribute is a Boolean flag that designates if the object should be created as a singleton or not. Spring.NET has built-in support for making objects singletons, but since I do not require such functionality, I set this attribute to false.

Finally, the type attribute defines the actual type of the object to be created. This is the type that will actually be loaded and returned when the factory is queried. This string takes the form of "Type, Assembly", indicating not only the type of object, but also which assembly the type is located in. As indicated in the configuration file, the type desired is one of the types shown in Figure 2.

By simply changing the type of the implementation class listed in the configuration file from "ImplementationClass1" to "ImplementationClass2", you can dynamically alter the class that is returned to the client, all without a recompile.

Enhancing Extensibility

Until now, I have simply moved responsibility of object creation to an external factory implementation and configuration file. While this type of declarative configuration can be seen as more desirable than a static, custom factory implementation, there is much more that can be accomplished by using containers.

Let's say that you have published the IDomainObjectInterface in a public API, and you would like to allow users of the API to create their own implementations of the interface, all while still utilizing several existing clients that have already been built to use the IDomainObjectInterface. Getting the users' implementation to your clients could prove to be difficult, especially since you know nothing about how the class will be built or deployed. ImplementationClass3 is a third implementation of the IDomainObjectInterface, which is similar to the ImplementationClass1 and ImplementationClass2 classes, except this one resides in a separate assembly. Previously, both implementation classes and the interface resided in the same assembly:

public class ImplementationClass3 : IDomainObjectInterface
{
    public ImplementationClass3(){}
    public string Name
    {
        get { return "Implementation Class 3"; }
    }
}

Using a framework like Spring.NET, getting my ConsoleRunner class to use the new ImplementationClass3 is easy to accomplish, and only requires a minor change to my original config.xml configuration file. The following is the configuration file with the necessary changes made. The only difference lies within the type attribute, which has been updated to point to the ImplemenationClass3 type and the SimpleDIExampleExtension assembly:

<?xml version="1.0" encoding="utf-8" ?>
<objects xmlns="http://www.springframework.net">
    <object name="DomainObjectImplementationClass"
            singleton="false"
            type="ImplementationClass3, SpringDIExampleExtension"/>
</objects>

When the ConsoleRunner class is rerun, an instance of ImplementationClass3 will be instantiated and returned. All of this is accomplished without a recompile of ConsoleRunner, even though ImplemenationClass3 resides in an assembly that's physically separate from the original implementation classes.

Dependency Resolution

Now that you have seen how containers can aid in object creation, let's take a look at how dependencies between objects are handled. The following class, DependentClass, has one read/write property that contains a message for the class to hold:

public class DependentClass
{
    private string _message;

    public DependentClass(){}

    public string Message
    {
        set { _message = value; }
        get { return _message; }
    }
}

I will configure the container to automatically insert a message into the DependentClass.Message property. I will also dynamically insert an instance of the configured DependentClass object into the ImplemenationClass4.DependentClass property.

Figure 6 shows a new implementation of IDomainObjectInterface, ImplemenationClass4. As you can see, not only does ImplementationClass4 implement the IDomainObjectInterface, it also has an extra property, DependentClass, which holds an instance of DependentClass.

Figure 6 New Implementation of IDomainObjectInterface (Close Figure 6)
public class ImplementationClass4 : IDomainObjectInterface
{
    private DependentClass _dependentClass;

    public ImplementationClass4(){}

    public DependentClass DependentClass
    {
        get { return _dependentClass; }
        set { _dependentClass = value; }
    }

    public string Name
    {
        get { return _dependentClass.Message; }
    }
}

Figure 7 shows my updated config.xml file. There are three changes from the configuration files shown previously. The first is the addition of a new <object> element that configures an instance of the DependentClass to be used. All of the previously explained attributes of the <object> element are present, but this object definition has an extra element underneath the main object definition. The <property> element configures a property for a given object definition. In this case the name attribute contains the name of the property to populate; here it's DependentClass.Message. Since the DependentClass.Message property is a basic type, its configured value is wrapped in <value> tags. The text contained inside these tags is the configured value of the DependentClass.Message that will be populated at instantiation.

Figure 7 Updated config.xml (Close Figure 7)
<?xml version="1.0" encoding="utf-8" ?>
<objects xmlns="http://www.springframework.net">
    <object name="DomainObjectImplementationClass"
            singleton="false"
            type="ImplementationClass4, SpringDIExample">
        <property name="DependentClass">
            <ref object="DomainObjectDependentClass"/>
        </property>
    </object>
    <object name="DomainObjectDependentClass"
            singleton="false"
            type="DependentClass, SpringDIExample">
        <property name="Message"><value>Dependent Class</value></property>
    </object>
</objects>

The second change concerns my original DomainObjectImplementationClass definition. A new <property> element has been added to the definition, configured to populate the ImplementationClass4.DependentClass property. Since the value of this property is an instance of a complex type, a <ref> tag is used instead of wrapping the value in <value> tags. The object attribute of the <ref> tags references the name of a previously configured object definition, in this case, "DomainObjectDependentClass".

The last change to the configuration file can be found within the first object definition. I have updated the type reference to use the ImplementationClass4 class.

Now, redeploy the new config.xml file, and rerun the ConsoleRunner class. Notice that the configured DependentClass.Message property is displayed. The dependencies have been populated and resolved and the client app is using the new classes, all without knowing what class it's using and all without requiring a recompile.

Conclusion

Dependency Injection is a worthwhile concept to explore for use within apps that you develop. Not only can it reduce coupling between components, but it also saves you from writing boilerplate factory creation code over and over again. Spring.NET is an example of a framework that provides a ready to use DI container, but it is not the only .NET lightweight container out there. Other containers include Pico and Avalon.

Send your questions and comments to mmpatt@microsoft.com.

Griffin Caprio is a Microsoft certified independent consultant in Chicago. He has been helping deploy .NET solutions for the past four years for clients ranging from small startups to large Fortune 500 companies. He is an active developer on several OSS .NET projects, including .NET Mock Objects and Spring.Net.


Security Briefs: Credentials and Delegation

Security Briefs
Credentials and Delegation
Keith Brown

Code download available at: SecurityBriefs0509.exe (124KB)

I get loads of security questions from friends and former students, and recently I've gotten a number of questions about building secure data-driven Web sites for internal enterprise systems. I've decided to answer them here to hopefully save you some headaches in your own projects.

Q Can you tell me why Microsoft Internet Explorer some-times prompts me for credentials when I'm already logged on to my computer?

A I hear this question frequently. If you configure IIS to force integrated authentication, you'd expect that domain users would be able to use their single sign-on with your Web site, right? I remember being surprised by this behavior when I first saw it.

Here are the symptoms: the client is prompted for a password even though she's already logged on to the domain on her workstation. If she cancels the password prompt, she is denied access to the Web site. If she enters her domain credentials (user name and password), she is admitted without further prompts.

Here's what's going on under the covers. Whenever a client fires up a browser and points it to a Web site that requires Windows® integrated authentication, just like any initial HTTP request that Microsoft® Internet Explorer sends, the GET request goes up to the Web server without any sort of proof of who the client is. Internet Explorer has no idea which pages require authentication and which don't, so it errs on the side of protecting its client's anonymity, which is good! When the server receives this anonymous request, it cannot accept it, and thus returns an error message along with headers requesting client authentication.

This is where things get tricky. If Internet Explorer evaluates the URL to be part of the Trusted sites or Local intranet zones, the default security policy says that it should go ahead and complete the authentication handshake silently with the Web server. The client never sees all this handshaking going on: all she knows is that she is able to surf to the Web site without running into any problem.

On the other hand, if Internet Explorer can't place the URL in any other zone, it will fall into the Internet zone by default. The Internet zone is a catchall bucket, and has a rather restrictive security policy by default. Figure 1 shows the setting that's relevant here: in the Internet zone, Internet Explorer will staunchly protect your anonymity, even when a server requests client authentication. This is why the password prompt pops up—Internet Explorer is letting the user know that the Web server is going to find out who she is. If she cancels the prompt, Internet Explorer will not complete the authentication handshake and she will most likely be denied access to the Web site. If instead she types in her credentials, Internet Explorer will use them to complete the handshake, and if the server accepts the credentials, the user will likely be granted access.

Figure 1 Configuring the Local Intranet Zone
Figure 1 Configuring the Local Intranet Zone

This still doesn't answer the question, though. Why would an internal site be considered part of the Internet zone and be subject to this more constrained policy? The answer is that the way zones are calculated, unless you're using a NETBIOS name when you surf to an internal Web site, the URL will most likely fall into the Internet zone bucket. The default heuristic for calculating if an URL is part of the Local intranet zone is to see if there are any dots in the name. Compare the following URLs:

http://www.microsoft.com/default.aspx
http://207.46.130.108/default.aspx
http://webserver/default.aspx

The first and second URLs will send you directly into the Internet zone by default. The third leads to the Local intranet zone, even though all three URLs may actually reference the same server. There are several ways around this. The first option is to ensure that all of your links to internal Web sites use NETBIOS names instead of DNS names. But this doesn't work well for companies that turn off support for NETBIOS. A much cleaner solution would be to ensure that all client machines in the domain are configured to recognize certain sites as internal.

You can experiment with this second option by opening the Internet Options control panel dialog and selecting the Security tab. From there, click the Local intranet zone and press the Sites button, followed by the Advanced button if you're running Windows XP. The dialog you arrive at allows you to add a list of Web sites that should be considered part of the Local intranet zone. You should consider working with your domain administrators to ensure that this list is populated appropriately for all domain clients via group policy. I've shown the location of this setting in group policy in Figure 2.

Figure 2 Configuring Internet Zone Group Policy
Figure 2 Configuring Internet Zone Group Policy

It's pretty important to get this right. A user should never be prompted to type in her domain password after she's already logged in. Too much of that just paves the way for Trojan horses that steal passwords simply by asking for them.

Q How can my internal Web site use the client's single sign-on credentials to access SQL Server?

A The answer depends a lot on whether SQL Server™ is installed on the same machine as the Web server or not. Let's keep things simple and imagine for a moment that SQL Server is right there on the Web server.

In order to pass the client's credentials through to SQL Server, you should ensure that you are impersonating the client when you establish the connection. You can do this either with code or a configuration file. Figure 3 shows one approach for doing it in code from an .aspx page. This approach allows you to run your Web server under a fixed identity and gives you the choice of when you want to use your client's credentials (as opposed to your server's credentials) to access resources. As you can tell by the explicit cast to WindowsIdentity, I'm making some assumptions about how IIS will be authenticating clients. In this Web application I've simply turned off anonymous access in IIS and required Windows integrated authentication, which will force all clients to authenticate right at the front door. This may be a good strategy for internal-facing enterprise Web applications because it leverages the client's single sign on, assuming you have dealt with the tricky little issue I mentioned in the answer to the first question.

Figure 3 Programmatic Impersonation (Close Figure 3)
<%@ page language='c#'%>
<%@ import namespace='System.Data'%>
<%@ import namespace='System.Data.SqlClient'%>
<%@ import namespace='System.Security.Principal'%>

<%
string connStr = "integrated security=sspi;database=pubs";
using (SqlConnection conn = new SqlConnection(connStr)) {
    WindowsIdentity id = (WindowsIdentity)User.Identity;
    WindowsImpersonationContext wic = id.Impersonate();
    try {
        conn.Open(); // open while impersonating the client
    }
    finally {
        wic.Undo(); // stop impersonating
    }

    string sql = "select au_fname, au_lname from authors";
    SqlCommand cmd = new SqlCommand(sql, conn);
    using (IDataReader r = cmd.ExecuteReader()) {
        while (r.Read()) {
            Response.Output.WriteLine("{0} {1}<br>", r[0], r[1]);
        }
    }
}
%>

If you would rather impersonate all the time, you can simply configure your Web application to impersonate the client by default:

<configuration>
    <system.web>
        <identity impersonate='true'/>
    </system.web>
</configuration>

Now all connections to SQL Server from your .aspx pages will use the client's credentials, but so will any requests for local resources such as files. This may not be what you want. If you're impersonating by default and you need to temporarily stop impersonating to open a file or other resource using your Web server's identity, you can do that too:

// stop impersonating
WindowsImpersonationContext wic =
    WindowsIdentity.Impersonate(IntPtr.Zero);
try {
    // using the server's credentials, not the client's
}
finally {
    wic.Undo(); // resume impersonating
}

Note that in both cases where I wrote code to change security contexts, I was careful to use a try/finally block to ensure that no matter what happened, I was able to get back to my previous security context. This is vital. You don't want an exception thrown at the wrong time to cause your application to run code under an unexpected security context.

Q What should I do if SQL Server isn't on the same machine

A I see I'm not going to get away with explaining only the simple case, so imagine that your connection string specifies a remote server. Now the Web server on machine B will impersonate a remote client who is logged onto machine A, and will use the client's credentials to access SQL Server on machine C. This requires more than impersonation: it requires delegation. When the client on machine A authenticates with the Web server on machine B, she proves that she knows a secret associated with her account (typically a password). But she doesn't tell machine B what the password is. In normal circumstances, machine B has no way of proving to machine C that it really does have a valid logon for the client, but C needs proof.

You may have tried this before. If an .aspx page impersonates its client and then uses integrated security to open a SQL Server connection to a remote database using the following connection string

integrated security=sspi;database=pubs;server=C
the result is a rather cryptic exception from SqlConnection.Open:
Login failed for user '(null)'. Reason: Not associated with a
trusted SQL Server connection.

I highly recommend auditing success as well as failure of logon events during these types of tests. With auditing turned on, in this case you'll see that somebody named ANONYMOUS LOGON has logged onto machine C. What's happening is that because the Web server doesn't have network credentials for the client, when it impersonates the client and tries to open a connection to the database, instead of authenticating, it ends up establishing what is called a null session instead, thus the "null" in the error message from SQL Server. And the null session is denied access by default, which is a good thing.

The first step to making this work is to go into Active Directory® and grant the Web server the right to delegate client credentials. You need to tell Active Directory that you have designed machine B to pass through client credentials, and that you trust B enough to assume that it won't abuse this privilege by using delegated client credentials to access other servers on the network other than the one it's designed to access, which in this case is the database server C. You might even be able to eliminate this particular assumption, as I'll show you later.

By default, IIS 6.0 worker processes run under the built-in Network Service logon, which uses the computer's domain credentials. So you must visit the computer account for B in Active Directory and designate that computer account as "Trusted for delegation." The user interface for doing this differs a bit between Windows 2000 and Windows Server™ 2003. The former simply has a checkbox on the first property page you see when you bring up the computer account properties. The latter actually has an entire property page dedicated to delegation, and since it provides a superset of the functionality in Windows 2000, that's where I'll do my demos. Figure 4 shows the Windows 2000 compatible delegation setting (I've elided the lower portion of this dialog for now, but I'll come back to it later). This setting is equivalent to checking the "Trusted for delegation" box for a computer in a Windows 2000 domain.

Figure 4 Windows 2000 Delegation
Figure 4 Windows 2000 Delegation

With these new settings in place, the client should log off and back on again to clear her Kerberos ticket cache, and the next time she connects, her credentials should flow to SQL Server as desired. Of course things aren't usually this easy, as you will see next.

Q I've done all that you said, but delegation still doesn't work. What's wrong?

A First of all, you need to realize that delegation of credentials in Windows is a feature specific to Kerberos. But Windows doesn't always use Kerberos to authenticate clients, even if those clients are using domain accounts. What's really being used under the covers is a meta-protocol whose friendly name is "Negotiate." The two protocols negotiated are Kerberos and Windows NT LAN Manager (NTLM), in that order, and if you negotiate down to NTLM, delegation is not going to work. For example, if either the Web server or SQL Server is running under a local account (such as the default ASPNET account on Windows 2000) or the built-in Local Service account, Kerberos cannot be used, so NTLM will be negotiated, and delegation will not work. In order to have a chance at negotiating Kerberos, both client and server must be using domain accounts to authenticate.

A much more subtle issue is that Kerberos requires something called a Service Principal Name (SPN) in order to function. In order to get a Kerberos ticket for a server, the client must have a way of identifying the server's domain account. Physically this is because the ticket she gets will be encrypted using a master key that is derived from the server's password, and when she sends it to the server, the server must be able to decrypt the ticket. Logically this helps assure the client that the server is not being spoofed by a bad guy, because the bad guy doesn't know the server's password and therefore cannot use the ticket in a mutual authentication Kerberos handshake.

This all sounds really technical, but an SPN is really quite a simple idea: it's just a way to uniquely identify a service running on a particular machine and listening on a particular port. For example, if the system administrator is planning to run a default instance of SQL Server on a machine whose DNS name is data.acme.com, he would create a user account (let's call it ACME\DataAccount) in the domain, and configure SQL Server to use that account. He would then add the following SPN to the ACME\DataAccount user account:

MSSQLSvc/data.acme.com:1433

This tells Active Directory that there is an authorized instance of a service of class "MSSQLSvc" listening on port 1433 on data.acme.com, and that it is supposed to be running as a user called DataAccount in the ACME domain. If anyone needs to get a ticket for that service, the ticket should be encrypted with the master key for ACME\DataAccount.

Now if SQL Server runs with high privilege, say as SYSTEM, it registers this SPN automatically each time it starts. But running SQL Server as SYSTEM is really a bad idea from a security standpoint. For example, it's "game over" if an attacker is able to use a SQL injection vulnerability in your application to execute the xp_cmdshell extended stored proc in a SQL Server that's running as SYSTEM. Besides reviewing your code for holes like this, you should prefer to run SQL Server under a normal user account to provide defense in depth.

Running SQL Server under a least privileged user account means you'll need to register an SPN manually. It's not hard, but it's one of those things that can trip you up on your way to getting delegation working if you aren't aware that you need to accomplish it.

There is a tool called SETSPN.EXE that ships with the various server flavors of Windows. It's part of the support tools installation that you'll find in a subdirectory on the CD. Here's how to use it to see what SPNs are associated with a particular user account:

setspn –L ACME\DataAccount
The following is the command for adding the SPN that I mentioned for SQL Server:
setspn –A MSSQLSvc/data.acme.com:1433 ACME\DataAccount

For delegation to work, you need Kerberos to be negotiated between the client and the Web server, and between the Web server and the database. This means that both the Web server and the database must have the appropriate SPNs. Internet Explorer asks for tickets using the HTTP service class, so if you're running your Web application on a machine called Web under a custom domain account called ACME\WebAccount, you'll need to register an SPN for that account, like so:

setspn –A http/web.acme.com ACME\WebAccount

Q I am having some trouble when attempting to diagnose delegation problems. Do you have any helpful tips?

A Absolutely. The first thing you should do if you suspect that incorrect security configuration is causing legitimate users to have trouble accessing your system is to enable auditing of logon events on the machines hosting the database and Web server. To do this, drill into the local security policies of those machines under Security Settings | Local Policies | Audit Policy | Audit logon events. If the UI won't allow you to change these settings, you'll need to talk to the domain administrator because they are being controlled by group policy. Turn on auditing for both success and failure, then run your scenario again.

Follow the flow of the client's request by looking at the Web server's log first (use the event viewer and look at the security log). If all went well, you should see a successful logon event with your client's name on it. If you open up the log entry, you should see that the logon was established through Kerberos. If you see NTLM, you know you've negotiated down, and should review the tips I've presented here to see if there's something you missed. On the database machine, you should also see a login for the same user, via Kerberos. What will often happen when you're first getting delegation set up is that you'll instead see a logon event from a user called ANONYMOUS LOGON. This indicates a null session, and this will happen if the Web server authenticates with the back end while impersonating a client who used NTLM to connect. Remember, NTLM doesn't support delegation.

Another tool that I've found useful for diagnosing Kerberos problems is called KERBTRAY.EXE, which ships with the resource kit for Windows Server operating systems. You can download the tools for the Windows Server 2003 resource kit from Windows Server 2003 Resource Kit Tools. You shouldn't be without these tools.

When run, KERBTRAY.EXE adds a tray icon from which you can both list all the Kerberos tickets in the interactive user's logon session and purge those tickets. Purging tickets is convenient if you want to retry a scenario without having to log off and log back on. For example, when you change delegation settings in Active Directory, it's a really good idea to purge your client's ticket cache before trying to connect again.

Listing tickets can be very helpful, because you can see if the Web server's ticket has the ok-as-delegate flag set (it won't if you're using constrained delegation, but more on that in a moment). You can also see if the client's Ticket-Granting Ticket (TGT) has the forwardable flag set. If it does not, the client's account likely has been marked "sensitive and cannot be delegated". Unfortunately, KERBTRAY.EXE can't be used to determine which tickets the Web server has when it's impersonating you. But I've written a helpful sample (included in the download) that allows you to list the tickets and echo them back via an .aspx page (use this while impersonating the client). This will help you determine whether the client's credentials have indeed been forwarded to the middle tier in a delegation scenario.

Q Do you consider it to be dangerous to trust a Web server to delegate credentials?

A That's a very astute observation! (By now, you're probably starting to wonder who is writing these questions aren't you?)

Imagine what happens if the Web server is somehow compromised, and an attacker can run code of his choosing in the Web application's process. The attacker simply has to wait for clients to come along, at which point he can impersonate them and access any machine on the network using their network credentials. Think how awful it would be if a domain administrator from an entirely different domain (perhaps the human resources domain) happened to log onto your Web application during an attack like this. This is one reason why all high-privileged accounts such as domain administrators should be marked with the "sensitive and cannot be delegated" flag in Active Directory. I sure wish that flag could be set for a group (like Domain Admins).

If you have a Windows Server 2003 domain and your Web server is also running on Windows Server 2003, you don't need to trust the Web server nearly this much in order to support delegation. I mentioned earlier that there is a whole new delegation infrastructure in Windows Server 2003, and Figure 5 shows how you can make use of it. Here, instead of choosing the Windows 2000-compatible mode of delegation, I've chosen to use a feature called constrained delegation. This allows me to provide a list of back-end servers to which the Web server should be allowed to delegate credentials. This is actually a list of service principal names, so the one item I've got in the list should look familiar. That's the SPN for SQL Server listening on port 1433 on the machine data.acme.com.

Figure 5 Constrained Delegation in Windows
Figure 5 Constrained Delegation in Windows

With constrained delegation, domain controllers will refuse to issue delegated tickets for any clients, unless the SPN being requested is in the allowed-to-delegate-to list that I've shown in Figure 5. This allows the Web server to use its clients' credentials to talk to SQL Server, but no other servers on the network. This is a very good thing, and if you've upgraded to Windows Server 2003, you should absolutely be using it.

If you're using KERBTRAY.EXE to view the client's tickets, note that under constrained delegation, the Web server's ticket won't be marked ok-as-delegate. This is because constrained delegation works very differently from normal Kerberos TGT forwarding, which is what happens when you use the Windows 2000-compatible delegation option. Under constrained delegation, the client does not forward its TGT to the server, because that would allow the server to use those credentials anywhere on the network. Instead, the client just performs a normal Kerberos handshake with the Web server, and the Web server uses a special extension to Kerberos called S4U2Proxy to obtain a ticket to the back end on the client's behalf. The domain controller will only issue these proxy tickets for SPNs on the Web server's allowed-to-delegate-to list, shown in Figure 5. For more details on the S4U Kerberos extensions, see my April 2003 MSDN®Magazine column at Security Briefs: Exploring S4U Kerberos Extensions in Windows Server 2003.

Just remember, building a distributed, multitier system means making tradeoffs. In this case, the tradeoff is between connection pooling and delegation. While connection pooling might eke a bit more throughput out of your system, delegation means your database doesn't have to trust that Web server so much, and that can be a really good thing if your Web server is ever compromised. Security is never free. But before making a decision one way or another, do some threat modeling and performance profiling, and decide which option is right for you.

Send your questions or comments for Keith to briefs@microsoft.com.

Keith Brown is a co-founder of Pluralsight, specializing in developing and delivering high-quality training for software developers. Keith's most recent book, The .NET Developer's Guide to Windows Security, is available at www.pluralsight.com/keith/book. Keith also keeps a Web log and contact information at www.pluralsight.com/keith.


.NET Matters: Stream Decorator, Single-Instance Apps

.NET Matters
Stream Decorator, Single-Instance Apps
Stephen Toub

Code download available at: NETMatters0509.exe (172KB)

Q I'm using a method from a third-party library that expects me to pass a Stream to it. I want to be able to trace that library's activities with my Stream in order to better understand some behavior in my application. Is there a way to do this?

A You have several options. The first and simplest option is to make use of the classic Decorator design pattern. A decorator is an object that has the same interface as another object it contains. In object-oriented terms, it is an object that has an "is-a" and a "has-a" relationship with a specific type. Consider the CryptoStream class in the System.Security.Cryptography namespace. CryptoStream derives from Stream (it "is-a" Stream), but it also accepts a Stream in its constructor and stores that Stream internally (it "has-a" stream); that underlying stream is where the encrypted data is stored. CryptoStream is a decorator.

You can implement your own Stream-derived decorator class that lets you intercept and monitor each and every call to the stream. An example implementation is shown in Figure 1. The InterceptStream class derives from Stream and overrides all of the methods to be intercepted and traced. It also has a constructor that accepts a Stream as a parameter and stores the specified stream to a private member variable. Each of the overridden methods delegates to the corresponding method on the stored stream, but first makes a call to a delegate with information about the invocation, including what method was called and what the parameters to that call were. One might use InterceptStream as follows:

void DisplayStreamCall(MethodBase method, params object [] arguments)
{
    Console.WriteLine(method.Name + ":");
    foreach(object arg in arguments) Console.Write("\t" + arg);
}
...
Stream s = ... // get the stream to be monitored
s = new InterceptStream(s, new InterceptStreamHandler(
    DisplayStreamCall));
This technique is useful for more than just debugging. For example, it is also useful for tracking progress in an application. Consider an application that uses a BinaryFormatter to deserialize a very large file. Passing an InterceptStream to the formatter instead of the original stream will let you keep track of how much of the target stream has been processed by the formatter, possibly using that information to update a progress bar.

Figure 1 Decorator Pattern with Stream (Close Figure 1)
public delegate void InterceptStreamHandler(
    MethodBase method, params object[]arguments);

public class InterceptStream : Stream
{
    private Stream _stream;
    private InterceptStreamHandler _callback;

    public InterceptStream(Stream s, InterceptStreamHandler callback)
    {
        if (s == null) throw new ArgumentNullException("s");
        if (callback == null) throw new ArgumentNullException("callback");
        _stream = s;
        _callback = callback;
    }

    public override IAsyncResult BeginRead(byte[] buffer, int offset,
        int count, AsyncCallback callback, object state)
    {
        _callback(MethodBase.GetCurrentMethod(), buffer, offset,
            count, callback, state);
        return _stream.BeginRead (buffer, offset, count, callback, state);
    }

    public override IAsyncResult BeginWrite(byte[] buffer, int offset,
        int count, AsyncCallback callback, object state)
    {
        _callback(MethodBase.GetCurrentMethod(), buffer, offset,
            count, callback, state);
        return _stream.BeginWrite (buffer, offset, count,
            callback, state);
    }

    public override bool CanRead
    {
        get
        {
            _callback(MethodBase.GetCurrentMethod());
            return _stream.CanRead;
        }
    }

    ... // other overrides of methods and properties on Stream

    public override string ToString()
    {
        _callback(MethodBase.GetCurrentMethod());
        return _stream.ToString();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        _callback(MethodBase.GetCurrentMethod(), buffer, offset, count);
        _stream.Write(buffer, offset, count);
    }

    public override void WriteByte(byte value)
    {
        _callback(MethodBase.GetCurrentMethod(), value);
        _stream.WriteByte(value);
    }
}

The decorator pattern is a useful and flexible approach, though it is only appropriate when the target type is an interface or is a non-sealed type whose methods you're interested in are virtual and can be overridden. This is the case with Stream. From the client's perspective, using a decorator Stream like InterceptStream won't change the resulting functionality as long as the third-party library is only using the Stream object through the interface provided by Stream. For example, you could run into problems if the library is explicitly checking the actual type of the Stream and is then making some decisions based on that information.

Consider a library that provides some general utility with the stream, but then also performs additional functionality if the stream is a NetworkStream. If you pass to that stream an InterceptStream that wraps a NetworkStream instead of just passing in that NetworkStream, when the library checks the type of the Stream, it'll find that it's an InterceptStream and won't perform the functionality it does specially for the NetworkStream.

There's another option that solves this problem at the expense of some efficiency, and believe it or not it's .NET remoting that comes the rescue. To understand this approach, you need at least a basic understanding of the .NET remoting infrastructure, specifically of proxies. When you access an object across a remoting boundary, you're actually accessing it through at least two proxies. These proxies take the calls you make and forward them on to the target, taking care of any work necessary to properly marshal the call to and the response from the target (based on whether the target is on a different machine, in a different process, in a different application domain, and so on).

The main proxy involved in this operation is one that derives from RealProxy, which lives in the System.Runtime.Remoting.Proxies namespace. This is the proxy that is responsible for actually routing the call information to the target. It is also used to obtain the other proxy, known as a transparent proxy. This proxy is generated for you by the runtime, and through a bit of trickery it has an interface that exactly matches that of the target type. When you hold a reference to a remote object, you're actually holding a reference to a transparent proxy. Information about any invocations on that proxy (be they method calls, property gets and sets, and even field accesses) are packaged up into an internal MessageData object and are passed along to the corresponding RealProxy. The RealProxy translates that MessageData into an IMessage (a public interface that resides in the System.Runtime.Remoting.Messaging namespace) which is then passed to the RealProxy's Invoke method and shuttled off to the target object as appropriate.

I mention all of this because you can actually create your own RealProxy-derived class and explicitly use it with your own objects, even if those objects live in the same application domain as each other. Consider implementing a RealProxy that traces any invocations before and after executing the calls on the actual target object. You could then create one of these proxies to wrap the target object, use the proxy's GetTransparentProxy method to get the corresponding transparent proxy, and then cast that transparent proxy to your target type. Instead of handing out the original reference to the type, you can hand out this proxy in its stead, and any calls through that proxy reference will be traced. While this may sound complicated, the code in Figure 2 is all that's needed to create the necessary proxies.

Figure 2 InterceptionProxy (Close Figure 2)
public delegate void InterceptedMessageHandler(IMessage msg);

public class InterceptionProxy : RealProxy
{
    public static MarshalByRefObject Intercept(MarshalByRefObject target,
        InterceptedMessageHandler pre, InterceptedMessageHandler post)
    {
        if (target == null) throw new ArgumentNullException("target");
        if (pre == null && post == null) return target;
        InterceptionProxy p = new InterceptionProxy(target, pre, post);
        MarshalByRefObject proxy = (MarshalByRefObject)
            p.GetTransparentProxy();
        if (proxy == null) throw new InvalidOperationException(
            "Unable to create a proxy for " + target.GetType());
        return proxy;
    }

    private MarshalByRefObject _instance;
    private InterceptedMessageHandler _pre, _post;

    private InterceptionProxy(MarshalByRefObject target,
        InterceptedMessageHandler pre, InterceptedMessageHandler post) :
        base(target.GetType())
    {
        _instance = target;
        _pre = pre;
        _post = post;
    }

    public override IMessage Invoke(IMessage msg)
    {
        if (_pre != null) _pre(msg);
        msg = RemotingServices.ExecuteMessage(
            _instance, (IMethodCallMessage)msg);
        if (_post != null) _post(msg);
        return msg;
    }
}

To use this class, you simply call InterceptionProxy.Intercept, passing in the target object and two delegates, one of which will be called before a call is passed along to the target and the other which will be called after the call has completed:

fs = (FileStream)InterceptionProxy.Intercept(fs,
    new InterceptedMessageHandler(WritePreMessage),
    new InterceptedMessageHandler(WritePostMessage));

Both delegates accept as a parameter an IMessage representation of information about the invocation; the former's IMessage (typically an instance of System.Runtime.Remoting.Messaging.Message) contains information about the call, while the latter's IMessage (typically an instance of System.Runtime.Remoting.Messaging.ReturnMessage) contains information about the results. Information from each of these objects can be accessed either through the IMessage.Properties collection or through the properties of the more specialized interfaces IMethodCallMessage and IMethodReturnMessage.

The static Intercept method first validates the parameters. If both of the supplied delegates are null, then there's no point in intercepting the call, so the original object is returned rather than a proxy. Assuming the arguments are all valid, a new InterceptionProxy is created to store the supplied data and to serve as the proxy for the target object (note that there's nothing to prevent you from creating two different proxies for the same object). GetTransparentProxy is then called on this real proxy and the result is returned to the caller. Whenever a call is made on this transparent proxy, it'll result in the InterceptionProxy's Invoke method being called with the relevant data stored in the IMessage. This IMessage is then passed to RemotingServices.ExecuteMessage, which takes care of delegating the call to the actual target.

Of course, this approach also has its downsides. First, there is a performance penalty involved with using real proxies for interception. As such, it would be a good idea to take some measurements to determine whether this has a noticeable impact on your solution. Second, as real and transparent proxies are part of the remoting infrastructure, they only work with types that are remotable, specifically those that are marshaled by reference. In other words, this technique only works with types derived from MarshalByRefObject (directly or indirectly). Luckily for your purposes, Stream is one such type, and there are plenty of others in the Framework. A third downside to this approach is that only calls made on the proxy are traced, so if an object hands out a non-proxied reference to itself, any calls made on that non-proxied reference will not be traced. Consider the following class:

class MyClass : MarshalByRefObject
{
    public MyClass GetThis  { get { return this; } }
}
Now consider using this class as follows:
MyClass mc = new MyClass();
mc = (MyClass)InterceptionProxy.Intercept(mc, ...);
Console.WriteLine(RemotingServices.IsTransparentProxy(mc));
Console.WriteLine(RemotingServices.IsTransparentProxy(mc.GetThis));
RemotingServices.IsTransparentProxy inspects the object passed to it and returns whether the object is a transparent proxy. The previous code snippet will thus print out "True" and "False", as the mc reference is a proxy but the reference it leaked through its GetThis property is not. Still, in many situations this technique can be a very handy tool for your code utility belt.

Q I'm writing a Windows® Forms application with Visual Basic® and would like to ensure that a user can only have one instance of it open at any one time. Additionally, if the user tries to start a new instance of that application, I'd like the currently running instance to be notified with the command-line arguments passed to the secondary instance. What's the easiest way to do this?

A Basic support for single-instance applications is fairly straightforward, and there are a variety of approaches to handling this. All of them involve some sort of shared resource that instances of the application can create and lock. If a second instance of an application is not successful in doing so, it knows that a previous instance already exists and that it should exit. The second requirement you pose of passing the command-line arguments to the initial instance is more complicated, so I'll address that later.

The most common shared resource used in managed applications for the creation of single-instance applications is a mutex. A mutex is a synchronization primitive provided by the operating system that allows for interthread and interprocess communication. A mutex is typically used to protect a shared resource, providing exclusive access to it.

When one thread acquires the mutex, no other threads will be able to acquire it until the thread that did so originally releases it. As such, applications can use a mutex to provide single-instance functionality. On startup, the application will attempt to create and acquire a mutex with a known name. If it can, then it is the first instance. If it can't, then another instance of the application has already created and acquired the mutex, thus alerting this new instance to the fact that it is a duplicate and should exit.

In the .NET Framework, a mutex is represented through the System.Threading.Mutex class. This class derives from WaitHandle and is a thin wrapper around the kernel's mutex object, which can be created using the Win32® CreateMutex function and opened using the Win32 OpenMutex function.

First, you need to come up with a name for the mutex. The use of a name allows multiple instances of the application to create or access the same mutex instance, so choosing a unique name ahead of time is important. Some typical examples of names used for this purpose include the name of the entry assembly for the application (available from [Assembly].GetEntryAssembly().FullName), the entry assembly's type library GUID (accessed with Marshal.GetTypeLibGuidForAssembly), or a predefined GUID used to uniquely identify the application. The name can also be autogenerated at runtime, but the algorithm for doing so must be created in such a way that all instances of the same application will generate the same name. With the name in hand, a mutex is instantiated using the constructor that accepts three arguments. Here is an example of using a mutex in a Visual Basic .NET application to ensure a single-instance of that application:

Sub Main()
    Dim mutexName As String = "3af65962-66bf-4135-bc5a-b0f986d8e995"
    Dim grantedOwnership As Boolean
    Dim singleInstanceMutex As Mutex = _
        New Mutex(True, mutexName, grantedOwnership)
    Try
        If Not grantedOwnership Then Return
        ... ' Main application code here
    Finally
        singleInstanceMutex.Close()
    End Try
End Sub

The first parameter to the constructor is a Boolean value that indicates whether the application wants to immediately acquire the mutex once it's created, and the third parameter is a Boolean passed by reference that upon completion of the constructor will store whether the created mutex was acquired. The second parameter is the name for the mutex. Once the mutex has been created, the grantedOwnership parameter can be checked to see whether the mutex was acquired by this instance of the application. If it was, then it is the first instance and the application should proceed normally. If it wasn't, then another instance of this application already exists, and this new instance should exit. It should be noted, however, that there is a possible denial of service (DoS) attack lurking here. Typically when one thinks of DoS attacks, images of Web applications come to mind. But a DoS attack is anything that prevents usage of a service, of any kind. In this case, a malicious party can learn the name of the mutex used to provide this single-instance functionality and can acquire the mutex first, thus forcing any instances of the application to act as if they're not the first instance and to close down. Just something to keep in mind.

That answers the part of your question about providing basic single-instance functionality, but it doesn't address how command-line arguments can be passed from a secondary instance to the primary instance. This is a harder problem as it involves more significant interprocess communication. One possibility is to use .NET remoting to communicate these parameters between instances. The primary instance can start a remoting server to listen for any other instances' command-line arguments. When another instance determines that it is not the first, it can connect to the primary's remoting server and pass to it the command-line arguments; the primary instance can then react to these command-line arguments in whatever way it sees fit.

So while basic support for single-instance applications as described earlier is fairly easy, a robust implementation is harder to come by. There are lots of issues involved: securing the connection between instances, mitigating the aforementioned DoS attack through liberal use of access control lists ( ACLs), choosing a good wait handle name, and so on. Luckily, the nice folks on the Visual Basic team have already done the hard work for you.

Figure 3 Single-Instance Windows Forms Application in Visual Basic 2005
Figure 3 Single-Instance Windows Forms Application in Visual Basic 2005

In Visual Basic 2005, creating a single-instance Windows Forms application is as easy as clicking a checkbox in the properties page for the application (shown in Figure 3). Then in the ApplicationEvents.vb file, you can handle the Me.StartupNextInstance event:

Namespace My
    Class MyApplication
        Private Sub MyApplication_StartupNextInstance( _
                ByVal sender As Object, _
                ByVal e As StartupNextInstanceEventArgs) _
                Handles Me.StartupNextInstance
            ... ' use e.CommandLine here
        End Sub
    End Class
End Namespace

The StartupNextInstanceEventArgs object passed to this event handler contains the command-line arguments from secondary instances of the application, and you can process them here however you see fit. And while the command-line arguments are received on a background thread, Visual Basic marshals the call to this event to the GUI thread. Under the covers, Visual Basic is using a technique similar to the one described earlier, though it is much more robust and secure.

Rather than using a mutex, Visual Basic uses an EventWaitHandle but in a similar way and with the same purpose (EventWaitHandle is the new base class in the .NET Framework 2.0 for AutoResetEvent and ManualResetEvent). The name for this wait handle is derived from the entry assembly's type library GUID and assembly version. If the instance of the wait handle does not already exist, Visual Basic sets up a remoting server and a secure channel through which to receive command-line arguments from other instances of the app. When command-line arguments are received, Visual Basic raises the StartupNextInstance event as was already described. If, however, the EventWaitHandle already exists, instead of creating a remoting server, Visual Basic connects to the existing instance's remoting server and passes to it the command-line arguments from the secondary instance. In order to enable discovery of the URI of the remoting server, the initial instance writes the URI to a memory- mapped file with a known name (similar to the name used for the event); secondary instances can then retrieve the URI from this location and correctly connect to the remoting server.

While this single-instance application functionality is easily accessible to developers using Visual Basic, the kind folks on the Visual Basic team have implemented this capability in such a way that other languages can also take advantage of it. Here I'll demonstrate how to create single-instance Windows Forms apps in C#.

All of the functionality just described is available from the WindowsFormsApplicationBase class as part of the Microsoft.VisualBasic.ApplicationServices namespace in Microsoft.VisualBasic.dll. If you examine in ildasm.exe a single-instance application created with Visual Basic, you'll find a class called MyApplication that derives from WindowsFormsApplicationBase. This class's constructor sets various protected properties on the base class, including one called IsSingleInstance. You can do the same thing in C#.

Additionally, the WindowsFormsApplicationBase class provides a public property MainForm which references the form instance to load and display at startup, and it provides the public event StartupNextInstance you've already seen used. You can put all of that together to create the class shown in Figure 4. Then, in the Main method of the application, instead of using the default code to start up the main form, as shown in the top half of Figure 5, you can use code like that in the bottom half, and you've got yourself a robust single-instance Windows Forms application in C#. Of course, you'll need to add a reference to Microsoft.VisualBasic.dll and import the appropriate namespace for this to work.

Figure 5 Using SingleInstanceApplication (Close Figure 5)

Default Startup Code
static class Program
{
    [STAThread]
    static void Main()
    {
        Application.EnableVisualStyles();
        Application.Run(new Form1());
    }
}
Single-Instance Startup Code
static class Program
{
    [STAThread]
    static void Main()
    {
        Application.EnableVisualStyles();
        SingleInstanceApplication.Run(new Form1(),
            StartupNextInstanceHandler);
    }

    static void StartupNextInstanceHandler(
        object sender, StartupNextInstanceEventArgs e)
    {
        // do whatever you want here with e.CommandLine...
    }
}
Figure 4 SingleInstanceApplication in C# (Close Figure 4)
public class SingleInstanceApplication : WindowsFormsApplicationBase
{
    private SingleInstanceApplication()
    {
        base.IsSingleInstance = true;
    }

    public static void Run(Form f,
        StartupNextInstanceEventHandler startupHandler)
    {
        SingleInstanceApplication app = new SingleInstanceApplication();
        app.MainForm = f;
        app.StartupNextInstance += startupHandler;
        app.Run(Environment.GetCommandLineArgs());
    }
}

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


C++ at Work: Copy Constructors, Assignment Operators, and More

C++ At Work
Copy Constructors, Assignment Operators, and More
Paul DiLascia

Code download available at: CAtWork0509.exe (276KB)

Q I have a simple C++ problem. I want my copy constructor and assignment operator to do the same thing. Can you tell me the best way to accomplish this?

Shadi Hani

A At first glance this seems like a simple question with a simple answer: just write a copy constructor that calls operator=.

CFoo::CFoo(const CFoo& obj)
{
   *this = obj;
}
Or, alternatively, write a common copy method and call it from both your copy constructor and operator=, like so:
CFoo::CFoo(const CFoo& obj)
{
   CopyObj(obj);
}
CFoo& CFoo::operator=(const CFoo& rhs)
{
  CopyObj(rhs);
  return *this;
}

This code works fine for many classes, but there's more here than meets the eye. In particular, what happens if your class contains instances of other classes as members? To find out, I wrote the test program in Figure 1. It has a main class, CMainClass, which contains an instance of another class, CMember. Both classes have a copy constructor and assignment operator, with the copy constructor for CMainClass calling operator= as in the first snippet. The code is sprinkled with printf statements to show which methods are called when. To exercise the constructors, cctest first creates an instance of CMainClass using the default ctor, then creates another instance using the copy constructor:

CMainClass obj1;
CMainClass obj2(obj1);

Figure 1 Copy Constructors and Assignment Operators (Close Figure 1)
// cctest.cpp: Simple program to illustrate a problem calling
// operator= from copy constructor. This can result in member
// objects getting initialized twice. To compile: cl cctest.cpp
//
#include <stdio.h>

//#define GOOD

//////////////////
// Class used as member of main class.
//
class CMember {
public:
   CMember()
   {
      printf(" CMember: default ctor\n");
   }

   CMember(const CMember&)
   {
      printf(" CMember: copy-ctor\n");
   }

   CMember& operator=(const CMember&) {
      printf(" CMember: operator=\n");
      return *this;
   }
};

//////////////////
// Main class
//
class CMainClass {
protected:
   CMember m_obj;

public:
   CMainClass()
   {
      printf(" CMainClass: default ctor\n");
   }

#ifndef GOOD
   // Copy-ctor calls operator=. This is not generally recommended.
   //
   CMainClass(const CMainClass& rhs)
   {
      printf(" CMainClass: copy-ctor\n");
      *this = rhs;
   }
#else
   // Below is a better way to initialize m_obj.
   //
   CMainClass(const CMainClass& rhs) : m_obj(rhs.m_obj)
   {
      printf(" CMainClass: copy-ctor\n");
   }
#endif

   CMainClass& operator=(const CMainClass& rhs)
   {
      printf(" CMainClass: operator=\n");
      m_obj = rhs.m_obj;
      return *this;
   }
};

int main()
{
   printf("Creating instance with default ctor:\n");
   CMainClass obj1;

   printf("Creating instance with copy-ctor:\n");
   CMainClass obj2(obj1);

   printf("Performing assignment:\n");
   obj2 = obj1;
}

If you compile and run cctest, you'll see the following printf messages when cctest constructs obj2:

CMember: default ctor
CMainClass: copy-ctor
CMainClass: operator=
CMember: operator=

The member object m_obj got initialized twice! First by the default constructor, and again via assignment. Hey, what's going on?

In C++, assignment and copy construction are different because the copy constructor initializes uninitialized memory, whereas assignment starts with an existing initialized object. If your class contains instances of other classes as data members, the copy constructor must first construct these data members before it calls operator=. The result is that these members get initialized twice, as cctest shows. Got it? It's the same thing that happens with the default constructor when you initialize members using assignment instead of initializers. For example:

CFoo::CFoo()
{
   m_obj = DEFAULT;
}
As opposed to:
CFoo::CFoo() : m_obj(DEFAULT)
{
}

Using assignment, m_obj is initialized twice; with the initializer syntax, only once. So, what's the solution to avoid extra initializations during copy construction? While it goes against your instinct to reuse code, this is one situation where it's best to implement your copy constructor and assignment operator separately, even if they do the same thing. Calling operator= from your copy constructor will certainly work, but it's not the most efficient implementation. My observation about initializers suggests a better way:

CFoo::CFoo(const CFoo& rhs) : m_obj(rhs.m_obj) {}

Now the main copy ctor calls the member object's copy ctor using an initializer, and m_obj is initialized just once by its copy ctor. In general, copy ctors should invoke the copy ctors of their members. Likewise for assignment. And, I may as well add, the same goes for base classes: your derived copy ctor and assignment operators should invoke the corresponding base class methods. Of course, there are always times when you may want to do something different because you know how your code works—but what I've described are the general rules, which are to be broken only when you have a compelling reason. If you have common tasks to perform after the basic objects have been initialized, you can put them in a common initialization method and call it from your constructors and operator=.

Q Can you tell me how to call a Visual C++® class from C#, and what syntax I need to use for this?

Sunil Peddi

Q I have an application that is written in both C# (the GUI) and in classic C++ (some business logic). Now I need to call from a DLL written in C++ a function (or a method) in a DLL written in Visual C++ .NET. This one calls another DLL written in C#. The Visual C++ .NET DLL acts like a proxy. Is this possible? I was able to use LoadLibrary to call a function present in the Visual C++ .NET DLL, and I can receive a return value, but when I try to pass some parameters to the function in the Visual C++ .NET DLL, I get the following error:

Run-Time Error Check Failure #0—The value of ESP was not properly saved
across a function call. This is usually a result of calling a function
declared with one calling convention with a function pointer declared
with a different calling convention.
How can I resolve this problem?

Giuseppe Dattilo

A I get a lot of questions about interoperability between the Microsoft® .NET Framework and native C++, so I don't mind revisiting this well-covered topic yet again. There are two directions you can go: calling the Framework from C++ or calling C++ from the Framework. I won't go into COM interop here as that's a separate issue best saved for another day.

Let's start with the easiest one first: calling the Framework from C++. The simplest and easiest way to call the Framework from your C++ program is to use the Managed Extensions. These Microsoft-specific C++ language extensions are designed to make calling the Framework as easy as including a couple of files and then using the classes as if they were written in C++. Here's a very simple C++ program that calls the Framework's Console class:

#using <mscorlib.dll>
#using <System.dll> // implied
using namespace System;
void main()
{
  Console::WriteLine("Hello, world");
}

To use the Managed Extensions, all you need to do is import <mscorlib.dll> and whatever .NET assemblies contain the classes you plan to use. Don't forget to compile with /clr:

cl /clr hello.cpp

Your C++ code can use managed classes more or less as if they were ordinary C++ classes. For example, you can create Framework objects with operator new, and access them using C++ pointer syntax, as shown in the following:

DateTime d = DateTime::Now;
String* s = String::Format("The date is {0}\n", d.ToString());
Console::WriteLine(s);
Console::WriteLine(s->Length);
Here, the String s is declared as pointer-to-String because String::Format returns a new String object.

The "Hello, world" and date/time programs seem childishly simple—and they are—but just remember that however complex your program is, however many .NET assemblies and classes you use, the basic idea is the same: use <mscorlib.dll> and whatever other assemblies you need, then create managed objects with new, and use pointer syntax to access them.

So much for calling the Framework from C++. What about going the other way, calling C++ from the Framework? Here the road forks into two options, depending on whether you want to call extern C functions or C++ class member functions. Again, I'll take the simpler case first: calling C functions from .NET. The easiest thing to do here is use P/Invoke. With P/Invoke, you declare the external functions as static methods of a class, using the DllImport attribute to specify that the function lives in an external DLL. In C# it looks like this:

public class Win32 {
   [DllImport("user32.dll", CharSet=CharSet.Auto)]
   public static extern int MessageBox(IntPtr hWnd, String text,
      String caption, int type);
}
This tells the compiler that MessageBox is a function in user32.dll that takes an IntPtr (HWND), two Strings, and an int. You can then call it from your C# program like so:
Win32.MessageBox(0, "Hello World", "Platform Invoke Sample", 0);

Of course, you don't need P/Invoke for MessageBox since the .NET Framework already has a MessageBox class, but there are plenty of API functions that aren't supported directly by the Framework, and then you need P/Invoke. And, of course, you can use P/Invoke to call C functions in your own DLLs. I've used C# in the example, but P/Invoke works with any .NET-based language like Visual Basic® .NET or JScript®.NET. The names are the same, only the syntax is different.

Note that I used IntPtr to declare the HWND. I could have got away with int, but you should always use IntPtr for any kind of handle such as HWND, HANDLE, or HDC. IntPtr will default to 32 or 64 bits depending on the platform, so you never have to worry about the size of the handle.

DllImport has various modifiers you can use to specify details about the imported function. In this example, CharSet=CharSet.Auto tells the Framework to pass Strings as Unicode or Ansi, depending on the target operating system. Another little-known modifier you can use is CallingConvention. Recall that in C, there are different calling conventions, which are the rules that specify how the compiler should pass arguments and return values from one function to another across the stack. The default CallingConvention for DllImport is CallingConvention.Winapi. This is actually a pseudo-convention that uses the default convention for the target platform; for example, StdCall (in which the callee cleans the stack) on Windows® platforms and CDecl (in which the caller cleans the stack) on Windows CE .NET. CDecl is also used with varargs functions like printf.

The calling convention is where Giuseppe ran into trouble. C++ uses yet a third calling convention: thiscall. With this convention, the compiler uses the hardware register ECX to pass the "this" pointer to class member functions that don't have variable arguments. Without knowing the exact details of Giuseppe's program, it sounds from the error message that he's trying to call a C++ member function that expects thiscall from a C# program that's using StdCall—oops!

Aside from calling conventions, another interoperability issue when calling C++ methods from the Framework is linkage: C and C++ use different forms of linkage because C++ requires name-mangling to support function overloading. That's why you have to use extern "C" when you declare C functions in C++ programs: so the compiler won't mangle the name. In Windows, the entire windows.h file (now winuser.h) is enclosed in extern "C" brackets.

While there may be a way to call C++ member functions in a DLL directly using P/Invoke and DllImport with the exact mangled names and CallingConvention=ThisCall, it's not something to attempt if you're in your right mind. The proper way to call C++ classes from managed code—option number two—is to wrap your C++ classes in managed wrappers. Wrapping can be tedious if you have lots of classes, but it's really the only way to go. Say you have a C++ class CWidget and you want to wrap it so .NET clients can use it. The basic formula looks something like this:

public __gc class Widget
{
private:
  CWidget* m_pObj; // ptr to native object
public:
  Widget()  { m_pObj = new CWidget; }
  ~Widget() { delete m_pObj; }
  int Method(int n) { return m_pObj->Method(n); }
  // etc.
};

The pattern is the same for any class. You write a managed (__gc) class that holds a pointer to the native class, you write a constructor and destructor that allocate and destroy the instance, and you write wrapper methods that call the corresponding native C++ member functions. You don't have to wrap all the member functions, only the ones you want to expose to the managed world.

Figure 2 shows a simple but concrete example in full detail. CPerson is a class that holds the name of a person, with member functions GetName and SetName to change the name. Figure 3 shows the managed wrapper for CPerson. In the example, I converted Get/SetName to a property, so .NET-based programmers can use the property syntax. In C#, using it looks like this:

// C# client
MPerson.Person p = new MPerson.Person("Fred");
String name = p.Name;
p.Name = "Freddie";

Figure 3 Managed Person Class (Close Figure 3)
// Managed Wrapper for native CPerson class. Compile with /clr. (Managed
// Extensions.)
//
#include <afxwin.h>
#include "person.h"

#using <mscorlib.dll>
#include <vcclr.h>
using namespace System;

// Should have Assembly Information here, but this is just a lame demo
// program.
//
//[assembly: AssemblyTitle("MPerson")]
//[assembly: AssemblyDescription("MPerson wrappper")]
//etc.

// Every assembly needs a namespace for managed code to use.
namespace MPerson {

//////////////////
// Managed wrapper for native CPerson class
//
public __gc class Person
{
private:
    CPerson* m_pPerson; // ptr to native object

    // Common construction allocates native object
    void Construct()
    {
        m_pPerson = new CPerson("");
    }

    // private because CPerson has no default ctor
    Person()
    {
        Construct();
    }

public:
    Person(String* n)
    {
        Construct();
        Name = n; // invoke property assignment
    }

    ~Person()
    {
        delete m_pPerson;
    }

    // Convert native Get/SetName functions to CLR properties.
    __property String* get_Name()
    {
        return m_pPerson->GetName();
    }

    __property void set_Name(String* n)
    {
        const WCHAR  __pin* str = PtrToStringChars(n);
        USES_CONVERSION;
        LPCTSTR lpsz = W2T(str);
        m_pPerson->SetName(lpsz);
    }
};

}
Figure 2 Native CPerson Class (Close Figure 2)

person.h
#include <windows.h>
#include <string>
using namespace std;

#ifndef DLLCLASS
#define DLLCLASS __declspec(dllimport)
#endif

#ifdef _UNICODE
#define tstring wstring
#else
#define tstring string
#endif

//////////////////
// Native C++ class
//
class DLLCLASS CPerson {
protected:
    tstring m_name;
public:
    CPerson(LPCTSTR n);
    ~CPerson();
    LPCTSTR GetName()           { return m_name.c_str(); }
    void SetName(LPCTSTR n) { m_name = n; }
};
person.cpp
#define DLLCLASS __declspec(dllexport)
#include "person.h"

////////////////////////////////////////////////////////////////
// Implementation of Person class
//
CPerson::CPerson(LPCTSTR n) : m_name(n)
{
}

CPerson::~CPerson()
{
}

Using properties is purely a matter of style; I could equally well have exposed two methods, GetName and SetName, as in the native class. But properties feel more like .NET. The wrapper class is an assembly like any other, but one that links with the native DLL. This is one of the cool benefits of the Managed Extensions: You can link directly with native C/C++ code. If you download and compile the source for my CPerson example, you'll see that the makefile generates two separate DLLs: person.dll implements a normal native DLL and mperson.dll is the managed assembly that wraps it. There are also two test programs: testcpp.exe, a native C++ program that calls the native person.dll and testcs.exe, which is written in C# and calls the managed wrapper mperson.dll (which in turn calls the native person.dll).

Figure 4 Interop Highway
Figure 4 Interop Highway

I've used a very simple example to highlight the fact that there are fundamentally only a few main highways across the border between the managed and native worlds (see Figure 4). If your C++ classes are at all complex, the biggest interop problem you'll encounter is converting parameters between native and managed types, a process called marshaling. The Managed Extensions do an admirable job of making this as painless as possible (for example, automatically converting primitive types and Strings), but there are times where you have to know something about what you're doing.

For example, you can't pass the address of a managed object or subobject to a native function without pinning it first. That's because managed objects live in the managed heap, which the garbage collector is free to rearrange. If the garbage collector moves an object, it can update all the managed references to that object—but it knows nothing of raw native pointers that live outside the managed world. That's what __pin is for; it tells the garbage collector: don't move this object. For strings, the Framework has a special function PtrToStringChars that returns a pinned pointer to the native characters. (Incidentally, for those curious-minded souls, PtrToStringChars is the only function as of this date defined in <vcclr.h>. Figure 5 shows the code.) I used PtrToStringChars in MPerson to set the Name (see Figure 3).

Figure 5 PtrToStringChars (Close Figure 5)
// PtrToStringChars, from vcclr.h

// get an interior gc pointer to the first character contained in a
// System::String object
//
inline const System::Char * PtrToStringChars(const System::String *s) {
    const System::Byte *bp = reinterpret_cast<const System::Byte *>(s);
    if( bp != 0 ) {
        unsigned offset = System::Runtime::CompilerServices::
            RuntimeHelpers::OffsetToStringData;
        bp += offset;
    }
    return reinterpret_cast<const System::Char*>(bp);
}

Pinning isn't the only interop problem you'll encounter. Other problems arise if you have to deal with arrays, references, structs, and callbacks, or access a subobject within an object. This is where some of the more advanced techniques come in, such as StructLayout, boxing, __value types, and so on. You also need special code to handle exceptions (native or managed) and callbacks/delegates. But don't let these interop details obscure the big picture. First decide which way you're calling (from managed to native or the other way around), and if you're calling from managed to native, whether to use P/Invoke or a wrapper.

In Visual Studio® 2005 (which some of you may already have as beta bits), the Managed Extensions have been renamed and upgraded to something called C++/CLI. Think of the C++/CLI as Managed Extensions Version 2, or What the Managed Extensions Should Have Been. The changes are mostly a matter of syntax, though there are some important semantic changes, too. In general C++/CLI is designed to highlight rather than blur the distinction between managed and native objects. Using pointer syntax for managed objects was a clever and elegant idea, but in the end perhaps a little too clever because it obscures important differences between managed and native objects. C++/CLI introduces the key notion of handles for managed objects, so instead of using C pointer syntax for managed objects, the CLI uses ^ (hat):

// handle to managed string
String^ s = gcnew String;

As you no doubt noticed, there's also a gcnew operator to clarify when you're allocating objects on the managed heap as opposed to the native one. This has the added benefit that gcnew doesn't collide with C++ new, which can be overloaded or even redefined as a macro. C++/CLI has many other cool features designed to make interoperability as straightforward and intuitive as possible.

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: Hacking the Immune System

{ End Bracket }
Hacking the Immune System
Nebojsa Jojic and David Heckerman

HIV is a major global health problem, with over 40 million people infected worldwide. Biomedical researchers are working to create a vaccine that would be effective against many different strains, taking into account all the individual differences in people's immune systems. Viral RNA sequences have been extracted from thousands of blood samples from infected patients. Each sequence is a string of about 9000 nucleotides serving as a program for viral assembly. Parts of patients' own genomes are sequenced in order to categorize their immune types, which are known to affect immune response and thus viral diversity through natural selection.

The resulting datasets present immense difficulty to biomedical researchers. The genetic code defies visualization in much the same way that computer binary code would without the software tools to parse, convert, and edit it in a higher language. But, as opposed to the binary code which was created by humans, the genetic code follows rules that are not fully understood. Therefore, researchers are turning to modern data-mining techniques to discover patterns of viral evolution. The problem of parsing the structure of the genetic sequences (RNA/DNA or proteins), is similar to the problem of parsing the structure of other natural signals, such as images or sounds. At Microsoft Research, we have been working on machine-learning and data-mining algorithms that aim to automatically parse such data for uses relevant to Microsoft, such as video/image browsing, spam filtering, and speech recognition. These same techniques can be applied to medicine.

The human immune system acts in many different ways to control infection. One arm of the immune system recognizes short (8-11 long) amino-acid sub-sequences in the viral protein called epitopes. Many human cells have the ability to send such short pieces of various internal proteins to the surface of the cell, where these markers representing viral presence in the cell are exposed to recognition by killer T-cells. Killer cells destroy the cells that expose foreign peptides on their surface, and usually ignore those that only show pieces of normal human protein. Being sensitive only to these short signatures (hashes, if you will) makes the immune system resource-effective, but provides an opportunity for the virus to evade immune pressure by mutating to present proteins that look like human protein sequences on their surface. The goal of vaccination is to elicit immune response to as many epitopes as possible, so the virus can't evade detection. The problem with putting a lot of epitopes into a vaccine is that long vaccines can cause illness. To shorten the vaccine, epitope overlap can be exploited.

Several years ago, we introduced a new representation of images, which we call epitome, and the associated learning algorithms, which have a very similar flavor. Epitome was used as a basis of image parsing and recognition tasks (www.research.microsoft.com/~jojic/epitome.html) and later also for analysis of text, sound, video, and genes. While the image epitome organizes visual memory, the HIV epitome serves as an artificial organization of cellular memory, which is delivered as a vaccine into the cells. Once delivered, the cells chop up and present the pieces of the epitome on their surface, triggering the response from killer cells which then turn into memory cells capable of rapidly extinguishing the infection by strains containing these patterns. (For an illustration, see the protein epitome link at www.research.microsoft.com/~jojic).

Each individual will learn different viral targets (epitopes) from exposure to the vaccine. When these targets are unknown, one approach is to treat every possible 8-11 amino acid pattern in every known viral strain as a potential epitope and fit them all into a short epitome. The overlap of epitopes within one strain and the repetition of these short patterns across different strains make it possible to capture over 70 percent of viral variability with epitomes that are just 5-10 times longer than a single strain. The immense diversity of the immune systems is complicating this research and there again arises a need for machine-learning techniques. We are working with scientists from Harvard University, Massachusetts General Hospital, University of Washington, Los Alamos National Laboratory, and The Royal Perth Hospital in Australia on the problem of discovering epitopes. The machine-learning techniques involved are essentially the same as the ones we used in other predictors, for example in the spam filters. We are also currently working with some of these scientists on lab experiments that should verify the efficacy of epitomes in binding assays. These experiments will show us where the T-cells taken from infected patients bind on the epitomes constructed.

Nebojsa Jojic, Ph.D. works at Microsoft Research, where he has conducted research in the areas of computer vision, computational biology, signal processing and machine learning.

David Heckerman, Ph.D, M.D. is founder and manager of the Machine Learning and Applied Statistics Group at Microsoft Research.