 MSDN Magazine January 2008

MSDN Magazine January 2008

IIS 7.0: Enhance Your Apps with the Integrated ASP.NET Pipeline

Mike Volodarsky demonstrates how IIS 7.0 lets you add performance and security upgrades to PHP apps without touching a line of PHP code.

Mike Volodarsky World Ready: Around the World with ASP.NET AJAX Applications

The .NET Framework has excellent internationalization support, but JavaScript does not. If you're using ASP.NET AJAX, learn what you need to do to adapt.

Guy Smith-Ferrier WCF Syndication: HTTP Programming with WCF and the .NET Framework 3.5

Introducing Web-centric features of Windows Communication Foundation in the .NET Framework 3.5, including the HTTP programming model and the new syndication API.

Justin Smith SQL Server: Uncover Hidden Data to Optimize Application Performance

SQL Server 2005 gathers a lot of useful data that you can use to identify areas where database performance can be improved. Here's what you need to improve performance.

Ian Stirk Look it Up: Managing Directory Security Principals in the .NET Framework 3.5

Here's an overview of the new System.DirectoryServices.AccountManagement class in the .NET Framework 3.5 and how it simplifies working with directory services.

Joe Kaplan and Ethan Wilansky Editor's Note: A New Year—A Fresh Perspective

The results of our reader survey are in. This month, Howard Dierking shares the takeaways.

Howard Dierking Toolbox: Advanced Unit Testing, Object Mocking, Profiling Your Apps and More

This month we look at advanced unit testing, object mocking, profiling, and lots more.

James Avery CLR Inside Out: Marshaling between Managed and Unmanaged Code

Marshaling is an important part of .NET interop. It allows you to call into unmanaged code from managed code. This column will help you get started.

Yi Zhang and Xiaoying Guo Data Points: Accessing Data from a Mobile Application

This month John Papa takes a look at developing a mobile application that can access data on your application server.

John Papa Advanced Basics: Reflection and Generics

This month Advanced Basics flaunts the power of generics and reflection and shows how you get more flexible and efficient development by combining the two.

Ken Getz Cutting Edge: Customize Controls with AJAX Extenders

AJAX Extenders extend the behavior and features of ordinary Web controls so you can reduce postbacks and control input even better than with AJAX alone.

Dino Esposito Wicked Code: Drag and Drop with ASP.NET AJAX

Jeff Prosise shows how you can implement drag-and-drop functionality in your Web app with ASP.NET AJAX.

Jeff Prosise Foundations: Templates for Uncommon Controls

Charles Petzold demonstrates how Windows Presentation Foundation (WPF) templates let you turn common controls into uncommon visual objects.

Charles Petzold Extreme ASP.NET: Encapsulate Silverlight with ASP.NET Controls

To implement Silverlight in ASP.NET pages, you can encapsulate your Silverlight elements in ASP.NET controls. Here's how.

Fritz Onion { End Bracket }: Passion for the Craft

Scott Hanselman ruminates on the term "5:01 developer" and why success in any field requires passion and craftsmanship.

Scott Hanselman


IIS 7.0: Enhance Your Apps with the Integrated ASP.NET Pipeline

IIS 7.0

Enhance Your Apps with the Integrated ASP.NET Pipeline

Mike Volodarsky
This article is based on a prerelase version of the IIS 7.0 FastCGI component. All information herein is subject to change.
Code download available at: PHPandIIS2008_01.exe (171KB)
This article discusses:
  • ASP.NET Integrated mode
  • Adding user authentication
  • Enabling search-engine-friendly URLs
  • Upgrading performance with output caching
This article uses the following technologies:
IIS 7.0, .NET Framework
Contents
IIS and PHP
Setting Up the App
Securing the Gallery
Forms Authentication
Search-Engine-Friendly URLs
Generating Friendly Hyperlinks
Testing Performance
ASP.NET Output Caching
Deployment and Testing
Wrapping Up

A
lmost a year ago I wrote an overview of IIS 7.0 in MSDN® Magazine (see "IIS 7.0: Explore the Web Server for Windows Vista and Beyond" at msdn.microsoft.com/msdnmag/issues/07/03/IIS7). That was several months before IIS 7.0 was released with Windows Vista®. Since then, users have had a chance to experience the new IIS 7.0 componentized and extensible architecture and other improvements firsthand.

In the time since Windows Vista shipped, we've been hard at work making sure IIS 7.0 will be a reliable and secure Web server in Windows Server® 2008, honing its stability, performance, and support for hosting environments. We've also taken a hard look at what it means for IIS 7.0 to be a flexible Web application platform. Besides being an excellent platform for Microsoft application frameworks like ASP and ASP.NET, we wanted it to be a great platform for a wide variety of other application frameworks in use today. To facilitate this, we added support for FastCGI, an open Web server standard that allows application frameworks such as PHP, Ruby on Rails, and Perl to be hosted on IIS. We also collaborated with Zend Technologies, the creator of PHP, to bring a solid, high-performance PHP implementation to Windows® and IIS.

IIS 7.0 goes beyond simply providing production support for popular application frameworks. With its new Microsoft® .NET Framework extensibility model, IIS 7.0 can leverage the power of ASP.NET integrated mode to add key functionality to applications developed with any framework. This can enable you to add cool features such as access control or new URL schemes, or dramatically improve performance, often without touching a single line of code.

In this article, I am going to dive deep into the IIS 7.0 ASP.NET integration features to enhance an application that was not developed with ASP.NET in mind. I'll show you how it's possible to both augment an application with existing ASP.NET features and add to the app by developing new features that take advantage of IIS-level ASP.NET extensibility.

The application in question is a popular PHP image gallery application called Qdig (qdig.sourceforge.net). Without so much as touching a single line of PHP code, I'll show you how to add some handy new features to the image gallery. First, I will password-protect the gallery using the ASP.NET membership and forms authentication features. I will also upgrade it to use search-engine-friendly URLs instead of unsightly, parameterized query-string URLs. Finally, I will use the ASP.NET output cache to dramatically improve the performance of the application.

But first, let's look at some history behind the new PHP support in IIS, which is at the core of allowing application frameworks like PHP to enjoy the full set of IIS 7.0 features.

IIS and PHP

IIS had always supported PHP, but in a way that precluded many real-life PHP applications from being hosted in production environments. This was due to limitations in the two ways IIS offered for running PHP applications: using the Common Gateway Interface (CGI) protocol or using the PHP ISAPI extension.

Because CGI requires a separate process for each request, apps hosted using CGI would perform poorly on Windows. Conversely, PHP apps using the IIS high-performance multithreaded ISAPI interface would often suffer from instability due to the lack of thread safety in some popular PHP extensions.

In an attempt to solve these problems, the IIS team developed the FastCGI component. The open FastCGI protocol allows PHP and many other application frameworks that require a single-threaded environment (including Ruby on Rails, Perl, and Python) to run more reliably on IIS. Unlike the standard CGI implementation, FastCGI enables process reuse by maintaining a pool of worker processes, each processing no more than one request at a time, thus resulting in much-improved performance. FastCGI also benefitted from a community-centric development and testing model.

At the same time, Zend Technologies worked to improve the general performance and stability of the PHP scripting engine and core extensions on Windows, making numerous performance improvements and fixing dozens of Windows-specific bugs.

As I am writing this, php.net/downloads offers PHP 5.2.3, the third release of PHP optimized for Windows hosting, with fast non-thread-safe builds optimized for the IIS FastCGI platform. The IIS FastCGI support is built into Windows Server 2008 starting with the Beta 3 release, and is available as a separate Tech Preview download for Windows Vista, Windows XP, and a Go-Live release for Windows Server 2003 (iis.net/fastcgi). When Windows Vista SP1 ships, this component will also be available as part of the installation package. In the near future, final versions for Windows XP and Windows Server 2003 will be released as well. You can read more about the options you have for running IIS FastCGI today at mvolo.com/blogs/serverside/archive/2007/10/09/IIS-FastCGI-and-PHP_3A00_-What-you-absolutely-need-to-know-to-host-PHP-applications-on-IIS-6-and-IIS-7.aspx.

Setting Up the App

IIS 7.0 with FastCGI makes setting up a PHP application pretty simple. To get started, I created a Web site and added a myphpgallery host header binding so it could be accessed as http://myphpgallery from my local machine. (I also added the myphpgallery hostname to %windir%\system32\drivers\etc\hosts so my machine knew where to find it.)

Next, I downloaded the latest version of Qdig from qdig.sourceforge.net and unzipped it into the Web site root. To prepare for testing the gallery, I dropped a bunch of images into the root of the Web site (you can also create subdirectories and drop images in there as well). In this case, I used some of the sample images included with Windows Vista.

I downloaded the latest non-thread-safe PHP build for Windows (PHP 5.2.3 as we go to press), from php.net/downloads and unzipped it to C:\php. I was pretty much good to go at this point, although I needed to make a few tweaks to php.ini that are required for Qdig:

  1. Rename php-recommended.ini to php.ini
  2. Set "register_long_arrays=On"
  3. Enable the GD extension with "extension=php_gd2.dll"
  4. Set the correct extension path with "extension_dir=./ext"

From the IIS perspective, allowing the PHP app to run requires only a couple of steps. First, add the PHP/FastCGI handler mapping to the PHP-CGI.EXE, as described in the following articles: for Windows Vista: go.microsoft.com/fwlink/?LinkId=104195 and for Windows Server 2008 and Windows Vista SP1: go.microsoft.com/fwlink/?LinkId=104196. Then add index.php as the default document.

Finally, because Qdig generates thumbnails on the fly, I needed to grant Write access to the qdig-files subdirectory of the application to the IIS_IUSRS group in order to allow the IIS worker process to write to it.

That's it. At this point I could hit http://myphpgallery and see the images as shown in Figure 1.

Figure 1 The Qdig Gallery
Figure 1 The Qdig Gallery(Click the image for a smaller view)
Figure 1 The Qdig Gallery
Figure 1 The Qdig Gallery (Click the image for a larger view)

Securing the Gallery

Qdig is a simple image gallery focused on doing one thing really well: letting you browse your image collection via the Web. As such, it doesn't have some of the more complex features that are often needed for Web applications. One such feature is access control, which allows you to restrict access to parts or all of your Web application.

Unfortunately, implementing access control in PHP requires implementing a credential store and cookie-based authentication from the ground up, and it can be quite hard to get right. By contrast, ASP.NET makes access control a breeze by providing a complete solution with the membership service and a set of built-in credential store providers, the forms authentication module, and a set of pre-made Login controls.

In previous versions of IIS, this wouldn't mean much for the Qdig gallery, since it is not an ASP.NET application. But in IIS 7.0, the ASP.NET Integrated mode engine is designed specifically for this scenario, allowing ASP.NET features to be used with any content, including other application frameworks. Using the functionality of ASP.NET Integrated mode, the Qdig gallery can take advantage of the ASP.NET membership, forms authentication, and Login controls just like a native ASP.NET application. In fact, if you already have an existing ASP.NET application that uses forms authentication, you can simply drop in the Qdig gallery and, with just a few steps, enable it to have the same user security as the rest of your application. This results in consistent user security across your entire site.

To implement a forms authentication solution from scratch for the gallery, I first had to choose a membership credential store provider to store user credentials. ASP.NET 2.0 comes with two built-in providers, one for SQL ServerTM and one for Active Directory®.

The SQL Server provider can be used with SQL Server 2005 Express Edition, which is a great option for many applications. The best thing about using SQL Server 2005 Express Edition with the membership provider is its ability to automatically create and deploy the database tables to your application's App_Data subdirectory on first use. This doesn't require any external database deployment and produces a credential store database that you can Xcopy with your application files.

The SQL Server membership provider is, by default, configured to use a connection string that uses the local SqlExpress instance to connect to the aspnetdb.mdf database inside the App_Data directory for your application. To make it work, I simply took the following three steps:

  1. Downloaded and installed SQL Server 2005 Express Edition, leaving it running as a named SqlExpress instance (default). (You can download it at microsoft.com/sql/editions/express.)
  2. Created an App_Data subdirectory in your application root, and made it writable by IIS_IUSRS.
  3. Opened IIS Manager and created a few users for the application. To do this, select your Web site in the left-hand tree view, and click the .NET Users feature icon (see Figure 2). Be sure to close the tool after you are finished, as SQL Server Express Edition allows only one user identity to access the database at a time.

Figure 2 Configuring Users in IIS Manager
Figure 2 Configuring Users in IIS Manager(Click the image for a smaller view)
Figure 2 Configuring Users in IIS Manager
Figure 2 Configuring Users in IIS Manager (Click the image for a larger view)

Note the integration between IIS Manager and the ASP.NET membership infrastructure—you can manage both users and roles directly from the administration tool where you configure other IIS authentication and access control features. In fact, when you create the first user, IIS Manager will automatically create the membership database in your application if one doesn't already exist.

Forms Authentication

Now that membership is ready to go, I need to enable forms authentication for the application. I will also need to allow the forms authentication module that implements the authentication service to run for all requests in the application, not just for ASP.NET requests (this is the default behavior for backwards compatibility). I can do all of this directly in the Web site's web.config file as follows:

<configuration>
    <system.webServer>
        <modules>
            <remove name="FormsAuthentication" />
            <add name="FormsAuthentication"
             type="System.Web.Security.FormsAuthenticationModule" />
        </modules>
    </system.webServer>
    <system.web>
        <authentication mode="Forms" />
    </system.web>
</configuration>

Note that even though the FormsAuthentication module is already declared at the server level, it is necessary to remove it and re-declare it to clear the preCondition attribute, which is set to managedHandler by default. This forces the module to only run for requests to managed (ASP.NET) handlers. Since you want forms authentication to protect the PHP app as well, you need to remove this.

At this point, I have configured our application to use forms authentication. However, I did not indicate that any of the content actually requires an authenticated user—I left Anonymous Authentication enabled to allow anonymous access to the site—so the authentication will never be used. In actuality, I only need to allow anonymous access to the login page and require authentication for the rest of the content. I can do this by configuring declarative authorization rules in configuration. The complete configuration in my web.config file is shown in Figure 3.

Figure 3 Web.config (Close Figure 3)
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <!-- The PHP handler mapping -->
        <handlers>
            <add name="PHP via FastCGI" path="*.php" verb="*"                            modules="FastCgiModule" scriptProcessor="
                c:\php\php-cgi.exe" resourceType="Unspecified" />
        </handlers>
        <!-- Add index.php as a default document -->
        <defaultDocument>
            <files>
                <add value="index.php" />
            </files>
        </defaultDocument>
        <security>
            <!-- Deny access to anonymous users -->
            <authorization>
                <add accessType="Deny" users="?" />
            </authorization>
        </security>
        <!-- Let the FormsAuthentication module run for all requests -->
        <modules>
            <remove name="FormsAuthentication" />
            <add name="FormsAuthentication" type=
                "System.Web.Security.FormsAuthenticationModule" />
        </modules>
    </system.webServer>
    <system.web>
        <!-- Turn on Forms Authentication for this application -->
        <authentication mode="Forms" />
    </system.web>
</configuration>

With this configuration, access to the PHP gallery will now be allowed only to authenticated users. Users that are not authenticated will be rejected by the URL authorization feature and redirected by forms authentication to the login page where they can log in using the membership service and the SQL Server credential database.

The final piece of the jigsaw puzzle is the login page, which ties all of this together by providing a login Web form, validating the user's credentials with the Membership service, and issuing the user an encrypted forms authentication login ticket. If you are starting to roll up your sleeves at this point and looking around for your coding hat, don't; this login page is pretty much already done for you. All you need to do is create a simple page in your application root called login.aspx (you can override the path to the login page in the <forms> configuration section). This newly created page uses a single Login control:

<html>
  <head>
    <title>Login to my blog</title>
  </head>
  <body>
    <form runat="server">
      <asp:Login runat="server" />
    </form>
  </body>
</html>

The Login control will automatically display a login form, validate your user credentials after submission, and issue the forms authentication ticket. This ticket will then be used to automatically authenticate you for the rest of the time on the Web site, until it expires due to inactivity or you close the browser window.

Figure 4 shows a spruced-up version of the login page; it is using a couple of other login-related controls in order to provide a more convenient user experience. The LoginView control allows you to select between an anonymous user's view and an authenticated user's view, with the LoginName control providing the authenticated user's name and the LoginStatus control providing a logout link for authenticated users. You have the ability to specify the way in which the login controls appear to the client by going ahead and setting one of myriad formatting properties or by theming or skinning them with custom CSS.

Figure 4 Fancy Login Page (Close Figure 4)
<html>
  <head>
    <title>Login to view the gallery</title>
  </head>
  <body>
    <form runat="server">
      <asp:LoginView runat="server">
        <AnonymousTemplate>
          Please log in to proceed.
          <br /><br />
          <asp:Login DestinationPageUrl="/"  runat="server" />
        </AnonymousTemplate>
        <LoggedInTemplate>
          Welcome <asp:LoginName runat="server" />!
          <br /><br />
          <asp:LoginStatus runat="server" />
        </LoggedInTemplate>
      </asp:LoginView>
    </form>
  </body>
</html>

That's it. Now, when you access your gallery, you will be automatically redirected to the login page shown in Figure 5. To log in, you will need to enter user credentials for one of the users you created earlier, and you are in business!

Figure 5 The Completed Login Page
Figure 5 The Completed Login Page(Click the image for a smaller view)
Figure 5 The Completed Login Page
Figure 5 The Completed Login Page (Click the image for a larger view)

Search-Engine-Friendly URLs

By default, the Qdig image gallery uses ugly-but-functional querystring URLs to navigate through your image gallery. Here is an example of a Qdig URL from my gallery:

http://myphpgallery/index.php?Qwd=./Mike&Qif=Flower.jpg

Not particularly pretty or memorable. More importantly, it makes it more difficult for search engines to properly index the content of your gallery.

What I would like to do is make Qdig use search-engine-friendly URLs, which are all the rage for Web 2.0 applications these days because they create a much more intuitive browsing experience for users and are much more effective at getting your Web site indexed well by search engines. For example, a friendly URL version of the one I showed earlier would be:

http://myphpgallery/index.php/Mike/Flower.jpg

This causes the subgallery and image name to be put into the URL path, making it more intuitive, and the important keywords are placed in the absolute path of the URL, making it more easily indexed by search engines.

Interestingly enough, the server will correctly route this URL to index.php by default, recognizing the remaining "/Mike/Flower.jpg" as the PATH INFO segment. However, the problem with this is that this URL does not follow the URL format expected by the Qdig application's script, index.php. Even though we can get the URL to invoke the index.php script, Qdig will not find the required querystring parameters and won't know how to work correctly.

Fortunately, the ASP.NET Integrated pipeline comes to the rescue again, this time allowing me to write a small .NET module that can rewrite incoming URLs on the fly from a more friendly format to Qdig-native URLs, causing the correct gallery view to be returned to the client. Thanks to the ASP.NET Integrated pipeline, this module can run for PHP requests the same way it can for requests to ASP.NET content.

To do this, I will implement a managed module using the ASP.NET IHttpModule pattern. You can read more about building such modules for IIS 7.0 in the article "Developing IIS7 Modules and Handlers with the .NET Framework" at mvolo.com/blogs/serverside/archive/2007/08/15/Developing-IIS7-web-server-features-with-the-.NET-framework.aspx.

The module will do the following:

  1. Subscribe to the BeginRequest stage of request processing.
  2. Intercept requests with URLs in a friendly format.
  3. Extract the subgallery path and the image file name from the URL.
  4. Rewrite the requests to the Qdig index.php, passing the subgallery path and image file name in querystring parameters as expected by Qdig.

Writing this module was actually quite simple. First, I needed to create a C# class that implements the System.Web.IHttpModule interface and wires up an event handler inside the Init method to the BeginRequest event:

public class QdigSEFModule : IHttpModule
{
    public void Init(HttpApplication application)
    {
        application.BeginRequest += new EventHandler(OnBeginRequest);
    }
    ...
}

Then I implement the rewriting functionality in the OnBeginRequest method as shown in Figure 6.

Figure 6 Rewriting URLs (Close Figure 6)
public void OnBeginRequest(Object source, EventArgs e)
{
    HttpApplication app = (HttpApplication)source;
    HttpContext context = app.Context;

    // Extract the gallery path and image filename from the url
    // code omitted for clarity ...

    // Build the rewritten url
    String rewrittenUrl = String.Format("~/index.php?Qwd=.{0}&Qif={1}",
                                        galleryPath,
                                        imageFileName);

    String originalQueryString =
             context.Request.ServerVariables["QUERY_STRING"];
    if (!String.IsNullOrEmpty(originalQueryString))
    {
        rewrittenUrl += "&" + originalQueryString;
    }

    // Rewrite the url
    context.Server.TransferRequest(rewrittenUrl);
}

Note the use of HttpServerUtility.TransferRequest, a new ASP.NET API in the .NET Framework 3.5 intended to allow correct rewriting in the IIS 7.0 ASP.NET Integrated pipeline. This API forces the request processing to be stopped for the current request and executes a new child request to the target URL. This allows managed modules to completely transfer request processing to another URL, regardless of the destination content type. You can find the complete source code for this module in the download for this issue.

After writing the module, I need to deploy it to the application. There are a number of ways to do this, including compiling the module into a .NET assembly and deploying it to the BIN directory of the application. However, I will pursue the easiest option: simply copying the source code of the module and saving it as QdigSEFRewriter.cs in the App_Code subdirectory of the application. The ASP.NET compilation system will automatically compile the code during application startup and make it available to the ASP.NET runtime.

The final step is to enable the module to run in the application by registering it in the <modules> configuration section of the application's web.config file, where a number of built-in ASP.NET modules such as forms authentication are already registered by default. Building on the web.config file created for access control, I can add the new module like so:

<system.webServer>
    <modules>
        <remove name="FormsAuthentication" />
        <add name="FormsAuthentication" type=
            "System.Web.Security.FormsAuthenticationModule" />
        <!-- Add the SEF url rewriting module -->
        <add name="QdigSEFUrlsModule" type="QdigModules.QdigSEFModule" />
    </modules>
</system.webServer>

At this point, you can begin accessing the Qdig gallery using friendly URLs instead of the Qdig native querystring URLs. Now, to access the Flower.jpg file in the Mike subdirectory of the gallery, I can simply request the file like this:

http://myphpgallery/Mike/Flower.jpg/show

Notice that I chose to use the /show suffix for friendly URLs to differentiate them from direct links to the image files and to prevent my module from rewriting URLs that it is not intended to rewrite. You can, of course, modify the rewriting scheme in a way that suits your needs.

Generating Friendly Hyperlinks

Unfortunately, I'm not quite finished. While galleries can now be accessed with friendly URLs, the gallery pages themselves still contain hyperlinks in the old format:

<a href="http://index.php?Qwd=./Mike&amp;Qif=Arctica.jpg&amp;Qiv=thumbs&amp;
Qis=M" title="First Image"> ... </a>

A user clicking on these links will still see the unsightly querystring URLs, and a search engine indexing the content will not be able to correlate the friendly URLs for the parent page with the links in the page body.

It would be really handy to make Qdig generate hyperlinks using the same friendly URLs that I enabled in the previous step. But doing this would entail rewriting the PHP script to generate different hyperlinks, which is something I didn't want to do.

It turns out that I don't have to, thanks to the ASP.NET response filtering facility. I can write another managed module that filters the Qdig responses on the fly, replacing the hyperlinks in the old format with links in the new, friendly format.

This time, my module will do the following:

  1. Subscribe to the PreRequestHandlerExecute stage of request processing (just before the handler executes).
  2. Check whether the URL is for the Qdig index.php script.
  3. Set the response filter stream on the response, which will use regular expressions to replace all occurrences of friendly hyperlinks.

The module class is pretty simple, since all it does is selectively wire up the filter stream (see Figure 7). The real work is done in the QdigSEFFilter class, which is responsible for replacing all occurrences of Qdig hyperlinks with hyperlinks using the new format. The filter class implements the abstract System.IO.Stream class and is used by the ASP.NET runtime to filter the response at the end of request processing. The ASP.NET Integrated pipeline engine allows the response filter to operate on any response, even those not generated by ASP.NET, and so can be used to process PHP responses as well as static files and ASP pages as long as the response is buffered. IIS 7.0 buffers all responses by default (up to the configured limits), which allows this mechanism to work.

Figure 7 Generating Friendly Hyperlinks (Close Figure 7)
public class QdigSEFFilterModule : IHttpModule
{
    public void Init(HttpApplication application)
    {
        application.PreRequestHandlerExecute += new EventHandler(
            OnPreRequestHandlerExecute);
    }

    public void Dispose(){}

    public void OnPreRequestHandlerExecute(Object source, EventArgs e)
    {
        HttpApplication app = (HttpApplication)source;
        HttpContext context = app.Context;

        if (context.Request.Path.Equals(
                VirtualPathUtility.ToAbsolute("~/index.php")))
        {
            // wire up the response filter
            context.Response.Filter = new QdigSEFFilter(context);
        }
    }
}

My filter implementation will buffer all of the incoming response bytes pushed through its stream, convert them to a string using the charset of the response, and perform the regular expression replacement of URLs. Then it will re-encode the string into the original charset and push the response data back to the runtime. You can find the full implementation of the module and the response filter in the code download.

I will again deploy the module code as a separate source file, QdigSEFFilter.cs, into the App_Code directory, and register the module in the <modules> configuration section in the application's web.config. At this point, the modules section will look like this:

<system.webServer>
    <modules>
        <remove name="FormsAuthentication" />
        <add name="FormsAuthentication" type=
            "System.Web.Security.FormsAuthenticationModule" />
        <!-- Add the SEF url rewriting module -->
        <add name="QdigSEFUrlsModule" type="QdigModules.QdigSEFModule" />
        <!-- Add the response filter module -->
        <add name="QdigSEFFilterModule" type=
            "QdigModules.QdigSEFFilterModule" />
    </modules>
</system.webServer>

After refreshing the Qdig page at http://myphpgallery/ and logging in, all of the hyperlinks on the page have changed to use the friendly URLs (see Figure 8). So far, I haven't had to change a single line of the original application, after making several significant upgrades to its functionality with existing ASP.NET features and new functionality developed for the ASP.NET Integrated pipeline.

Figure 8 The Revised Gallery with Friendly URLs
Figure 8 The Revised Gallery with Friendly URLs(Click the image for a smaller view)
Figure 8 The Revised Gallery with Friendly URLs
Figure 8 The Revised Gallery with Friendly URLs (Click the image for a larger view)

Testing Performance

At this point, I have added quite a bit of functionality to Qdig, even though I haven't changed any of the application's source code. So how much does all this new functionality impact the performance of the application? Performance has historically been a concern for PHP applications on the IIS platform due to the process-launching overhead of the CGI mechanism on Windows. FastCGI support promises to significantly improve application performance by removing this overhead, making IIS a much more attractive platform for hosting PHP and other FastCGI-compliant applications. However, if the application enhancements I just introduced add a significant portion of this overhead back, they may not be so attractive after all when considering deploying PHP applications on IIS.

In reality, however, the feature enhancements I added have little effect on Qdig's performance. Figure 9 shows a summary of the performance tests I performed on the application.

Figure 9 Performance Results (Close Figure 9)

Test | Requests per Second (more is better)
Qdig with CGI | 32
Qdig with FastCGI | 93
Qdig and friendly URLs with FastCGI | 88
Hello.php with CGI | 51
Hello.php with FastCGI | 2239

The test was performed with the Microsoft Web Capacity Analysis Tool (WCAT) 6.3 using 100 concurrent virtual clients with full CPU utilization on the server, requesting a series of Qdig URLs inside the gallery. For the last test, friendly URLs were used instead of Qdig URLs to utilize the conversion functionality. Authentication was disabled for ease of testing.

Moving from CGI to FastCGI, you see an improvement of almost three times the throughput. After adding the two friendly URL modules, I only saw a 5 percent drop in throughput, which is negligible.

Unfortunately, Qdig is extremely CPU-bound, which negates the bulk of the possible improvement provided by FastCGI. By comparison, for a simple hello.php script, I saw an improvement of 43 times the throughput when going from CGI to FastCGI. Even after adding a response filter—a pretty expensive operation—the impact on overall throughput is negligible in comparison with the overhead of the application itself.

This is pretty much the worst situation to be in as far as server performance tuning is concerned, because there is very little you can do outside of the application to improve its performance. Changing the application code itself is often impossible unless you are the application's developer, and even then it may be difficult. For me it's certainly out of scope as I decided not to modify the application itself when I started.

ASP.NET Output Caching

The output cache is a feature of ASP.NET that allows reuse of responses produced by Web applications for subsequent requests to the same resources. In fact, ASP.NET provides an extensive set of features for caching both application data inside ASP.NET applications with its Cache API and an application's entire responses with the output cache. Both features can often enhance an application's performance and significantly reduce the load on the back-end resources. New in IIS 7.0 is the ability to use the ASP.NET output cache for non-ASP.NET content types, which is exactly what I will use to give the image gallery a little boost.

As opposed to piecemeal application performance tuning that often requires a large coding effort and removes chunks of the application's processing overhead, output caching tends to remove virtually the entire overhead for a percentage of the request workload. Its effectiveness depends on the locality of the application's usage: how often the same content is requested by the application's users. For typical applications, this often falls into the 90/10 rule, which basically means that 90 percent of requests always go to 10 percent of your content. If you can output-cache this content, it may mean a near 90 percent throughout and capacity improvement to your application.

Of course, if it was so easy to do, everyone would be doing it. Output caching does have a set of limitations, especially when it comes to dynamic content, which often makes deploying a caching solution difficult. The major limitations stem from the dynamic nature of the content. For example, your Web site's pages may produce different responses based on query-string parameters, time of day, or database updates. If you neglect to take these into account, you may return incorrect cached responses to your users, which will not be a good thing. Fortunately, the ASP.NET output cache takes all of these limitations into account and as a result provides a rich platform for deploying output caching solutions for your application.

The ASP.NET page parser provides support for the<%@ OutputCache %> directive, which allows many output caching settings to be configured directly in the ASPX page. In fact, this is how most developers enable output caching for their ASP.NET applications. Since Qdig is written in PHP, it can't take advantage of this feature. So instead I will write another module that dynamically configures the output cache using the ASP.NET HttpCachePolicy APIs, enabling the output cache module to correctly cache Qdig's responses:

public class QdigOutputCacheModule : IHttpModule
{
    public void Init(HttpApplication application)
    {
        application.PostAuthorizeRequest +=
            new EventHandler(OnPostAuthorizeRequest);
        application.PostRequestHandlerExecute +=
            new EventHandler(OnPostRequestHandlerExecute);
    }

    public void Dispose(){}
}

This time, this module will subscribe to two pipeline events: PostAuthorizeRequest and PostRequestHandlerExecute. The first, PostAuthorizeRequest, occurs right before the output cache module attempts to look up an existing response for the request and return it without further processing. The PostRequestHandlerExecute event occurs right after the handler (PHP) produces the response and before the output cache module attempts to save the response for subsequent use.

In the PostAuthorizeRequest event, I need to tell the output cache module how the Qdig responses vary by querystring parameters, so it can find the right response (see Figure 10). By configuring the cache to vary by the querystring parameters, I am enabling the Qdig gallery to operate correctly by responding to the user input such as changing the navigation mode or clicking the next link to go to the next image. Without this, the first request will end up caching its response and returning it for all subsequent requests to the gallery regardless of which image was requested. Also, I will vary by the authenticated user (see the GetVaryByCustomString implementation further), so that the cache correctly gives each authenticated user his own cached copy.

Figure 10 Determining the Correct Cache Variation (Close Figure 10)
public void OnPostAuthorizeRequest(Object source, EventArgs e)
{
    HttpApplication app = (HttpApplication)source;
    HttpContext context = app.Context;

    // Make sure we only process requests to QDIG's index.php.
    String requestPath = context.Request.Path;
    if (!requestPath.Equals(VirtualPathUtility.ToAbsolute
        ("~/index.php")))
        return;

    //  Set up the vary by querystring information
    HttpCacheVaryByParams varyByParams =
      context.Response.Cache.VaryByParams;
    varyByParams["Qwd"] = true; //  gallery path
    varyByParams["Qif"] = true; //  image to display
    varyByParams["Qiv"] = true; //  navigation mode
    varyByParams["Qis"] = true; //  image size
    varyByParams["Qtmp"] = true;//  other control information

    // if user is set, also vary by user (See global.asax)
    context.Response.Cache.SetVaryByCustom("AuthenticatedUser");
}

In the PostRequestHandlerExecute event, I need to determine whether the response should be cached and configure the appropriate settings allowing it to be cached (see Figure 11). Note the AddFileDependency calls. In the omitted code, I determine the physical path to the gallery directory and the image, if specified, and add a File Dependency to the response. This uses the CacheDependency mechanism supported by the output cache to monitor these resources for changes and to invalidate the cached response when they are modified. This allows me to keep the response cached for a long time, only invalidating it when the underlying data changes. When someone drops a new image or creates a new subgallery, the cache automatically removes the affected cached responses and any new requests will receive the updated pages.

Figure 11 Configuring the Response to be Cached (Close Figure 11)
public void OnPostRequestHandlerExecute(Object source, EventArgs e)
{
    HttpApplication app = (HttpApplication)source;
    HttpContext context = app.Context;

    // Make sure we only process requests to QDIG's index.php.
    String requestPath = context.Request.Path;
    if (!requestPath.Equals(VirtualPathUtility.ToAbsolute
        ("~/index.php")))
        return;

    // Enable this response to be output cached
    context.Response.Cache.SetCacheability(HttpCacheability.Public);
    context.Response.Cache.SetExpires(DateTime.Now +
      TimeSpan.FromMinutes(5));

    // code omitted for clarity ...

    // add the dependencies to the response
    context.Response.AddFileDependency(physicalDirectoryPath);
    context.Response.AddFileDependency(physicalFileName);
}

The CacheDependency mechanism is a powerful abstraction, allowing you to create your own CacheDependency implementations that can monitor any kind of underlying data—for example, making a Web service call or checking registry keys. The built-in CacheDependency implementations include support for monitoring files, directories, and SQL Server database tables.

Deployment and Testing

I deployed the latest module by dropping its QdigOutputCache.cs source file into the App_Code directory, just like I did before, and registered it in web.config:

<system.webServer>
    <modules>
      ...
      <!-- Let the OutputCache module run for all requests -->
      <remove name="OutputCacheModule" />
      <add name="OutputCacheModule" type=
          "System.Web.Caching.OutputCacheModule" />
      <!-- Add the custom output caching module -->
      <add name="QdigOutputCacheModule" type=
          "QdigModules.QdigOutputCacheModule" />
    </modules>
</system.webServer>

Besides adding the QdigOutputCacheModule, I also re-added the OutputCacheModule in order to remove its managedHandler precondition as I did before for forms authentication. This enables the ASP.NET OutputCacheModule to run during requests to PHP content and provide the output caching service.

Finally, I created a global.asax file (see Figure 12) that contains my implementation of the GetVaryByCustomString, a function that allows my module to vary the cached response by the authenticated user. This is important, because it maintains separate cache copies for each authenticated user, preventing one user from accidentally getting another user's personalized view. Since Qdig does not provide personalized views, this isn't an issue, but I show it anyway as a best practice.

Figure 12 Global.asax Implements GetVaryByCustomString (Close Figure 12)
<%@ Application language="C#" %>
<script runat="server" language="c#">
public override string GetVaryByCustomString(HttpContext context,
    string s)
{
    if ("AuthenticatedUser".Equals(s))
    {
        // vary by authenticated user
        String currentUser = String.Empty;
        if (context.User != null)
        {
            currentUser = context.User.Identity.Name;
        }

        return currentUser;
    }
    else
    {
        // unknown vary string
        return null;
    }
}
</script>

Now let's check whether performance was further improved. The complete results, including the previous tests, are shown in the graph in Figure 13.

Figure 13 Performance Results
Figure 13 Performance Results(Click the image for a smaller view)
Figure 13 Performance Results
Figure 13 Performance Results (Click the image for a larger view)

As expected, output caching did not disappoint, raising the performance of Qdig from 88 requests per second in the baseline FastCGI plus search-engine-friendly (SEF) URL benchmark, to 1386 requests per second on a fully utilized server.

All in all, after moving a native PHP application from CGI to FastCGI, and upgrading it with friendly URLs and output caching, the end result shows a performance improvement of over 40 times the original application. And that is certainly nothing to scoff at.

Wrapping Up

As you can see, the IIS 7.0 Integrated pipeline allows you to add powerful functionality to any application, bringing the power and convenience of ASP.NET and the .NET Framework to apps written on other frameworks. For development teams, this ability is key to leveraging investments in existing applications and developer skill sets. Instead of forcing the applications to be rewritten in order to move to the new platform, IIS 7.0 allows existing apps to benefit from incremental improvements brought about by existing IIS and ASP.NET features, as well as new features developed using the IIS 7.0 end-to-end extensibility model.

In this article, I barely scratched the surface of what is possible with the IIS 7.0 extensibility APIs. While I looked exclusively at the runtime Web server extensibility, IIS 7.0 also provides complete extensibility at the configuration, administration, and GUI management levels, creating myriad exciting opportunities to upgrade and mold your Web server to your needs. To learn more about extending IIS 7.0, as well as download a number of ready-to-use IIS 7.0 plug-ins, be sure to check out my blog, mvolo.com.

Mike Volodarsky is a Technical Program Manager on the Web Platform and Tools Team at Microsoft. Over the past four years, he has driven the design and development of the core feature-set for ASP.NET 2.0 and IIS 7.0. He is now focusing on helping customers leverage the power of these technologies in Windows Server 2008.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


World Ready: Around the World with ASP.NET AJAX Applications

World Ready

Around the World with ASP.NET AJAX Applications

Guy Smith-Ferrier
Code download available at: InternationalizingASPNETAJAX2008_01.exe (167KB)
This article discusses:
  • Globalization support in ASP.NET AJAX
  • Need for support in JavaScript
  • Assembly-based AJAX localization
  • File-based AJAX localization
This article uses the following technologies:
ASP.NET AJAX
Contents
The Assembly-Based Localization Model
How It Works
The File-Based Localization Model
Inside the File-Based Model
Choosing a Localization Model
Localizing the ASP.NET AJAX Framework
Creating a Release Version of the Framework
Globalizing ASP.NET AJAX
La Fin

I
t is a fact that ASP.NET 2.0 has an excellent localization model for Web applications, and the Microsoft® .NET Framework includes considerable globalization support as well. However, this support is limited to server-side code. ASP.NET AJAX applications include a significant amount of client-side code, and this is not covered by the standard ASP.NET localization model. Here I'll explain the localization and globalization models for ASP.NET AJAX applications and show how you can localize JavaScript resources and add limited globalization support. I'll assume you have some knowledge of ASP.NET and ASP.NET AJAX, along with a basic understanding of internationalizing ASP.NET 2.0 applications.

The Assembly-Based Localization Model

The Microsoft ASP.NET AJAX framework includes two models for localizing JavaScript resources: one that is assembly-based and one that is file-based. Let's look first at the assembly-based model.

The assembly-based model uses resources—regular .NET Framework .resx files—that you would use in any ASP.NET or Windows® Forms application. The resources are embedded in an assembly and ultimately retrieved using the ResourceManager class. To explain this process, I will work through an example—a single button that invokes a JavaScript function that displays a localizable message to the user using an alert. If you plan to follow along and create the example yourself, use the names exactly as indicated here; otherwise, it can be difficult to identify the root of problems if your example fails to produce the correct result.

So let's start by creating an AJAX-Enabled Web Application Project (File | New | Project | ASP.NET AJAX-Enabled Web Application). Note that you will need Visual Studio® 2005 SP1 for this project template (the AJAX Web Site Project can't be used as a substitute for this example because there is no clear access to the resulting assembly). Call the project AJAXEnabledWebApplicationI18N1 (I18N is an abbreviation for Internationalization because Internationalization starts with an "I," has 18 more letters, and ends with an "N"). Add a button to the form and an OnClientClick event to call a JavaScript function named displayAlert:

<asp:Button ID="Button1" runat="server" OnClientClick="return displayAlert();"  Text="Button"/>

Now add a JScript file called DisplayFunctions.js, like so:

function displayAlert()
{
  window.alert(DisplayFunctionsResources.Greeting);
  return false;
}

The displayAlert function uses window.alert to display DisplayFunctionsResources.Greeting. Note that DisplayFunctionsResources.Greeting doesn't exist yet, but it is the localizable resource, and I refer to it as if it were a strongly typed resource class. I will return to this idea later. Using the Properties Window, set the Build Action for the DisplayFunctions.js file to Embedded Resource. This tells the compiler to embed DisplayFunctions.js as a resource in the resulting AJAXEnabledWebApplicationI18N1.dll assembly.

Next we want to make it easy for the user to set the culture used by the Web application, so we add Culture and UICulture attributes to the page header and set them to auto. Of course, these page attributes are added automatically if you use "Tools | Generate Local Resource" as you would normally do when localizing an ASP.NET application, but I want to add them manually in this example because I want it to be clear that the ASP.NET AJAX localization models do not require you to also use the ASP.NET localization model.

Now add a Resource file and call it DisplayFunctionsResources.resx. Add a resource entry called Greeting and give it a value of "Hello World". At this point, you should be able to see what the reference to DisplayFunctionsResources.Greeting in DisplayFunctions.js will be pointing at. Now add a Resource file called DisplayFunctionsResources.fr.resx and a resource entry called Greeting, and give it the value "Bonjour Le Monde". This resource file uses the standard .NET Framework .resx file-naming convention of including the culture name (here fr for French) before the .resx file extension.

You'll just have to trust me while performing the next step until I explain later how it all works. For now, you need to know that the resources in your Web application (DisplayFunctions.js, DisplayFunctionsResources.resx, and DisplayFunctionsResources.fr.resx) are made available to your page using the WebResource and ScriptResource HTTP handlers. These handlers do not make all resources available automatically; instead, they look for assembly attributes that mark the resources they are allowed to expose. Open the project's AssemblyInfo.cs file and add these assembly attributes to the file:

[assembly: System.Web.UI.WebResource(
"AJAXEnabledWebApplicationI18N1.DisplayFunctions.js",
"text/javascript")]

[assembly: System.Web.UI.ScriptResource(
"AJAXEnabledWebApplicationI18N1.DisplayFunctions.js", "AJAXEnabledWebApplicationI18N1.DisplayFunctionsResources", "DisplayFunctionsResources")]

To expose these resources to the page, you need to add the ASP.NET AJAX ScriptManager to the page and configure it to generate links to localizable resources. You do this by setting the ScriptManager's EnableScriptLocalization property to true. In addition, you need to indicate which resources the page can reference by adding a ScriptReference to the ScriptManager. The complete ScriptManager element follows:

<asp:ScriptManager ID="ScriptManager1" runat="server"
EnableScriptLocalization="true">
  <Scripts>
    <asp:ScriptReference
      Name="AJAXEnabledWebApplicationI18N1.DisplayFunctions.js"
      Assembly="AJAXEnabledWebApplicationI18N1" />
  </Scripts>
</asp:ScriptManager>

The ScriptReference includes the name of the JavaScript resource and a reference to the resource's assembly.

Finally, you can run your completed Web application. Click on the button and you will see the English resource ("Hello World"), assuming that your system's default language is set to English. To see the French resources in Internet Explorer®, open Tools | Internet Options, click on the Languages button, click on the Add button, select French (France), click OK twice, refresh the page, and finally click on the button in your application and the French resource ("Bonjour Le Monde") will be shown.

How It Works

So now you have a working localized ASP.NET AJAX application, but it was built without a good understanding of how the framework actually makes it work. In this section, I will connect the dots. It all begins with web.config. Take a look at the HTTP handlers section and you will find a ScriptResource handler:

<add verb="GET,HEAD" path="ScriptResource.axd"
type="System.Web.Handlers.ScriptResourceHandler,
System.Web.Extensions, Version=1.0.61025.0,
Culture=neutral, PublicKeyToken=31bf3856ad364e35"
validate="false"/>

This ScriptResourceHandler is referenced in your page when you add EnableScriptLocalization="true" to your ScriptManager. If you go ahead and enable script localization in a freshly created, unchanged ASP.NET AJAX-Enabled Web Application, view the source HTML and you will see the lines shown in Figure 1.

Figure 1 WebResource and ScriptResource References (Close Figure 1)
<script src="/WebResource.axd?d=wLnl3WuFAiWB3W8aelty5A2&amp;
t=632969072944906146" type="text/javascript">
</script>

<script src="/ScriptResource.axd?d=
ekhhG997JkU79ensgQhEZepkzUh3YLeLuz9-kWs0_QimDVEmoCSwcFNOHayaCQX6y2v7StlLcqRUUHTtwG4zJloCtMocAEx2dXPoMNyO-AE1&amp;
t=633132543078401746" type="text/javascript">
</script>

<script src="/ScriptResource.axd?d=
ekhhG997JkU79ensgQhEZepkzUh3YLeLuz9-kWs0_QimDVEmoCSwcFNOHayaCQX6y2v7StlLcqRUUHTtwG4zJjtbgQARGVzU-hP34jwIti3qwKit4BmBA5oDqgYMBzoU0&amp;
t=633132543078401746" type="text/javascript">
</script>

The WebResource reference will be there regardless of whether you have enabled script localization, but, since all of the handlers use the same parameters, that's where we'll start.

The key to understanding what ASP.NET AJAX is doing lies in understanding the parameters. In Figure 1, you can see that there are two parameters: d and t. The parameter d represents the data and t is a date/time stamp. The t parameter is simply the DateTime (represented in ticks) of the assembly that contains the resource. A value of 632969072944906146 represents 20th October 2006 02:14:54, which is the date/time stamp for System.Web.dll. The benefit of this parameter is that the result returned by the HTTP handler will be cached by the browser as long as the URI does not change. This means that if the assembly that serves the resource changes, its date/time stamp will change and the ScriptManager will generate a different URI (because there will be a different t parameter value) and the cached value will be replaced.

The d parameter is encrypted so it makes no sense to the naked eye. However, the System.Web.UI.Page class has internal static methods—DecryptString and EncryptString—that allow the Page to build and un-build these strings. The decrypted version of the d parameter (wLnl3WuFAiWB3W8aelty5A2) in WebResource in Figure 1 is s|WebForms.js. The vertical bar is simply used to delimit different values in the string. The "s" indicates that the data is script, and "WebForms.js" is the name of the resource to retrieve. The WebForms.js resource is retrieved from System.Web.dll.

The ScriptResource d parameters make for more interesting reading. The d parameters shown in Figure 1 are encrypted versions of this string

ZSystem.Web.Extensions,1.0.61025.0,31bf3856ad364e35|MicrosoftAjax.js|en

and this string:

ZSystem.Web.Extensions,1.0.61025.0,31bf3856ad364e35|
MicrosoftAjaxWebForms.js|en

The first character (Z) indicates that the result should be returned in GZip compressed format. The next parameter is the fully qualified assembly name where the resource can be found (the ASP.NET AJAX System.Web.Extensions.dll in this example). The parameter following that is the name of the resource to retrieve (in this instance, MicrosoftAjax.js), and the final parameter is the culture for which it should be retrieved (en for English in this case).

If you view the HTML source for the sample application, you will see a third ScriptResource that is generated as a result of including the ScriptReference element in the ScriptManager. It looks like the following:

<script src="/ScriptResource.axd?d=SIfun5X-G8AZLQEDQrC3E6MU4JIgMj1Y2u4fjk0_keEO3Avx4Kbf3v3a
WEvpImtLlh3gxoqTDlXW5ZBtINnuNCTcGSSWUdsCxEJu9l3X
fPtkrSmJXLzRqze5gVmu6Xzz0&amp;t=633137261664170000"
type="text/javascript">
</script>

The decrypted version of this d parameter is ZAJAXEnabledWebApplicationI18N1|AJAXEnabledWebApplicationI18N1.DisplayFunctions.js|. It indicates that the ScriptResource HTTP handler should retrieve the AJAXEnabledWebApplicationI18N1.DisplayFunctions.js resource from the AJAXEnabledWebApplicationI18N1 assembly. Recall that we set the Build Action of this resource to embedded resource so it will be embedded in the assembly. It is also worth noting that the culture (at the end of the decrypted d parameter) is empty. When the browser's language is set to fr, the culture value in the d parameter is set to fr.

The final pieces of this jigsaw puzzle should fall into place if you consider the Z value at the beginning of the d parameter. Instead of Z for compressed, you might find a u, which indicates that the result should be returned uncompressed. So if you change this value to a u, rebuild the parameter (uAJAXEnabledWebApplicationI18N1|AJAXEnabledWebApplicationI18N1.DisplayFunctions.js|), encrypt it, and then call the ScriptResource HTTP handler directly, like this:

http://localhost:49573/ScriptResource.axd?d=t9iZ-3-wfmF_qcNwBHaQrktF2T3rVGzj376NN7kvehJFET0JsrpDTsYt9i
KmZpSmrU1GjtTCo9PIQmuAmMNNkN2Yc_ipVH7JL-jt_hmL9_v8
HtplfhERvz-WsOUxZ-1k0&t=633137261664170000

Then you can see exactly what the handler is returning. Here's the uncompressed, intelligible result:

// JScript File
function displayAlert()
{
  window.alert(DisplayFunctionsResources.Greeting);
  return false;
}
DisplayFunctionsResources={
"Greeting":"Hello World"
};

You can see that this is an amalgamation of the original DisplayFunctions.js file plus a JavaScript serialization of the DisplayFunctionsResources.resx. This amalgamation is serialized on request by the ScriptResource HTTP handler. Notice that the name of the JavaScript class is DisplayFunctionsResources. At first, you might think this is the name of the corresponding .resx file. However, this is a deliberate coincidence on my part. Recall the ScriptResource assembly reference from above:

[assembly: System.Web.UI.ScriptResource
("AJAXEnabledWebApplicationI18N1.DisplayFunctions.js", "AJAXEnabledWebApplicationI18N1.DisplayFunctionsResources", "DisplayFunctionsResources")]

The final parameter (DisplayFunctionsResources) is the name of the JavaScript class that the ScriptResource HTTP handler generates. I chose this name because I think it is helpful to use the same name for the class and the file name, as this is the convention used by strongly typed resource classes. However, you can make these names different if you prefer.

When you go ahead and switch the culture to fr, the ScriptResource HTTP handler returns an amalgamation using the DisplayFunctionResources.fr.resx resources (courtesy of the ResourceManager behavior that has existed since the release of the .NET Framework 1.0):

// JScript File
function displayAlert()
{
  window.alert(DisplayFunctionsResources.Greeting);
  return false;
}
DisplayFunctionsResources={
"Greeting":"Bonjour Le Monde"
};

The File-Based Localization Model

The second of the two AJAX localization models is what I call file-based. That is, the JavaScript files are not embedded in an assembly. Instead they are returned unchanged to the client. Clearly, this model is suitable for project types that do not have an easily identifiable assembly, but there are other benefits as well. Let's work through an example.

Create an AJAX-Enabled Web Site (File | New | Web Site... | ASP.NET AJAX-Enabled Web Site). You can call it anything you like, as the name is not used for any purpose. Add the same button as you did in the first exercise:

<asp:Button ID="Button1" runat="server" OnClientClick="return displayAlert();"  Text="Button"/>

Next, add a JScript file called DisplayFunctions.js consisting of the code shown here:

function displayAlert()
{
  window.alert("Hello World");
  return false;
}

Notice that in this example the text is clearly hardwired and is not in any way localizable.

Now create a second JScript file and call it DisplayFunctions.fr.js:

function displayAlert()
{
  window.alert("Bonjour Le Monde");
  return false;
}

As in the first example, add Culture and UICulture attributes to the page header and set the values to auto. Finally, set EnableScriptLocalization="true" to a ScriptManager and add a ScriptReference to the DisplayFunctions.js file:

<asp:ScriptManager ID="ScriptManager1" runat="server"
EnableScriptLocalization="true">
  <Scripts>
    <asp:ScriptReference Path="DisplayFunctions.js"
                         ResourceUICultures="fr"/>
  </Scripts>
</asp:ScriptManager>

The ScriptReference uses Path and ResourceUICultures attributes instead of the Name and Assembly attributes used in the assembly-based model. If you run the Web site now and switch between no language and French, you will see that the button results in the correctly localized message.

Inside the File-Based Model

The file-based model is clearly different from the assembly-based model. If you right-click the page in Internet Explorer and select view source, you will see the generated script tag here for no language:

<script src="DisplayFunctions.js" type="text/javascript"></script>

Here's the script tag for French:

<script src="DisplayFunctions.fr.js" type="text/javascript"></script>

So the first difference is that the file-based model has a separate and completely self-contained .js file for each culture. There are no .resx files containing resources that are subsequently converted to JavaScript classes. Each .js file contains both the logic and the resources. This is both good and bad: good because it allows you to customize the logic for each culture very easily, bad because if you don't want to customize the logic, you end up having to duplicate the logic for each culture simply to change the resources for each. Maintenance can get ugly.

Let's return to the ScriptReference you saw earlier:

<asp:ScriptReference Path="DisplayFunctions.js" ResourceUICultures="fr"/>

The Path attribute refers to the name of the JavaScript file (DisplayFunctions.js). The ResourceUICultures attribute lists the cultures for which a unique script exists, and it is required. You might argue that the ScriptResource HTTP handler could easily work this out from the file directory and that this attribute is therefore redundant. However, in order to boost performance, the static script files are served directly by IIS, so the ScriptResource handler never has the opportunity to evaluate the request. Because of this process, the list must be hardcoded.

Given what I have covered so far, the file-based model might appear somewhat lacking, but with a bit of reorganization it can be more useful. Recall from the assembly-based model that the JavaScript that was returned to the client by the ScriptResource HTTP handler is an amalgamation of the original JavaScript file plus a JavaScript class representing the resources from a .resx file. The same approach could be adopted for the file-based model. Instead of writing the same functions from earlier, write this

function displayAlert()
{
  window.alert("Hello World");
  return false;
}

and this:

function displayAlert()
{
  window.alert("Bonjour Le Monde");
  return false;
}

You could refactor this code so that the logic remains exactly the same for each culture and the resources are moved to the latter part of the JavaScript file. Try this

function displayAlert()
{
  window.alert(DisplayFunctionsResources.Greeting);
}

DisplayFunctionsResources={
'Greeting':'Hello World'
}

and this:

function displayAlert()
{
  window.alert(DisplayFunctionsResources.Greeting);
}

DisplayFunctionsResources={
'Greeting':'Bonjour Le Monde'
}

This certainly improves the maintenance story because, although you still have to apply the same change to each culture, it is significantly easier to do since the top part of the file is the same for every one.

Now consider a further revision. You could keep the logic in a separate common JavaScript file (such as DisplayFunctions.Common.js) and generate the localized versions from corresponding .resx files at build time using a custom build task following the same approach that the ScriptResource HTTP handler does for the assembly-based model. I have left this as an exercise for the reader.

Choosing a Localization Model

So now that you know both models, the obvious question is which one should you use? Figure 2 provides a comparison.

Figure 2 File-Based versus Assembly-Based Localization (Close Figure 2)

Criterion | File-Based | Assembly-Based
ScriptReference Attributes | Path, ResourceUICultures | Name, Assembly
Scripts can use localized resources | No | Yes
Scripts can be localized | Yes | No
Use with Web Application Projects | Yes | Yes
Use with Web Site Projects | Yes | No
Use with Control Libraries | Yes | Yes

The big benefit of the file-based model is the fact that you can easily customize the client-side logic and it can be used with Web Site Projects. On the other hand, the assembly-based model lets you easily leverage localized resources, and the JavaScript files can be embedded in an assembly. This makes the model ideal for use with control libraries. The nature of your work will probably dictate which model is best for you. As I use Visual Studio 2005 SP1 and don't often need to customize logic for different cultures, the assembly-based model works best for me.

Localizing the ASP.NET AJAX Framework

The .NET Framework is itself available in localized versions—.NET Framework Language Packs. When installed, the .NET Framework can draw upon these language packs to expose localized resources such as error messages and dialogs. The ASP.NET AJAX framework itself draws upon resources such as error messages that must be available on the client. And while it is not possible for sources outside of Microsoft to create new .NET Framework language packs, anyone can create localized versions of the ASP.NET framework. At the present time, the ASP.NET AJAX framework is available only in English, but when it becomes part of Visual Studio 2008, localized versions of the framework will be available. However, the localization support will be limited to the subset of languages that are supported by the .NET Framework, so the chances of getting, say, a Welsh localization of the ASP.NET AJAX framework are quite small. Hence, we still need a method to localize the ASP.NET AJAX framework.

Fortunately, this method already exists and is built on the file-based localization model already discussed. The ASP.NET AJAX assemblies contain English resources. If you open System.Web.Extensions.dll using Reflector (see Figure 3), you can see the JavaScript resources that are used by default.

Figure 3 Using Reflector to View the System.Web.Extensions.dll's JavaScript Resources
Figure 3 Using Reflector to View the System.Web.Extensions.dll's JavaScript Resources(Click the image for a smaller view)
Figure 3 Using Reflector to View the System.Web.Extensions.dll's JavaScript Resources
Figure 3 Using Reflector to View the System.Web.Extensions.dll's JavaScript Resources (Click the image for a larger view)

However, you can instruct the ScriptManager to use a different set of JavaScript files simply by adding a ScriptReference for the same JavaScript resource name, as you see here:

<asp:ScriptManager ID="ScriptManager1" runat="server"
  ScriptMode="Debug" EnableScriptLocalization="true">
  <Scripts>
    <asp:ScriptReference Name="MicrosoftAjax.js"
      Path="~/AjaxFramework/MicrosoftAjax.js"
      ResourceUICultures="fr" />
  </Scripts>
</asp:ScriptManager>

To see this in action, you need to create a new AJAX-Enabled Web Application Project, change the ScriptManager to the one shown directly above, add a folder called AjaxFramework, and copy the MicrosoftAjax.Debug.js file from the ASP.NET AJAX MicrosoftAjaxLibrary\System.Web.Extensions\1.061025.0 folder to the AjaxFramework folder. Now copy MicrosoftAjax.Debug.js to MicrosoftAjax.Debug.fr.js to create the French localization of the framework. Open MicrosoftAjax.Debug.fr.js, go to the bottom of the file where Sys.Res is located, find the argumentType field and change its value to "L\' objet ne peut pas être converti le type demandé." (This is the French translation of "Object cannot be converted to the required type.")

Now add a button to the form with an OnClientClick event that calls showError. Add the showError JavaScript function here to default.aspx:

function showError()
{
  window.alert(Sys.Res.argumentType);
  return false;
}

Add Culture and UICulture attributes to the page and assign them values of "auto". Run the application. When the browser's language is French, you'll see the French resources being displayed.

Creating a Release Version of the Framework

The problem is solved—but only as far as the debug version is concerned. If you set the ScriptManager's ScriptMode to Release, the application will fail at run time because it will be looking for the MicrosoftAjax.js or MicrosoftAjax.fr.js file, and these are clearly missing. The simplest solution is to copy the debug versions of these files and rename them to their release file names. This solves the initial problem, but obviously it means that the release and debug versions are identical and, more importantly, the release versions are not optimized and are bloated with comments and white space.

Fortunately, ASP.NET AJAX has a solution. The AJAX Control Toolkit includes a MSBuild task called JavaScriptCommentStripper to address this exact problem. This task takes a JavaScript file as an input and outputs the same file with the comments and white space stripped out. Figure 4 shows a simple MSBuild script that converts debug.js files into stripped versions of the same files. So in our example, the MicrosoftAjax.Debug.js and MicrosoftAjax.Debug.fr.js files would result in two additional files—MicrosoftAjax.Debug.js.Stripped and MicrosoftAjax.Debug.fr.js.Stripped. When you are happy with the result, rename these to MicrosoftAjax.js and MicrosoftAjax.fr.js, respectively, and you're all set.

Figure 4 JavaScriptCommentStripper MSBuild Task (Close Figure 4)
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="StripComments">
  <UsingTask TaskName="JavaScriptCommentStripperTask"
  AssemblyFile="C:\Program Files\Microsoft ASP.NET\ASP.NET 2.0 AJAX
  Extensions\AJAXControlToolkit\Binaries\JavaScriptCommentStripper.dll"/>

  <ItemGroup>
    <SourceFiles Include="*.Debug*.js"/>
  </ItemGroup>

  <Target Name="StripComments">
    <JavaScriptCommentStripperTask SourceFiles="@(SourceFiles)"
    DestinationFiles="@(SourceFiles->'$(TargetDir)%(RecursiveDir)
    %(Filename)%(Extension).Stripped')"/>
  </Target>

</Project>

You can see the effect that this task has on the JavaScript by opening up the before and after JavaScript files. Figure 5 shows the top part of the original Microsoft.Debug.js file.

Figure 5 The Top of Microsoft.Debug.js (Close Figure 5)
//-----------------------------------------------------------------------
// Copyright (C) Microsoft Corporation. All rights reserved.
//-----------------------------------------------------------------------
// MicrosoftAjax.js
// Microsoft AJAX Framework.

Function.__typeName = 'Function';
Function.__class = true;

Function.createCallback = function Function$createCallback(method,
    context) {
    /// <param name="method" type="Function"></param>
    /// <param name="context" mayBeNull="true"></param>
    /// <returns type="Function"></returns>
    var e = Function._validateParams(arguments, [
        {name: "method", type: Function},
        {name: "context", mayBeNull: true}
    ]);

Here's the stripped code from the Microsoft.Debug.js.Stripped file. It is clearly shorter (and less readable). In this example, the JavaScriptCommentStripper task has reduced the file in size from 260,096 bytes to 182,454 bytes:

Function.__typeName = 'Function';Function.__class = true;
Function.createCallback = function Function$createCallback(method, context) {var e = Function._validateParams(arguments, [
    {name: "method", type: Function},
    {name: "context", mayBeNull: true}
]);

Globalizing ASP.NET AJAX

To say that JavaScript is lacking in globalization support compared to the .NET Framework is something of an understatement. The .NET Framework has a vast ocean of globalization classes, and the support in JavaScript could fit into a puddle. And there's a bit of a dilemma surrounding adding more. A very large part of the System.Globalization namespace could be implemented in the ASP.NET AJAX framework, but should it be? Each class increases the download size, so the business case needs to be assessed. Consequently, even with globalization support in the ASP.NET AJAX framework, support is considerably more limited than in the .NET Framework. With that in mind, let's look at what we do have.

By default, globalization support in ASP.NET AJAX is disabled. To enable it, you must set the ScriptManager's EnableScriptGlobalization property to true. This simple act includes a __cultureInfo variable in the generated runtime page, which will look like this:

var __cultureInfo = '{"name":"en-GB","numberFormat":{
"CurrencyDecimalDigits":2,
"CurrencyDecimalSeparator":".","IsReadOnly":true,
"CurrencyGroupSizes":[3],"NumberGroupSizes":[3],
"PercentGroupSizes":[3],"CurrencyGroupSeparator":",
","CurrencySymbol":"£","NaNSymbol":"NaN",
"CurrencyNegativePattern":1,"NumberNegativePattern":1,
"PercentPositivePattern":0,"PercentNegativePattern":0,
"NegativeInfinitySymbol":"-Infinity","NegativeSign":"-
...

This variable is generated from the server's CultureInfo.CurrentCulture and respects the server's user settings. ScriptManager does not have an option to ignore the user's settings.

MicrosoftAjax.js contains a reference to this large variable:

Sys.CultureInfo.CurrentCulture = Sys.CultureInfo._parse(__cultureInfo);
delete __cultureInfo;

The variable is read in, parsed, assigned to a Sys.CultureInfo.CurrentCulture variable, and then released.

A few lines up from this in MicrosoftAjax.js you can see the definition of the InvariantCulture:

Sys.CultureInfo.InvariantCulture = Sys.CultureInfo._parse('{"name":"",
"numberFormat":
{"CurrencyDecimalDigits":2,"CurrencyDecimalSeparator":".",
"IsReadOnly":true,"CurrencyGroupSizes":[3],"NumberGroupSizes":[3],
"PercentGroupSizes":[3],"CurrencyGroupSeparator":",",
"CurrencySymbol":"\u00A4","NaNSymbol":"NaN",
"CurrencyNegativePattern":0,"NumberNegativePattern":1,
"PercentPositivePattern":0,"PercentNegativePattern":0,
"NegativeInfinitySymbol":"-Infinity","NegativeSign":"-
...

What is missing is a value for CurrentUICulture. You could either assume that it is the same as the CurrentCulture (a rather dubious assumption), or you could manufacture a solution that is similar to that of the CurrentCulture and inject a suitable variable into the generated HTML sent to the client.

So let's see what we can do with our globalization support. Create an AJAX-Enabled Web Application and set EnableScriptGlobalization to true in the ScriptManager. Add a Label and a Button and add an OnClientClick event to return a call to the showToday JavaScript function:

<script type="text/javascript">
function showToday()
{
  var date = new Date();
  $get('Label1').innerHTML = date.localeFormat("D");
  return false;
}
</script>

Add Culture and UICulture attributes to the page and assign them values of "auto". Run the application with the browser's language set to English or nothing, click on the button, and the date will display in the default format for the culture set in your Regional and Language Options dialog. Change the browser's language to "fr", refresh the page, click on the button, and the Label will show the date in French using the French date format (lundi 2 juillet 2007, for instance).

If you are familiar with the JavaScript Date class, you will be aware that it does not have the method called localeFormat that is used in the showToday function. This is one of the type extensions included in the ASP.NET AJAX framework. You can see the code for these in MicrosoftAjax.Debug.js. Here is the localeFormat method for the Date class:

Date.prototype.localeFormat = function Date$localeFormat(format) {
    /// <param name="format" type="String"></param>
    /// <returns type="String"></returns>
    var e = Function._validateParams(arguments, [
        {name: "format", type: String}
    ]);
    if (e) throw e;

    return this._toFormattedString(format,
Sys.CultureInfo.CurrentCulture);
}

For a full list of these type extensions, see ajax.asp.net/docs/ClientReference/Global/JavascriptTypeExtensions/default.aspx.

So, through the Sys.CultureInfo class, the Sys.CultureInfo.InvariantCulture, and Sys.CultureInfo.CurrentCulture properties and type extensions, the ASP.NET AJAX framework offers a limited globalization framework.

La Fin

The ASP.NET AJAX framework brings localization and globalization support to JavaScript on a level not previously seen outside of custom implementations. The framework offers a choice of two localization models, and one of them is likely to meet your needs. The globalization support is rudimentary compared to the richness of the .NET Framework, but that doesn't mean you can't get creative and add your own type extensions and JavaScript classes to support specific needs. I would imagine that AJAX will continue to see upgrades, as all technologies do, so expect to see enhancements to this model in future releases of ASP.NET AJAX.

Guy Smith-Ferrier is an MVP, an INETA speaker, and the author of .NET Internationalization (Addison-Wesley, 2006). He is a Microsoft Certified Professional Developer, has spoken at many European and U.S. conferences, and has written more than 50 articles for numerous magazines. Guy runs The .NET Developer Network (dotnetdevnet.com) in England. Find him at guysmithferrier.com.


WCF Syndication: HTTP Programming with WCF and the .NET Framework 3.5

WCF Syndication

HTTP Programming with WCF and the .NET Framework 3.5

Justin Smith
This article is based on a prerelase version of Visual Studio 2008. All information herein is subject to change.
This article discusses:
  • HTTP message basics
  • Adding information to URIs
  • WCF and HTTP
  • RSS and ATOM with WCF
This article uses the following technologies:
Windows Communication Foundation
Contents
HTTP Message Basics
Expressing Additional Information
REST and the Web
HTTP Programming with WCF
HTTP Verbs in WCF Contracts
Binding and Hosting
The New Syndication API
Creating a SyndicationFeed
Tying It All Together

W
hen it was first released as part of the Microsoft®.NET Framework 3.0, Windows®Communication Foundation (WCF) included a universal object model for Plain Old XML (POX) and SOAP messaging over a wide variety of transports. Because WCF also contains deep support for WS-* Web service standards, it is able to interoperate with other modern service platforms with relatively little difficulty. WCF in the .NET Framework 3.0 is highly extensible and includes solid manageability features like message logging, activity tracing, service throttling, instance management, and threading controls.

WCF in the .NET Framework 3.5 builds on the extensibility points in the .NET Framework 3.0 to include first-class support for building services that adhere to the principles of the Web. This includes an easy-to-use HTTP programming model, JavaScript Object Notation (JSON) messaging capabilities, and a new syndication API that makes it easy to create and consume syndicated content. With this feature set, WCF is now the service platform of choice for connecting services to Web clients, whether they are ASP.NET AJAX controls, SilverlightTMclients, or even browsers. These features also work in partial trust scenarios (like ASP.NET medium trust) so you can host WCF services in widely available hosting environments. To round it all out, there's also new tooling integrated into Visual Studio®2008 that dramatically reduces the amount of time it takes to get a service up and running.

Combining Web-centric communication with SOAP and WS-* standards into one service stack and object model is one of the features that makes WCF in the .NET Framework 3.5 so compelling. This means you can build a service that communicates within or across enterprise boundaries using SOAP and WS-*, and you can configure that same service to communicate externally using the protocols of the Web. In effect, WCF takes care of the plumbing in your service and lets you focus more closely on the functionality your service exposes.

This article explains some of the new Web-centric features of WCF in the .NET Framework 3.5. It starts with a level-setting discussion about some of the important architectural principles in HTTP and the Web, then moves to the new HTTP programming model in WCF, and, finally, to the new syndication API.

HTTP Message Basics

You probably use the Web almost every day, but how often do you think about the core concepts that make the Web work? It's not in my nature to carry on about architectural concepts, but some of the key ideas baked into the Web may not be obvious at first glance. In my view, a good understanding of these principles is a necessary first step when building services that adhere to the protocols of the Web. Keep in mind that entire books are dedicated to this topic, so I'll only hit the high points in this article. When in doubt, consult the HTTP/1.1 specification (RFC 2616).

HTTP is the native protocol that runs the Web. HTTP is a request/response protocol: every time you use your browser to navigate to a new Web page, you send an HTTP request message to an HTTP server (or proxy), which will return a response message to your browser. HTTP defines several methods (often called verbs informally) that identify operations that an HTTP server can perform (for example, GET, POST, PUT, and DELETE). The HTTP protocol also defines the notion of a resource that is the “target” of the request. Resources are identified by a URI which appears as data in the HTTP request message. Here is a trimmed down version of an HTTP request:

POST /myservice/PostAlbum HTTP/1.1
HOST: www.cloudsample.net
Content-Type: text/xml

<albumInfo>
  <albumId>15</albumId>
</albumInfo>

In this example, the resource URI is http://www.cloudsamples.net/myservice/PostAlbum and the HTTP method is POST. POST is the most common HTTP method for causing side-effects and has the most “liberal” interpretation in the protocol specification (and in common practice). The request message also has an XML payload (also known as the “entity body”) that contains a value for the albumId field.

The POST method is most commonly used for updating data over the web. However, most web traffic simply reads information, which happens using the GET method. A GET request asks for a data value (called a representation) that corresponds to the specified URI. Here’s a simple HTTP GET request:

GET /myservice/GetAlbum/15 HTTP/1.1
HOST: www.cloudsamples.net

GET requests have no payload – rather, the request consists solely of the request URI plus headers. In general, headers are used for content negotiation and cache management, with the URI containing the “significant” information needed to identify the resource in question. This emphasis on the URI is a recurring theme in web-centric applications.

HTTP response messages contain a status code, zero or more headers, and an optional payload/entity body. HTTP status code tells the client what happened, and the headers and payload convey the response information. Here’s a simple HTTP response that corresponds to our GET request above:

HTTP/1.1 200 OK
Content-Type: text/xml
Expires: Thu, 01 Jan 2009 13:00:00 GMT

<Album id="15" title="The rest of the story" />

This response indicates that the request was successfully processed (hence the 200 status code) and that the representation of the resource is the XML value that appears in payload. The Expires header indicates that this response can be cached until 1PM January 1, 2009. The value of standardized cache management headers is that web infrastructure (e.g., proxies, client software) can reduce network traffic and processing cost by satisfying GET requests without a full client-server round-trip (and subsequent server processing costs). For a great treatment on how HTTP cache management works, go to Mark Nottingham’s tutorial at http://www.mnot.net/cache_docs/

Expressing Additional Information

Applications that receive an HTTP GET use the information embedded in the URI to determine which resource to send as a reply. To illustrate, consider this set of URIs:

contoso.com/artists/Flaming+Hammer/HitMe
contoso.com/artists/Northwind/Overdone

In this case, Contoso Corporation has an application that serves resources related to music. You can probably deduce from these examples how the name of the artist and the name of the album can map into a URI:

contoso.com/artists/[artist]/[album]

When the application receives an HTTP GET message for http://contoso.com/artists/Northwind/Overdone, it should return a resource related to Northwind's Overdone album. There's clearly a pattern to the URI. It consists of some base information (contoso.com/artists) and some URI segments (or "holes") that will contain the values of the artist and album.

It is also common to embed the same sort of information in query string parameters. Though the format is different from the previous examples, the end result is the same. Here is a set of URI holes with a query string parameter:

contoso.com/artists/Flaming+Hammer?album=HitMe
contoso.com/artists/Northwind?album=Overdone

In this case, the URI and the query string are using the following syntax:

contoso.com/artists/[artist]?album=[album]

HTTP uses an extensible set of headers for expressing additional information. This information can be related to caching, the type of data in the message, the name of the sending application, the length of the content, the host operating system, and much more. Figure 1 shows an HTTP GET message (lines 1-8) and the reply message (lines 9-14) with several common headers.

Figure 1 HTTP GET Request and Response (Close Figure 1)
1    GET /PictureServices/Feed.svc/picture/Pictures;Bridge.jpg HTTP/1.1
2    Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
       application/x-ms-application, application/vnd.ms-xpsdocument,
       application/xaml+xml, application/x-ms-xbap,
       application/vnd.ms-excel, application/vnd.ms-powerpoint,
       application/msword, application/x-shockwave-flash,
       application/x-silverlight, */*
3    Accept-Language: en-us
4    UA-CPU: x86
5    Accept-Encoding: gzip, deflate
6    User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;
       .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322;
       .NET CLR 3.5.20706; .NET CLR 3.0.590; MS-RTC LM 8)
7    Host: www.cloudsamples.net
8    Proxy-Connection: Keep-Alive

9    HTTP/1.1 200 OK
10   Content-Type: image/jpeg
11   Server: Microsoft-IIS/7.0
12   X-Powered-By: ASP.NET
13   Date: Sat, 15 Sep 2007 18:57:11 GMT
14   Content-Length: 106333

The Accept header in the HTTP GET request indicates which data formats the client would like to receive. As you can see from the long list of values, the client can receive several kinds of data (images, office documents, Silverlight apps, and so on). The response to the HTTP GET contains data whose format is described by the value of the Content-Type header (image/jpeg as shown on line 10). This simple mechanism allows the sending and receiving application to coordinate data formats. Using MIME types in headers for data format coordination is not as expressive as the Web Services Description Language (WSDL) and XML Schema Definition (XSD) grammars of SOAP Web services, but it's enough for the Web.

REST and the Web

By looking at some key parts of HTTP, you have also seen some of the important principles that hold the Web together. In 2000, a PhD student named Roy Fielding finished a dissertation that, among other things, codified these principles into an architectural style known as Representational State Transfer (REST). (You can find Roy's dissertation atics.uci.edu/~fielding/pubs/dissertation/top.htm.)

Books, software conferences, and a seemingly innumerable collection of blogs have discussed the REST architectural style in fine-grained detail. It's not possible to cover all of its facets here, but the important principles to take away are:

  • Embrace the URI
  • HTTP GET is special
  • Content-Type is the data model

By baking these principles into your applications, you dramatically increase the interoperability and reach of your application's functionality. Interoperability and reach are key drivers behind most Web services initiatives. After all, a Web service is really a way to expose some functionality via a structured messaging scheme. The principles of the Web (HTTP request/response) are actually describing a subset of possible messaging schemes. Practically all platforms, applications, operating systems, and users understand the principles of the Web, so embracing them in your Web service application gives it improved reach and interoperability.

HTTP Programming with WCF

Building on the foundation of these Web basics, let's take a look at using WCF in the .NET Framework 3.5 to build applications that adhere to the principles of the Web. I'll start by examining some of the new building blocks of the HTTP programming model and progress upwards through the system.

If you're going to "embrace the URI," you will probably have to get used to building and parsing URIs. When done by hand, this task is error-prone and tedious. However, the new System.UriTemplate and System.UriTemplateMatch types do the heavy lifting of creating and parsing URIs, and are some of the fundamental building blocks for the new HTTP capabilities of WCF.

The UriTemplate type is the basic building block for defining the URI holes discussed earlier. The constructor of the UriTemplate type accepts a String that represents the holes in a URI. Once instantiated, a UriTemplate object exposes two instance methods for binding text values to the holes defined at instantiation. These methods return a URI whose holes are populated with values.

A UriTemplate object also exposes an instance method named Match for extracting the values of the holes of a System.Uri object. The Match instance method accepts two System.Uri objects. The first Uri is a base address, and the second Uri is candidate for matching. A System.UriTemplateMatch object is returned from the Match method. The UriTemplateMatch object contains a collection of values for the holes defined at UriTemplate instantiation time.

The code in Figure 2 illustrates how to use the UriTemplate and UriTemplateMatch types to round-trip a Uri. The result of this code is the following output:

Figure 2 Creating a Template-Based URI (Close Figure 2)
// create a URI bound to the template
Uri baseAddress = new Uri(@"http://localhost:2000");
UriTemplate template = new UriTemplate("{artist}?album={album}");
Uri boundUri = template.BindByPosition(baseAddress, "Northwind",
    "Overdone");
Console.WriteLine(boundUri.ToString());

// retrieve the value of the artist segment
UriTemplateMatch match = template.Match(baseAddress, boundUri);
String bandName = match.BoundVariables["artist"];
Console.WriteLine("the name of the artist is  {0}", bandName);

http://localhost:2000/Northwind?album=Overdone
the name of the artist is  Northwind

The System.UriTemplate and System.UriTemplateMatch types are missing the usual WCF System.ServiceModel moniker in the namespace. That's because their functionality is such that you don't have to use them inside a WCF application. They are quite handy anytime you need to create or parse a URI.

Though the UriTemplate and UriTemplateMatch types are quite easy to use, the WCF team wanted to make it even easier to use these two types in a WCF service. The goal was to offer a simple and intuitive way to map URI segments and query string parameters into application functionality. I think the model as shown in the following code fits the bill:

[ServiceContract]
public interface IPictureService
{
  [OperationContract]
  [WebGet(UriTemplate = "picture/{pictureId}")]
  Stream GetPicture(String pictureId);

  [OperationContract]
  [WebGet(UriTemplate = "picture/t/{pictureId}")]
  Stream GetPictureThumbnail(String pictureId);
}

The UriTemplate surfaces as an instance property on the WebGetAttribute, which, in turn, is applied to a service contract. Notice that the token between the curly braces maps to the name of the method parameter. Though this sample shows only one parameter, you can add many parameters, arrange them in any order, add them as query string parameters, and even use wildcards.

At run time, the value of the UriTemplate property is passed to the UriTemplate constructor. Both the Client runtime and the Dispatcher use the UriTemplate property value in this manner. The Dispatcher uses it to match incoming messages to an operation, and the Client uses it to ensure that a method on a proxy gets sent to the right URI. It's an easy-to-use, simple, and effective way to "embrace the URI" in your services.

HTTP Verbs in WCF Contracts
Additional Resources

  • Windows Communication Foundation Developer Center
  • .NET Framework Developer Center
  • Visual Studio 2008 Downloads
  • PictureServices Sample App

HTTP programming with WCF in the .NET Framework 3.5 makes it easy to map operations in a service contract to HTTP verbs. As you can probably tell from the name, applying the WebGetAttribute to an operation makes that operation available via an HTTP GET. As you saw in the previous code snippet, the WebGetAttribute defines an instance property named UriTemplate.

There are other instance properties on the WebGetAttribute type. The most notable are RequestFormat and ResponseFormat. As their names imply, the values of these properties dictate the message format for the operation. The RequestFormat and ResponseFormat properties are of type WebMessageFormat, an enumerated type that has two values: Xml and Json. The WCF infrastructure uses the values of these properties to set up the channel stack with the appropriate message encoder. A property value of WebMessageFormat.Xml will result in a channel stack that uses the XML encoder. Setting the value to WebMessageFormat.Json will result in a channel stack that uses the JSON encoder included in the .NET Framework 3.5. Since there are properties for both sides of the HTTP message exchange, it is possible for an application to receive an XML message and return a JSON message (or vice-versa).

The other attribute related to HTTP verbs is WebInvokeAttribute. Applying this attribute to an operation makes that operation available by any HTTP verb other than HTTP GET. The separation in the object model between HTTP GET and all other verbs reflects the frequency HTTP GET is used versus the other HTTP verbs. The WebInvokeAttribute object model is similar to the WebGetAttribute, but it includes an instance property named Method. The value of this property associates a particular HTTP verb to the operation. The Method property is of type String, so you can set the value to any standard HTTP verb, and even non-standard HTTP verbs. Figure 3 illustrates how to use the WebInvokeAttribute in a service contract.

Figure 3 Using WebInvokeAttribute (Close Figure 3)
[ServiceContract]
public interface IPictureService
{
  [OperationContract]
  [WebGet(UriTemplate = "picture/{pictureId}")]
  Stream GetPicture(String pictureId);

  [OperationContract]
  [WebGet(UriTemplate = "picture/t/{pictureId}")]
  Stream GetPictureThumbnail(String pictureId);

  [OperationContract]
  [WebInvoke(UriTemplate="update", Method="POST")]
  void UpdatePictureInfo(PictureInfo info);
}

The UpdatePictureInfo method has a parameter of type PictureInfo. At run time, a serialized version of a PictureInfo object is the payload of an HTTP POST message. Passing data this way allows applications to ship complex data types that are not otherwise expressible in a URI.

WCF in the .NET Framework 3.5 also makes it easy to interact with HTTP headers. Request and response HTTP headers are exposed via the System.ServiceModel.Web.WebOperationContext type. The WebOperationContext type is an extension of the System.ServiceModel.OperationContext type introduced in WCF in the .NET Framework 3.0, and its usage pattern is similar. Both are intended to be used within the implementation of a service object. The WebOperationContext type exposes members that simplify reading or setting HTTP header values, as well as retrieving information about the URI used to access the service object. The HTTP headers are stored as a collection, and the most common ones are exposed as individual properties. This example shows how to use the WebOperationContext type to follow the data model of the Web by setting the Content-Type header for an HTTP response:

public Stream GetPicture(string pictureId)
{
  // retrieve the Stream (omitted)
  Stream stream;

  // set the Content-Type to image/jpeg
  WebOperationContext.Current.OutgoingResponse.ContentType =
    "image/jpeg";

  return stream;
}

Binding and Hosting

So far, you've seen how to map URI segments to application functionality, how to map operations to HTTP verbs, and how to interact with HTTP headers. Now let's see how to connect these features together into a functioning service. To do this, you'll need to look at a few more types that are new in WCF.

WCF in the .NET Framework 3.0 introduced the concept of a binding, the abstraction over the transport and protocols of an endpoint. The .NET Framework 3.0 shipped with several bindings that included support for a wide array of transport and protocol choices.

WCF in the .NET Framework 3.5 includes a new binding named System.ServiceModel.WebHttpBinding. This binding is the abstraction of the principles of the Web and follows the same usage model as the binding types in the .NET Framework 3.0, so you can add it to an endpoint just like you would any other binding.

WCF in the .NET Framework 3.0 also introduced the concept of an endpoint behavior. A behavior is a way to extend the execution path of the messaging infrastructure. There are several behaviors that ship with the .NET Framework, and it is also easy to create your own.

WCF in the .NET Framework 3.5 includes a new endpoint behavior named System.ServiceModel.Description.WebHttpBehavior. This behavior does several tasks. For receiving applications, it sets up the filtering infrastructure to ensure, among other things, that received messages are dispatched to the appropriate method on the service object. In the .NET Framework 3.0 WCF used a combination of SOAP Action and target address as dispatch keys. The filtering mechanism setup by the WebHttpBehavior extends the existing filtering mechanisms to key off of UriTemplate matches and HTTP verbs.

Figure 4 shows how you can use WebHttpBinding and WebHttpBehavior to build a receiving application that embraces the principles of the Web. The advantage of this approach is that you can add one of these endpoints to an existing ServiceHost and by doing this you end up with one ServiceHost that has both SOAP/WS-* endpoints and REST endpoints.

Figure 4 Using WebHttpBinding and WebHttpBehavior (Close Figure 4)
ServiceHost host = new ServiceHost(typeof(IPictureContract),
    new Uri("http://localhost:5000"));
// instantiate a WebHttpBinding
WebHttpBinding binding = new WebHttpBinding();

// add an endpoint, using the WebHttpBinding
ServiceEndpoint ep = host.AddServiceEndpoint(
    typeof(IPictureContract),  binding, String.Empty);

// add the WebHttpBehavior to the endpoint
ep.Behaviors.Add(new WebHttpBehavior());

// open the ServiceHost to start listening
host.Open();

Sending applications follow the same usage model, but the WebHttpBehavior should be added to the endpoint behavior collection on the ChannelFactory<T> object. As per typical WCF practice, these options are also available in configuration files.

WCF in the .NET Framework 3.5 takes this easy-to-use model even further so that you don't have to add either WebHttpBinding or WebHttpBehavior in some cases. The new API includes two types that serve this purpose: System.ServiceModel.Web.WebServiceHost and System.ServiceModel.Web.WebServiceHostFactory. The WebServiceHost type automatically adds the WebHttpBehavior to all endpoints and does some extra validation at run time to ensure that all endpoints are compatible with the behavior. This saves you the step of adding the WebHttpBehavior to an endpoint.

The WebServiceHostFactory type was designed for use in IIS-hosted scenarios to remove the need for any imperative or configuration file hosting information. WCF in the .NET Framework 3.0 introduced the .svc file as an activation target for hosting WCF services in IIS. These files contain directives that look similar to page directives in ASP.NET. Adding WebServiceHostFactory to the Factory property of the .svc file directive creates a ServiceHost, adds an endpoint that uses the appropriate contract with the WebHttpBinding, adds WebHttpBehavior to that endpoint, and opens the ServiceHost:

<%@ ServiceHost
  Language="C#"
  Service="PictureService"
  Factory="System.ServiceModel.Web.WebServiceHostFactory" %>

The web.config file for the service does not contain any WCF-related entries.

The New Syndication API

Imagine an RSS feed. If you are at all like me, you immediately think of a news feed or a blog. Syndications—RSS and ATOM are syndication formats—are able to express much more. In the abstract, syndicated content is simply a way to represent a set of data. That set of data could be practically anything: the number of wombats in residence within a 50-mile radius of Sydney, the last 10 purchase orders over $100, or the number of boomerangs Contoso produced last month.

The .NET Framework 3.5 includes rich support for creating and consuming syndicated content. It supports creating and consuming both RSS 2.0 and ATOM 1.0 formats, various means of adding syndication extensions, and even the capability to implement additional formats. The programming model shields the developer from syndication format particulars, thereby helping to make it easy to use.

Up until version 3.5, there was no standard way to create or consume syndicated content with the .NET Framework. Though the new syndication functionality was written by the WCF team, it is not dependent on WCF. In fact, all of the syndication-related types are part of the System.Syndication namespace, though they reside in the System.ServiceModel.Web.dll assembly. You can use the syndication capabilities of the .NET Framework 3.5 from any process that hosts an AppDomain (ASP.NET, WPF applications, NT Services, and so on). This capability implies that you can serve or consume syndicated content over any transport, not just HTTP. However, when coupled with the HTTP programming model in WCF, you can add syndication to existing SOAP/WS-* services as well.

The new syndication API includes types that abstract an individual syndication feed and the items in a feed, as well as types that can transform that feed into a particular format. The System.Syndication.SyndicationFeed type is a format-neutral representation of a syndication feed. A SyndicationFeed contains a list of SyndicationItem objects. A SyndicationFeed without a set of SyndicationItem objects is like a pod without any peas, because a SyndicationItem object is just a representation of an item in a feed.

Once a SyndicationFeed has been populated with a set of SyndicationItem objects, a SyndicationFeedFormatter<T> can transform that feed into a particular format. There are two types derived from SyndicationFeedFormatter<T>: Rss20FeedFormatter and Atom10FeedFormatter. As their names imply, these types transform an instance of a SyndicationFeed into the RSS 2.0 and ATOM 1.0 formats, respectively.

Creating a SyndicationFeed

There are two ways to create a SyndicationFeed object. You can instantiate one and populate its members manually, or you can populate the entire SyndicationFeed from an existing feed. Both are easy to do and shield the developer from the quirks of a particular wire format. This code demonstrates how to build a SyndicationFeed object manually.

SyndicationFeed feed = new SyndicationFeed();
feed.Title.Text = "The Cybertopian Chronicle";

Title is just one of many properties you can set, so check the documentation for additional possibilities.

It is often the case that you want to read an existing feed and do something based on the contents of that feed. In the new syndication API, you can instantiate a SyndicationFeed and automatically populate its state from an existing feed. All you need to start is the URI of the existing feed or an XmlReader that's poised to read one. This code shows how to connect to an existing feed on the Web and extract some information from it:

Uri feedUri = new Uri("http://blogs.msdn.com/justinjsmith/atom.xml");
SyndicationFeed feed = SyndicationFeed.Load(feedUri);
Console.WriteLine(feed.Title.Text);

// outputs "The Cybertopian Chronicle"

The SyndicationItem type contains more than 35 members. Many of these are properties related to setting or retrieving fields like the item identifier, last updated time, published date, title, or the actual content. There are also numerous members that make it easy to extend the content stored in a SyndicationItem. There are many extensions to RSS and ATOM (Microsoft Simple List Extensions, Yahoo Media RSS, and GeoRSS, among many others), and both SyndicationFeed and SyndicationItem can be extended to include any existing RSS or ATOM extension.

Feeds can have many items, and loading them all at once is not an option for large feeds. The SyndicationFeed addresses this situation by exposing its set of SyndicationItem object as an IEnumerable<SyndicationItem>. This implementation makes it easier to deal with large numbers of SyndicationItem objects because it leverages the iterator capability of the .NET Framework 2.0. It's also possible to iterate over a set of SyndicationItem objects using LINQ. This can dramatically reduce the amount of code required to extract information from a feed.

The syndication API defines several types that transform a SyndicationFeed into the RSS 2.0 and ATOM 1.0 formats. In fact, you can build one SyndicationFeed, populate it with SyndicationItem objects, then expose that feed as both RSS and ATOM. Figure 5 demonstrates how to retrieve an ATOM 1.0 feed, transform it to RSS 2.0, and output the new RSS representation to the console.

Figure 5 Transforming a Feed (Close Figure 5)
// read an ATOM feed
Uri feedUri = new Uri("http://blogs.msdn.com/justinjsmith/atom.xml");
SyndicationFeed feed = SyndicationFeed.Load(feedUri);

// transform it to RSS
Rss20FeedFormatter formatter = new Rss20FeedFormatter(feed);
XmlWriter writer = XmlWriter.Create(Console.Out, null);

// write it to the Console
formatter.WriteTo(writer);
writer.Flush();

When you partner the syndication API with the HTTP programming model in WCF, you are able to expose a feed from a customized URI and return RSS or ATOM based on the composition of that URI. Figure 6 shows how to define an operation in a service contract that uses a query string parameter from a received HTTP GET to return either an RSS or an ATOM feed. Notice the use of the SyndicationFeedFormatter<SyndicationFeed> in the operation contract. Both Rss20FeedFormatter and Atom10FeedFormatter derive from SyndicationFeedFormatter<TSyndicationFeed>.

Figure 6 Exposing a Feed from a Customized URI (Close Figure 6)
[ServiceKnownType(typeof(Atom10FeedFormatter))]
[ServiceKnownType(typeof(Rss20FeedFormatter))]
[ServiceContract]
interface IPictureService {
  [OperationContract]
  [WebGet(UriTemplate="Pictures?format={format}")]
  SyndicationFeedFormatter<SyndicationFeed> Feed(String format);
}

class PictureService : IPictureService {

  public SyndicationFeedFormatter<SyndicationFeed> Feed(String format){
    // create the syndication feed
    SyndicationFeed feed = new SyndicationFeed();

    // add the items to the feed (omitted)

    // check the argument & return the right format
    if(format.ToLower() == "rss"){
      return new Rss20FeedFormatter(feed);
    }
    return new Atom10FeedFormatter(feed);

  }

}

Tying It All Together

To summarize, you've seen some of the basics of HTTP, how this relates to principles of the Web, how to use WCF to bake those principles into your application, and how to use the new syndication API to serve and consume RSS and ATOM feeds. To further demonstrate the capabilities of these new features, you can see them running together in a sample application called PictureServices at www.cloudsamples.net/pictureservices. You can run the sample online, browse the source code, and download the entire sample. Figure 7 shows PictureServices in action.

Figure 7 The PictureServices Feed
Figure 7 The PictureServices Feed(Click the image for a smaller view)
Figure 7 The PictureServices Feed
Figure 7 The PictureServices Feed (Click the image for a larger view)

In a nutshell, PictureServices can retrieve pictures from a local machine or from a Flickr feed. The application uses a simple provider model, and there are currently three providers: Windows Desktop Search, a folder on your file system, and Flickr. Once PictureServices retrieves the files from a provider, it syndicates those pictures and serves them locally. From a browser you can click through a feed and retrieve individual pictures. There's quite a bit more in the sample, so feel free to explore. For links to more information about the technologies discussed here, see the "Additional Resources" sidebar in this article.

Justin Smith works at Microsoft as a Technical Evangelist devoted to Web services. His areas of focus are WCF and BizTalk Services. Justin is also the author of Inside Windows Communication Foundation (Microsoft Press, 2007) and a frequent presenter at software development conferences. Read his blog at blogs.msdn.com/justinjsmith.


SQL Server: Uncover Hidden Data to Optimize Application Performance

SQL Server

Uncover Hidden Data to Optimize Application Performance

Ian Stirk
Code download available at: DMVsinSQLServer2008_01.exe (155KB)
This article discusses:
  • Dynamic management views and functions
  • Causes of server waits
  • Common index problems
  • Problems associated with logical I/O
This article uses the following technologies:
SQL Server
Contents
Causes of Server Waits
Reads and Writes
Missing Indexes by Database
Costly Missing Indexes
Unused Indexes
Costly Used Indexes
Often-Used Indexes
Logically Fragmented Indexes
Costly Queries by I/O
Costly Queries by CPU
Costly CLR Queries
Most-Executed Queries
Queries Suffering from Blocking
Lowest Plan Reuse
Further Work

M
any application performance problems can be traced to poorly performing database queries; however, there are many ways you can improve database performance. SQL ServerTM 2005 gathers a lot of information that you can use to identify the causes of such performance issues.

SQL Server 2005 collects data relating to running queries. This data, which is held in memory and starts accumulating after a server restart, can be used to identify numerous issues and metrics, including those surrounding table indexes, query performance, and server I/O. You can query this data via the SQL Server Dynamic Management Views (DMVs) and related Dynamic Management Functions (DMFs). These are system-based views and functions that present server state information that can be used to diagnose problems and tune database performance.

In this article, I will highlight areas where performance can be improved by using information that is already being gathered by SQL Server 2005. This approach is largely non-intrusive, as it collects and examines existing data, typically querying underlying system data.

I will demonstrate how to obtain this information, discuss the underlying DMVs, identify any caveats to be aware of when interpreting the data, and point you to additional areas where you might be able to realize performance improvements. To do this, I will present and examine a series of SQL scripts that detail the various aspects of the data collected by SQL Server 2005. A complete and fully commented version of this script can be downloaded from the MSDN® Magazine Web site.

Some of the steps I will discuss concentrate on the server as a whole, including all the databases hosted on a given server. When necessary, it is possible to concentrate on a given database by adding the appropriate filtering, such as adding its name to the query.

Conversely, some of the queries join to the sys.indexes DMV, which is a database-specific view that reports results only for the current database. In these cases, I amended the queries to iterate over all the databases on the server by using the system stored procedure sp_MSForEachDB, thus presenting server-wide results.

In order to target the most relevant records for a given performance metric, I will limit the number of records returned using the SQL TOP function.

Causes of Server Waits

A user typically experiences poor performance as a series of waits. Whenever a SQL query is able to run but is waiting on another resource, it records details about the cause of the wait. These details can be accessed using the sys.dm_os_wait_stats DMV. You can examine the accumulated cause of all the waits using the SQL script shown in Figure 1.

Figure 1 SQL Query Records Causes of Wait Times (Close Figure 1)
SELECT TOP 10
 [Wait type] = wait_type,
 [Wait time (s)] = wait_time_ms / 1000,
 [% waiting] = CONVERT(DECIMAL(12,2), wait_time_ms * 100.0
               / SUM(wait_time_ms) OVER())
FROM sys.dm_os_wait_stats
WHERE wait_type NOT LIKE '%SLEEP%'
ORDER BY wait_time_ms DESC;
Figure 1 SQL Query Records Causes of Wait Times
Figure 1 SQL Query Records Causes of Wait Times

The result of running this script lists the wait type ordered by the total time spent waiting. In my sample results, you can see that I/O is ranked relatively high as a cause of waiting. Note that I am only interested in logical I/O (reading/writing data in memory), rather than physical I/O, since after the initial load the data is typically in memory.

Reads and Writes

High I/O usage can be an indicator of poor data access mechanisms. SQL Server 2005 keeps track of the total number of reads and writes that each query uses to fulfill its needs. You can sum these numbers to determine which databases perform the most overall reads and writes.

The sys.dm_exec_query_stats DMV contains aggregate performance statistics for cached query plans. This includes information about the number of logical reads and writes and the number of times the query has executed. When you join this DMV to the sys.dm_exec_sql_text DMF, you can sum the number of reads and writes by database. Notice that I use the new SQL Server 2005 CROSS APPLY operator to handle this join. The script I use to identify which databases are using the most reads and writes is shown in Figure 2.

Figure 2 Identifying the Most Reads and Writes (Close Figure 2)
SELECT TOP 10
        [Total Reads] = SUM(total_logical_reads)
        ,[Execution count] = SUM(qs.execution_count)
        ,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
GROUP BY DB_NAME(qt.dbid)
ORDER BY [Total Reads] DESC;

SELECT TOP 10
        [Total Writes] = SUM(total_logical_writes)
        ,[Execution count] = SUM(qs.execution_count)
        ,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
GROUP BY DB_NAME(qt.dbid)
ORDER BY [Total Writes] DESC;
Figure 2 Identifying the Most Reads and Writes
Figure 2 Identifying the Most Reads and Writes

The results indicate which databases are reading and writing the most logical pages. The top set of data is sorted by Total Reads, while the bottom set is sorted by Total Writes.

As you can clearly see, DatabaseName is set to NULL in a couple of instances. This setting identifies ad hoc and prepared SQL statements. This detail is useful for identifying the degree of usage of native SQL—which in itself is a potential cause of many different problems. (For example, this indicates that query plans are not being reused, code is not being reused, and there is a potential concern in the area of security.)

A high value for tempdb may suggest excessive use of temporary tables, excessive recompiles, or an inefficient device. The results could be used to identify which databases are used primarily for reporting (lots of selecting of data) as opposed to transactional databases (lots of updates). Each database type, reporting or transactional, has different indexing needs. I will examine this in more detail in a moment.

Missing Indexes by Database

When SQL Server processes a query, the optimizer keeps a record of the indexes it attempts to use to satisfy the query. If these indexes are not found, SQL Server creates a record of the missing index. This information can be viewed using the sys.dm_db_missing_index_details DMV.

You can show which databases on a given server are missing indexes using the script shown in Figure 3. Discovering these missing indexes is important because the indexes will often provide an ideal path to retrieving query data. In turn, this can reduce I/O and improve overall performance. My script examines sys.dm_db_missing_index_details and sums up the number of missing indexes per database, making it easy to determine which databases need further investigation.

Figure 3 Identifying Missing Databases (Close Figure 3)
SELECT
    DatabaseName = DB_NAME(database_id)
    ,[Number Indexes Missing] = count(*)
FROM sys.dm_db_missing_index_details
GROUP BY DB_NAME(database_id)
ORDER BY 2 DESC;
Figure 3 Identifying Missing Databases
Figure 3 Identifying Missing Databases

Databases are quite often divided into transactional and reporting-based systems. It should be a relatively easy operation for you to apply the suggested missing indexes to the reporting databases. The transactional databases, on the other hand, will typically require further investigation into the impact of the additional indexes on the underlying table data.

Costly Missing Indexes

Indexes will have varying levels of impact on query performance. You can drill down into the most costly missing indexes across all the databases on the server, finding out which missing indexes are likely to have the most significant positive impact on performance.

The sys.dm_db_missing_index_group_stats DMV notes the number of times SQL has attempted to use a particular missing index. The sys.dm_db_missing_index_details DMV details the missing index structure, such as the columns required by the query. These two DMVs are linked together via the sys.dm_db_missing_index_groups DMV. The cost of the missing index (the total cost column) is calculated as the product of the average total user cost and the average user impact multiplied by the sum of the user seeks and user scans.

You can use the script shown in Figure 4 to identify the most costly missing indexes. The results of this query, which are ordered by Total Cost, show the cost of the most important missing indexes along with details about the database/schema/table and the columns required in the missing indexes. Specifically, this script identifies which columns are used in equality and inequality SQL statements. Additionally, it reports which other columns should be used as included columns in a missing index. Included columns allow you to satisfy more covered queries without obtaining the data from the underlying page, thus using fewer I/O operations and improving performance.

Figure 4 Cost of Missing Indexes (Close Figure 4)
SELECT  TOP 10
        [Total Cost]  = ROUND(avg_total_user_cost * avg_user_impact * (user_seeks + user_scans),0)
        , avg_user_impact
        , TableName = statement
        , [EqualityUsage] = equality_columns
        , [InequalityUsage] = inequality_columns
        , [Include Cloumns] = included_columns
FROM        sys.dm_db_missing_index_groups g
INNER JOIN    sys.dm_db_missing_index_group_stats s
       ON s.group_handle = g.index_group_handle
INNER JOIN    sys.dm_db_missing_index_details d
       ON d.index_handle = g.index_handle
ORDER BY [Total Cost] DESC;
Figure 4 Cost of Missing Indexes
Figure 4 Cost of Missing Indexes(Click the image for a smaller view)
Figure 4 Cost of Missing Indexes
Figure 4 Cost of Missing Indexes (Click the image for a larger view)

Note that the results do not specify the order in which the columns in the required index should be created. To determine this, you should inspect your collective SQL code base. As a general rule, the most selected columns should appear first in the index.

I should also point out that only the user columns (such as user_seeks and user_scans) are considered in calculating the cost of the missing index. This is because system columns tend to represent using statistics, Database Consistency Checking (DBCC), and Data Definition Language (DDL) commands, and these are less critical to fulfilling the business functionality (as opposed to the database admin functionality).

It is very important for you to remember that you need to take special consideration with regard to the potential cost of the additional index when any data modifications occur in the underlying table. For this reason, additional research into the underlying SQL code base should be undertaken.

If you find that numerous columns are recommended as columns to include, you should examine the underlying SQL since this may indicate that the catchall "SELECT *" statement is being used—if this turns out to indeed be the case, you should probably revise your select queries.

Unused Indexes

Unused indexes can have a negative impact on performance. This is because when the underlying table data is modified, the index may need to be updated also. This, of course, takes additional time and can even increase the probability of blocking.

When an index is used to satisfy a query and when it is updated as a result of updates applied to the underlying table data, SQL Server updates the corresponding index usage details. These usage details can be viewed to identify any unused indexes.

The sys.dm_db_index_usage_stats DMV tells how often and to what extent indexes are used. It is joined to the sys.indexes DMV, which contains information used in the creation of the index. You can inspect the various user columns for a value of 0 to identify unused indexes. The impact of the system columns is again ignored for the reasons discussed above. The script shown in Figure 5 will let you identify the most costly unused indexes.

Figure 5 Identifying Most Costly Unused Indexes (Close Figure 5)
-- Create required table structure only.
-- Note: this SQL must be the same as in the Database loop given in the following step.
SELECT TOP 1
        DatabaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
        ,user_updates
        ,system_updates
        -- Useful fields below:
        --, *
INTO #TempUnusedIndexes
FROM   sys.dm_db_index_usage_stats s
INNER JOIN sys.indexes i ON  s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE  s.database_id = DB_ID()
    AND OBJECTPROPERTY(s.[object_id], 'IsMsShipped') = 0
    AND    user_seeks = 0
    AND user_scans = 0
    AND user_lookups = 0
    AND s.[object_id] = -999  -- Dummy value to get table structure.
;

-- Loop around all the databases on the server.
EXEC sp_MSForEachDB    'USE [?];
-- Table already exists.
INSERT INTO #TempUnusedIndexes
SELECT TOP 10
        DatabaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
        ,user_updates
        ,system_updates
FROM   sys.dm_db_index_usage_stats s
INNER JOIN sys.indexes i ON  s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE  s.database_id = DB_ID()
    AND OBJECTPROPERTY(s.[object_id], ''IsMsShipped'') = 0
    AND    user_seeks = 0
    AND user_scans = 0
    AND user_lookups = 0
    AND i.name IS NOT NULL    -- Ignore HEAP indexes.
ORDER BY user_updates DESC
;
'

-- Select records.
SELECT TOP 10 * FROM #TempUnusedIndexes ORDER BY [user_updates] DESC
-- Tidy up.
DROP TABLE #TempUnusedIndexes
Figure 5 Identifying Most Costly Unused Indexes
Figure 5 Identifying Most Costly Unused Indexes(Click the image for a smaller view)
Figure 5 Identifying Most Costly Unused Indexes
Figure 5 Identifying Most Costly Unused Indexes (Click the image for a larger view)

The results of this query show the indexes that have not been used to retrieve data but have been updated as a result of changes in the underlying table. These updates are shown in the user_updates and system_updates columns. The results are sorted by the number of user updates that have been applied to the index.

You must collect enough information to ensure that the index is not used—you don't want to inadvertently remove an index that is perhaps critical for a query that is run only quarterly or annually. Also, note that some indexes are used to constrain the insertion of duplicate records or for ordering of data; these factors must also be considered before removing any unused indexes.

The basic form of the query is applied only to the current database, since it joins to the sys.indexes DMV, which is only concerned with the current database. You can extract results for all the databases on the server using the system stored procedure sp_MSForEachDB. The pattern I use to do this is explained in the sidebar "Looping through All Databases". I use this pattern in other sections of the script as well where I want to iterate over all the databases on the server. Additionally, I filtered out indexes that are of type heap, since these represent the native structure of a table without a formal index.

Costly Used Indexes

It can also be helpful to identify the indexes (among those that are used) that are most costly in terms of changes to the underlying tables. This cost has a negative impact on performance, but the index itself may be important for data retrieval.

The sys.dm_db_index_usage_stats DMV lets you see how often and to what extent indexes are used. This DMV is joined to the sys.indexes DMV, which contains details used in the creation of the index. Inspecting the user_updates and system_updates columns will show the indexes that are highest maintenance. Figure 6 provides the script used to identify the most costly indexes and shows the results.

Figure 6 Identifying the Most Costly Indexes (Close Figure 6)
-- Create required table structure only.
-- Note: this SQL must be the same as in the Database loop given in the following step.
SELECT TOP 1
        [Maintenance cost]  = (user_updates + system_updates)
        ,[Retrieval usage] = (user_seeks + user_scans + user_lookups)
        ,DatabaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
INTO #TempMaintenanceCost
FROM   sys.dm_db_index_usage_stats s
INNER JOIN sys.indexes i ON  s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE s.database_id = DB_ID()
    AND OBJECTPROPERTY(s.[object_id], 'IsMsShipped') = 0
    AND (user_updates + system_updates) > 0 -- Only report on active rows.
    AND s.[object_id] = -999  -- Dummy value to get table structure.
;

-- Loop around all the databases on the server.
EXEC sp_MSForEachDB    'USE [?];
-- Table already exists.
INSERT INTO #TempMaintenanceCost
SELECT TOP 10
        [Maintenance cost]  = (user_updates + system_updates)
        ,[Retrieval usage] = (user_seeks + user_scans + user_lookups)
        ,DatabaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
FROM   sys.dm_db_index_usage_stats s
INNER JOIN sys.indexes i ON  s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE s.database_id = DB_ID()
    AND i.name IS NOT NULL    -- Ignore HEAP indexes.
    AND OBJECTPROPERTY(s.[object_id], ''IsMsShipped'') = 0
    AND (user_updates + system_updates) > 0 -- Only report on active rows.
ORDER BY [Maintenance cost]  DESC
;
'

-- Select records.
SELECT TOP 10 * FROM #TempMaintenanceCost
ORDER BY [Maintenance cost]  DESC
-- Tidy up.
DROP TABLE #TempMaintenanceCost
Figure 6 Identifying the Most Costly Indexes
Figure 6 Identifying the Most Costly Indexes(Click the image for a smaller view)
Figure 6 Identifying the Most Costly Indexes
Figure 6 Identifying the Most Costly Indexes (Click the image for a larger view)

The results reveal the most high-maintenance indexes along with details about the database/table involved. The Maintenance cost column is calculated as the sum of the user_updates and system_updates columns. The usefulness of the index (shown in the Retrieval usage column) is calculated as the sum of the various user_* columns. It is important that you consider an index's usefulness when deciding whether an index can be removed.

Where bulk modifications of the data are involved, these results can help you identify indexes that should be removed before applying updates. You can then reapply these indexes after all the updates have been made.

Often-Used Indexes
Looping through All Databases

The sys.indexes DMV is a database-specific view. Therefore, queries that join to sys.indexes report results only for the current database. You can, however, use the system stored procedure sp_MSForEachDB to iterate over all the databases on the server and thus present server-wide results. Here's the pattern I use in these cases.

  1. I create a temporary table with the required structure similar to the main body of code. I give it a record that does not exist (an object_id of -999) so that the temporary table structure can be created.
  2. The main body of code executes, looping around all the databases on the server. Note that the number of records retrieved for each database (using the TOP statement) should be the same as the number of records I want to display. Otherwise, the results may not be truly representative of the TOP n records across all the databases on the server.
  3. The records are extracted from the temporary table and ordered by the column that I am interested in (in this case, the user_updates column).

You can use DMVs to identify which indexes are used most often—these are the most common paths to underlying data. These are indexes that could provide significant overall performance improvements if they could themselves be improved or optimized.

The sys.dm_db_index_usage_stats DMV contains details about how often the indexes are used to retrieve data via seeks, scans, and lookups. This DMV is joined to the sys.indexes DMV, which contains details used in the creation of the index. The Usage column is calculated as the sum of all the user_* fields. This can be done using the script shown in Figure 7. The results of this query show the number of times the index has been used, sorted by Usage.

Figure 7 Identifying Most-Used Indexes (Close Figure 7)
-- Create required table structure only.
-- Note: this SQL must be the same as in the Database loop given in the -- following step.
SELECT TOP 1
        [Usage] = (user_seeks + user_scans + user_lookups)
        ,DatabaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
INTO #TempUsage
FROM   sys.dm_db_index_usage_stats s
INNER JOIN sys.indexes i ON s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE   s.database_id = DB_ID()
    AND OBJECTPROPERTY(s.[object_id], 'IsMsShipped') = 0
    AND (user_seeks + user_scans + user_lookups) > 0
-- Only report on active rows.
    AND s.[object_id] = -999  -- Dummy value to get table structure.
;

-- Loop around all the databases on the server.
EXEC sp_MSForEachDB    'USE [?];
-- Table already exists.
INSERT INTO #TempUsage
SELECT TOP 10
        [Usage] = (user_seeks + user_scans + user_lookups)
        ,DatabaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
FROM   sys.dm_db_index_usage_stats s
INNER JOIN sys.indexes i ON s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE   s.database_id = DB_ID()
    AND i.name IS NOT NULL    -- Ignore HEAP indexes.
    AND OBJECTPROPERTY(s.[object_id], ''IsMsShipped'') = 0
    AND (user_seeks + user_scans + user_lookups) > 0 -- Only report on active rows.
ORDER BY [Usage]  DESC
;
'

-- Select records.
SELECT TOP 10 * FROM #TempUsage ORDER BY [Usage] DESC
-- Tidy up.
DROP TABLE #TempUsage
Figure 7 Identifying Most-Used Indexes
Figure 7 Identifying Most-Used Indexes(Click the image for a smaller view)
Figure 7 Identifying Most-Used Indexes
Figure 7 Identifying Most-Used Indexes (Click the image for a larger view)

The most-used indexes represent the most important access routes to the underlying data. Obviously, you don't want to remove these indexes; however, they are worth examining to ensure they are optimal. For example, you should make sure index fragmentation is low (especially for data that is retrieved in sequence) and that underlying statistics are up to date. And you should remove any unused indexes on the tables.

Logically Fragmented Indexes

Logical index fragmentation indicates the percentage of entries in the index that are out of order. This is not the same as the page-fullness type of fragmentation. Logical fragmentation has an impact on any order scans that use an index. When possible, this fragmentation should be removed. This can be achieved with a rebuild or reorganization of the index.

You can identify the most logically fragmented indexes using the following DMVs. The sys.dm_db_index_physical_stats DMV lets you view details about the size and fragmentation of indexes. This is joined to the sys.indexes DMV, which contains details used in the creation of the index.

Figure 8 shows the script used to identify the most logically fragmented indexes. The results, which are sorted by the percent of fragmentation, show the most logically fragmented indexes across all databases, along with the database/table concerned. Note that this can take a while to run initially (several minutes), so I've commented it out in the script download.

Figure 8 Identifying Logically Fragmented Indexes (Close Figure 8)
-- Create required table structure only.
-- Note: this SQL must be the same as in the Database loop given in the -- following step.
SELECT TOP 1
        DatbaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
        ,[Fragmentation %] = ROUND(avg_fragmentation_in_percent,2)
INTO #TempFragmentation
FROM sys.dm_db_index_physical_stats(db_id(),null, null, null, null) s
INNER JOIN sys.indexes i ON s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE s.[object_id] = -999  -- Dummy value just to get table structure.
;

-- Loop around all the databases on the server.
EXEC sp_MSForEachDB    'USE [?];
-- Table already exists.
INSERT INTO #TempFragmentation
SELECT TOP 10
        DatbaseName = DB_NAME()
        ,TableName = OBJECT_NAME(s.[object_id])
        ,IndexName = i.name
        ,[Fragmentation %] = ROUND(avg_fragmentation_in_percent,2)
FROM sys.dm_db_index_physical_stats(db_id(),null, null, null, null) s
INNER JOIN sys.indexes i ON s.[object_id] = i.[object_id]
    AND s.index_id = i.index_id
WHERE s.database_id = DB_ID()
      AND i.name IS NOT NULL    -- Ignore HEAP indexes.
    AND OBJECTPROPERTY(s.[object_id], ''IsMsShipped'') = 0
ORDER BY [Fragmentation %] DESC
;
'

-- Select records.
SELECT TOP 10 * FROM #TempFragmentation ORDER BY [Fragmentation %] DESC
-- Tidy up.
DROP TABLE #TempFragmentation
Figure 8 Identifying Logically Fragmented Indexes
Figure 8 Identifying Logically Fragmented Indexes(Click the image for a smaller view)
Figure 8 Identifying Logically Fragmented Indexes
Figure 8 Identifying Logically Fragmented Indexes (Click the image for a larger view)

Costly Queries by I/O

I/O is a measure of the number of reads/writes a query makes. This can be used as an indicator of how efficient a query is—queries that use a lot of I/O are often good subjects for performance improvements.

The sys.dm_exec_query_stats DMV provides aggregate performance statistics for cached query plans, including details about physical and logical reads/writes and the number of times the query has executed. It contains offsets used to extract the actual SQL from its contained parent SQL. This DMV is joined to the sys.dm_exec_sql_text DMF, which contains information about the SQL batch that the I/O relates to. The various offsets are applied to this batch to obtain the underlying individual SQL queries. The script is shown in Figure 9. The results, which are sorted by average I/O, show the average I/O, total I/O, individual query, parent query (if individual query is part of a batch), and the database name.

Figure 9 Identifying Most Costly Queries by I/O (Close Figure 9)
SELECT TOP 10
 [Average IO] = (total_logical_reads + total_logical_writes) / qs.execution_count
,[Total IO] = (total_logical_reads + total_logical_writes)
,[Execution count] = qs.execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
         (CASE WHEN qs.statement_end_offset = -1
            THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
          ELSE qs.statement_end_offset END - qs.statement_start_offset)/2)
        ,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
ORDER BY [Average IO] DESC;
Figure 9 Identifying Most Costly Queries by I/O
Figure 9 Identifying Most Costly Queries by I/O(Click the image for a smaller view)
Figure 9 Identifying Most Costly Queries by I/O
Figure 9 Identifying Most Costly Queries by I/O (Click the image for a larger view)

Since I/O is a reflection of the amount of data, the query shown in the Individual Query column can help you determine areas where I/O can be reduced and performance improved. It is possible to feed the query into the Database Tuning Advisor to determine whether any indexes/statistics should be added to improve the query's performance. Statistics include details about the distribution and density of the underlying data. This is used by the query optimizer when determining an optimal query access plan.

It might also be useful to check to see if there is a link between the table in these queries and the indexes listed in the Missing Indexes section. (But note that it is important to investigate the impact of creating indexes on tables that experience many updates since any additional indexes will increase the time taken to update the underlying table data.)

The script could be altered to report only reads or only writes, which is useful for reporting databases or transactional databases, respectively. You might also want to report on the total or average value and sort accordingly. High values for reads can suggest missing or incomplete indexes or badly designed queries or tables.

Some caution should be taken when interpreting results that use the sys.dm_exec_query_stats DMV. For example, a query plan could be removed from the procedure cache at any time and not all queries get cached. While this will affect the results, the results should nonetheless be indicative of the most costly queries.

Costly Queries by CPU

Another rather useful approach that you can take is to analyze the most costly queries in terms of CPU usage. This approach can be very indicative of queries that are performing poorly. The DMVs that I will use here are the same as the ones I just used for examining queries in terms of I/O. The query that you see in Figure 10 allows you to identify the most costly queries as measured by CPU usage.

Figure 10 Identifying Most Costly Queries by CPU (Close Figure 10)
SELECT TOP 10
 [Average CPU used] = total_worker_time / qs.execution_count
,[Total CPU used] = total_worker_time
,[Execution count] = qs.execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
         (CASE WHEN qs.statement_end_offset = -1
            THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
          ELSE qs.statement_end_offset END -
qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
ORDER BY [Average CPU used] DESC;
Figure 10 SQL Query Records Causes of Wait Times
Figure 10 SQL Query Records Causes of Wait Times(Click the image for a smaller view)
Figure 10 SQL Query Records Causes of Wait Times
Figure 10 SQL Query Records Causes of Wait Times (Click the image for a larger view)

This query returns the average CPU usage, the total CPU usage, the individual query and parent query (if the individual query is part of a batch), and the corresponding database name. And, as noted previously, it is probably worthwhile to run the Database Tuning Advisor on the query to determine if further improvements can be made.

Costly CLR Queries

SQL Server is increasingly making use of the CLR. Therefore, it can be helpful to determine which queries make the most use of the CLR, which includes include stored procedures, functions, and triggers.

The sys.dm_exec_query_stats DMV contains details about total_clr_time and the number of times the query has executed. It also contains offsets used to extract the actual query from its contained parent query. This DMV is joined to the sys.dm_exec_sql_text DMF, which contains information about the SQL batch. The various offsets are applied to obtain the underlying SQL. Figure 11 shows the query used to identify the most costly CLR queries.

Figure 11 Identifying Most Costly CLR Queries (Close Figure 11)
SELECT TOP 10
 [Average CLR Time] = total_clr_time / execution_count
,[Total CLR Time] = total_clr_time
,[Execution count] = qs.execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
         (CASE WHEN qs.statement_end_offset = -1
            THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
          ELSE qs.statement_end_offset END - qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats as qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
WHERE total_clr_time <> 0
ORDER BY [Average CLR Time] DESC;
Figure 11 Identifying Most Costly CLR Queries
Figure 11 Identifying Most Costly CLR Queries(Click the image for a smaller view)
Figure 11 Identifying Most Costly CLR Queries
Figure 11 Identifying Most Costly CLR Queries (Click the image for a larger view)

This returns the average CLR time, the total CLR time, the execution count, the individual query, the parent query, and the database name. Once again, it is worthwhile to run the Database Tuning Advisor on the query to determine if further improvements can be made.

Most-Executed Queries

You can modify the previous sample for costly CLR queries to identify the queries that are executed most often. Note that the same DMVs apply here. Improving the performance of a query that is executed very often can provide a more substantial performance improvement than optimizing a large query that is rarely run. (For a sanity check, this could be cross-checked by examining those queries that use the most accumulated CPU or I/O.) Another benefit of improving a frequently run query is that it also provides an opportunity to reduce the number of locks and transaction length. The end result is, of course, you've improved overall system responsiveness.

You can identify the queries that execute most often, using the query shown in Figure 12. Running this will display the execution count, individual query, parent query (if individual query is part of a batch), and the related database. Once again, it is worthwhile to run the query in the Database Tuning Advisor to determine if further improvements can be made.

Figure 12 Identifying Queries that Execute Most Often (Close Figure 12)
SELECT TOP 10
 [Execution count] = execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
         (CASE WHEN qs.statement_end_offset = -1
            THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
          ELSE qs.statement_end_offset END - qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
ORDER BY [Execution count] DESC;
Figure 12 Identifying Queries that Execute Most Often
Figure 12 Identifying Queries that Execute Most Often(Click the image for a smaller view)
Figure 12 Identifying Queries that Execute Most Often
Figure 12 Identifying Queries that Execute Most Often (Click the image for a larger view)

Queries Suffering from Blocking

Queries that suffer the most from blocking are typically long-running queries. After identifying these queries, you can analyze them to determine whether they can and should be rewritten to reduce blocking. Causes of blocking include using objects in an inconsistent order, conflicting transaction scopes, and updating of unused indexes.

The sys.dm_exec_query_stats DMV, which I've already discussed, contains columns that can be used to identify the queries that suffer the most from blocking. Average time blocked is calculated as the difference between total_elaspsed_time and total_worker_time, divided by the execution_count.

The sys.dm_exec_sql_text DMF contains details about the SQL batch that the blocking relates to. The various offsets are applied to this to obtain the underlying SQL.

Using the query shown in Figure 13, you can identify the queries that suffer the most from blocking. The results show the average time blocked, total time blocked, execution count, individual query, parent query, and related database. While these results are sorted by Average Time Blocked, sorting by Total Time Blocked can also be useful.

Figure 13 Identifying Queries Most Often Blocked (Close Figure 13)
SELECT TOP 10
 [Average Time Blocked] = (total_elapsed_time - total_worker_time) / qs.execution_count
,[Total Time Blocked] = total_elapsed_time - total_worker_time
,[Execution count] = qs.execution_count
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
         (CASE WHEN qs.statement_end_offset = -1
            THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
          ELSE qs.statement_end_offset END - qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
ORDER BY [Average Time Blocked] DESC;
Figure 13 Identifying Queries Most Often Blocked
Figure 13 Identifying Queries Most Often Blocked(Click the image for a smaller view)
Figure 13 Identifying Queries Most Often Blocked
Figure 13 Identifying Queries Most Often Blocked (Click the image for a larger view)

If you examine the query, you might find design problems (such as missing indexes), transaction problems (resources used out of order), and so on. The Database Tuning Advisor may also highlight possible improvements.

Lowest Plan Reuse

One of the advantages of using stored procedures is that the query plan is cached and can be reused without compiling the query. This saves time, resources, and improves performance. You can identify the query plans that have the lowest reuse to further investigate why the plans are not being reused. You may find that some can be rewritten to optimize reuse.

Figure 14 shows the script I've written to identify queries with the lowest plan reuse. This technique uses DMVs that I've already discussed, along with one I haven't yet mentioned: dm_exec_cached_plans. This DMV also contains details about query plans that have been cached by SQL Server. As you can see, the results provide the number of times a plan has been used (the Plan usage column), the individual query, the parent query, and the database name.

Figure 14 Identifying Queries with Lowest Plan Reuse (Close Figure 14)
SELECT TOP 10
 [Plan usage] = cp.usecounts
,[Individual Query] = SUBSTRING (qt.text,qs.statement_start_offset/2,
         (CASE WHEN qs.statement_end_offset = -1
            THEN LEN(CONVERT(NVARCHAR(MAX),
qt.text)) * 2 ELSE qs.statement_end_offset END -
qs.statement_start_offset)/2)
,[Parent Query] = qt.text
,DatabaseName = DB_NAME(qt.dbid)
,cp.cacheobjtype
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS qt
INNER JOIN sys.dm_exec_cached_plans as cp on qs.plan_handle=cp.plan_handle
WHERE cp.plan_handle=qs.plan_handle
ORDER BY [Plan usage] ASC;
Figure 14 Identifying Queries with Lowest Plan Reuse
Figure 14 Identifying Queries with Lowest Plan Reuse(Click the image for a smaller view)
Figure 14 Identifying Queries with Lowest Plan Reuse
Figure 14 Identifying Queries with Lowest Plan Reuse (Click the image for a larger view)

You can then examine the individual queries that come up in order to identify the reason for these plans not being reused more often, if at all. One possible reason is that the query is being recompiled each time it runs—this can occur if the query contains various SET statements or temporary tables. For a more detailed discussion about recompiles and plan caching, see the paper "Batch Compilation, Recompilation, and Plan Caching Issues in SQL Server 2005" (available at microsoft.com/technet/prodtechnol/sql/2005/recomp.mspx).

Note that you should also ensure that the query has had ample opportunity to execute multiple times. You can use an associated SQL trace file to confirm this.

Further Work

Keep in mind that the metrics exposed by the various DMVs are not stored permanently but are only held in memory. When SQL Server 2005 restarts, these metrics are lost.

You could create tables regularly based on the output from the DMVs, storing the results with a timestamp. You could then inspect these tables in order of timestamp to identify the impact of any application changes or the impact of a given job or time-based processing. For instance, what is the impact of month-end processing procedures?

Similarly, you could correlate a given trace file workload with the changes in such tables to determine the impact of a given workload (such as daily or month-end) on missing indexes, most used queries, or so on. The script I've included can be edited to create these tables, running on a periodic basis as part of an ongoing maintenance task.

It is also possible to create custom reports with Visual Studio 2005 that use the script discussed in this article. These can be easily integrated into SQL Server Management Studio, providing a more pleasing representation of the data.

Wherever possible, you should attempt to integrate the methods I've described with other methodologies, such as tracing and ratio analysis. This will give you a more complete picture of the changes needed to improve your database performance.

I've demonstrated here the usefulness of the wealth of information that SQL Server 2005 accumulates during the normal course if its work. Querying this information provides positive leads that should prove useful in the ongoing endeavor to improve query performance. You may, for example, discover the cause of server waits, locate unused indexes that are having a negative impact on performance, and determine which are the most common queries and which queries are the most costly. The possibilities are numerous once you start exploring this hidden data. There is a lot more to learn about DMVs and I hope this article has whetted your appetite to investigate them further.

Ian Stirk has been working in IT as a developer, designer, and architect since 1987. He holds the following qualifications: M.Sc., MCSD, MCDBA, and SCJP. Ian is a freelance consultant working with Microsoft technologies in London, England. He can be contacted at Ian_Stirk@yahoo.com.


Look it Up: Managing Directory Security Principals in the .NET Framework 3.5

Look it Up

Managing Directory Security Principals in the .NET Framework 3.5

Joe Kaplan and Ethan Wilansky
This article is based on a prerelase version of Visual Studio 2008. All information herein is subject to change.
Code download available at: DSAccountManagement2008_01.exe (160KB)
This article discusses:
  • The System.DirectoryServices.AccountManagement class
  • Active Directory Domain Services
  • Active Directory Lightweight Directory Services (AD LDS)
  • Managing user, computer, and group principals
This article uses the following technologies:
The .NET Framework 3.5, Visual Studio 2008
Contents
Directory Services Programming Architecture
Establishing Context
Creating a User Account
Creating Groups and Computers
Managing Group Membership
Finding Ourselves
Finding Matches
Difficult Search Operations Made Easy
Authenticating Users
Extensibility Model
Final Thoughts

D
irectories are an important though rarely mastered component of enterprise application development. For the Windows® platform, Microsoft provides three primary directory platforms: Active Directory® Domain Services, the local Security Account Manager (SAM) data store on every Windows computer, and the relatively new Active Directory Lightweight Directory Services or AD LDS (which you may have previously known as Active Directory Application Mode or simply ADAM). While most enterprise developers know at least the basics of SQL programming, far fewer have experience programming directory services.

The original version of the Microsoft® .NET Framework provided a set of classes for programming directory services within the System.DirectoryServices namespace. These classes were a simple managed interop layer over an existing COM-based OS component (specifically, the Active Directory Service Interfaces or ADSI). While the programming model was reasonably powerful, it was more generalized and pared down than the full ADSI model.

In the .NET Framework 2.0, Microsoft has added features to its System.DirectoryServices and provided two new namespaces: System.DirectoryServices.ActiveDirectory and System.DirectoryServices.Protocols. (We will refer to these as the ActiveDirectory namespace and the Protocols namespace, respectively, throughout this article.) The ActiveDirectory namespace introduced a wealth of new classes for strongly typed management of directory infrastructure-level components, such as servers, domains, forests, schema, and replication. The Protocols namespace went in a different direction, providing an alternative API for programming Lightweight Directory Access Protocol (LDAP). This worked directly with the Windows LDAP subsystem (wldap32.dll), skipping over the ADSI COM interop layer entirely (see Figure 1).

Figure 1 Microsoft Directory Services Programming Architecture
Figure 1 Microsoft Directory Services Programming Architecture(Click the image for a smaller view)
Figure 1 Microsoft Directory Services Programming Architecture
Figure 1 Microsoft Directory Services Programming Architecture (Click the image for a larger view)

Still, developers missed some of the strongly typed interfaces in ADSI that they had used for managing security principals, such as Users and Groups. You could perform most of these tasks using the more general classes in System.DirectoryServices, but these were not as easy to use as they should have been, and many tasks were downright obscure. Microsoft addressed this problem in the .NET Framework 3.5 by adding a new namespace designed specifically for managing security principals: System.DirectoryServices.AccountManagement. (We'll refer to System.DirectoryServices.AccountManagement as the AccountManagement namespace throughout this article.)

This new namespace has three primary goals: to simplify principal management operations across the three directory platforms, to make principal management operations consistent regardless of the underlying directory, and to provide reliable results for these operations so you are not required to know each and every one of the caveats and special cases.

By waiting a few years for the .NET landscape to gel, Microsoft has actually outdone its previous work in ADSI by providing an even better API for these features that takes advantage of .NET capabilities while also providing much better support for new directory platforms such as AD LDS.

Directory Services Programming Architecture

Before you continue, see this article's code download for prerequisites you should already have in place in order to use these techniques. Now let's begin. Figure 1 depicts the overall programming architecture of System.DirectoryServices. The AccountManagement namespace, like the ActiveDirectory namespace, is an abstraction layer on top of System.DirectoryServices. And System.DirectoryServices is itself an abstraction layer on top of ADSI. The AccountManagement namespace also relies on the Protocols namespace for a small set of its features, such as high-performance authentication. The dark blue shading in Figure 1 shows the parts of the directory services programming architecture on which AccountManagement relies.

Figure 2 shows key types in the AccountManagement namespace. Unlike the namespaces added in the .NET Framework 2.0, AccountManagement has a relatively small surface area. With the exception of some supporting types, such as enumerations and exception classes, the namespace is composed of three primary components: a tree of Principal-derived classes representing strongly typed User, Group, and Computer objects; a PrincipalContext class used to establish a connection to the underlying store; and a PrincipalSearcher class (with supporting types) used for finding objects in the directory.

Figure 2 Key Classes in System.DirectoryServices.AccountManagement
Figure 2 Key Classes in System.DirectoryServices.AccountManagement(Click the image for a smaller view)
Figure 2 Key Classes in System.DirectoryServices.AccountManagement
Figure 2 Key Classes in System.DirectoryServices.AccountManagement (Click the image for a larger view)

Under the hood, the AccountManagement namespace uses a provider design pattern for operating over the three supported directory platforms. As a result, the members of the various Principal classes behave similarly, regardless of the underlying directory store. This design is the key to providing simplicity and consistency.

Establishing Context

You use the PrincipalContext class to establish a connection to the target directory and specify credentials for performing operations against the directory. This approach is similar to how you would go about establishing context with the DirectoryContext class in the ActiveDirectory namespace.

The PrincipalContext constructor has a variety of overloads for providing the exact options you need for establishing context. If you've worked with the DirectoryEntry class in System.DirectoryServices, many of the PrincipalContext options will look familiar. However, three of the PrincipalContext options—ContextType, name, and container—are much more specific than the input parameters you use with the DirectoryEntry class. This specificity ensures that you use proper input parameter combinations. In System.DirectoryServices and ADSI, these three parameters of the PrincipalContext constructor are combined into a single string called the path. With these components separated, it's easier to understand what is intended by each and every part of the path.

You use the ContextType enum to specify the type of target directory: Domain (for Active Directory Domain Services), ApplicationDirectory (for AD LDS), or Machine (for the local SAM database). In contrast, when using System.DirectoryServices, you use the provider component of the path string (typically "LDAP" or "WinNT") to specify the target store. ADSI then reads this value under the hood to load the appropriate provider.

The classes in the AccountManagement namespace make this task easier and more consistent by ensuring that you only use providers supported by the Framework. It also avoids annoying provider misspellings and incorrect case issues common in ADSI and System.DirectoryServices programming. That said, AccountManagement does not support less popular ADSI providers, such as IIS and Novell Directory Services.

You use the name parameter on the PrincipalContext constructor in order to provide the name of the specific directory to connect to. This can be the name of a specific server, machine, or domain. It's important to note that if this parameter is null, AccountManagement will attempt to determine a default machine or domain for the connection based on your current security context. However, if you want to connect to an AD LDS store, you must specify a value for the name parameter.

The container parameter allows you to specify the target location in the directory for establishing context. You must not specify this parameter when using Machine ContextType, as the SAM database is not hierarchical. Conversely, you must provide a value when using ApplicationDirectory since AD LDS does not publish a defaultNamingContext attribute to be used when trying to infer the directory root object. This parameter is optional with Domain ContextType, and if it's not specified, AccountManagement will use defaultNamingContext.

The additional parameters (username, password, and the ContextOptions enum) let you supply plain text credentials if necessary and specify the various connection security options to be used.

All of the directories support the Windows Negotiate authentication method. If no option is specified for the Machine store, Windows Negotiate authentication is used. The default for Domain and ApplicationDirectory, however, is Windows Negotiate with both signing and sealing.

Note that Active Directory Domain Services and AD LDS also support LDAP simple bind. You generally want to avoid using this with Active Directory Domain Services, but it may be necessary with AD LDS if you want a user in the AD LDS store to perform principal operations.

If you specify null for the user name or password parameters, AccountManagement will use the current Windows security context. If you do specify credentials, the formats supported for user name are SamAccountName, UserPrincipalName, and NT4Name. Figure 3 shows three ways to establish context.

Figure 3 Three Examples of Establishing Context (Close Figure 3)
// create a context for a domain called Fabrikam pointed
// to the TechWriters OU and using default credentials
PrincipalContext domainContext = new PrincipalContext(
  ContextType.Domain,"Fabrikam","ou=TechWriters,dc=fabrikam,dc=com");

// create a context for the current machine SAM store with the
// current security context
PrincipalContext machineContext = new PrincipalContext(
  ContextType.Machine);

// create a context for an AD LDS store pointing to the
// partition root using the credentials for a user in the AD LDS store
// and SSL for encryption
PrincipalContext ldsContext = new PrincipalContext(
    ContextType.ApplicationDirectory, "sea-dc-02.fabrikam.com:50001",
    "ou=ADAM Users,o=microsoft,c=us",
    ContextOptions.SecureSocketLayer | ContextOptions.SimpleBind,
    "CN=administrator,OU=ADAM Users,O=Microsoft,C=US ", "pass@1w0rd01");

There is another subtle but important distinction between the AccountManagement PrincipalContext and the System.DirectoryServices DirectoryEntry classes when the bind operation occurs. PrincipalContext connects and binds to the underlying directory upon object creation, whereas DirectoryEntry doesn't bind until you perform another operation that forces the connection. As a result, with PrincipalContext, you get immediate feedback about whether a connection is successfully bound to a directory.

Creating a User Account

Now you should have a fairly good understanding of how AccountManagement uses PrincipalContext to connect and bind to a container. Next we will discuss a typical DirectoryServices operation—creating a user account. In the process, each code sample assigns a value to one mandatory attribute, adds two optional attributes, sets a password, enables the user account, and commits the changes to the directory.

Here we use the domainContext variable that was introduced in example 1 of Figure 3 to create a new UserPrincipal:

// create a user principal object
UserPrincipal user = new UserPrincipal(domainContext,
     "User1Acct", "pass@1w0rd01", true);

// assign some properties to the user principal
user.GivenName = "User";
user.Surname = "One";

// force the user to change password at next logon
user.ExpirePasswordNow();

// save the user to the directory
user.Save();

The domainContext establishes the connection to the directory and the security context used to perform the operation. Then, in a single line of code, we create a new user object, set the password, and enable it. After that, we use the GivenName and Surname properties to set the corresponding directory attributes in the underlying store. Before saving the object to the underlying directory store, the password expires, forcing the user to change the password upon first logon.

By comparison, Figure 4 demonstrates the equivalent steps required to create a user account in System.DirectoryServices. The container variable in the first code line is a DirectoryEntry class object, which uses a path for its connection. The path specifies the provider, a domain, and container (TechWriters OU). It also connects using the current user's security context. The container variable is similar to the domainContext in the previous example of creating a user principal.

Figure 4 Create an Account with System.DirectoryServices (Close Figure 4)
    DirectoryEntry container =
    new DirectoryEntry("LDAP://ou=TechWriters,dc=fabrikam,dc=com");
// create a user directory entry in the container
DirectoryEntry newUser = container.Children.Add("cn=user1Acct", "user");

// add the samAccountName mandatory attribute
newUser.Properties["sAMAccountName"].Value = "User1Acct";

// add any optional attributes
newUser.Properties["givenName"].Value = "User";
newUser.Properties["sn"].Value = "One";

// save to the directory
newUser.CommitChanges();

// set a password for the user account
// using Invoke method and IadsUser.SetPassword
newUser.Invoke("SetPassword", new object[] { "pAssw0rdO1" });

// require that the password must be changed on next logon
newUser.Properties["pwdLastSet"].Value = 0;

// enable the user account
// newUser.InvokeSet("AccountDisabled", new object[]{false});
// or use ADS_UF_NORMAL_ACCOUNT (512) to effectively unset the
// disabled bit
newUser.Properties["userAccountControl"].Value = 512;

// save to the directory
newUser.CommitChanges();

Aside from the fact that this code is quite a bit longer than the AccountManagement example, it is also more complex. We need to know more about the underlying data model in the directory and we need to know how to handle certain account management features, such as setting an initial password and enabling a user object by undoing the disabled flag. This gets a bit clunky when you have to use the reflection-based Invoke and InvokeSet methods to call underlying ADSI COM interface members. It is this complexity that has made directory services programming so frustrating for many developers.

In addition, if we were trying to create the same user account in AD LDS or the local SAM database, there would have been even more differences. AD LDS and Active Directory Domain Services use a different mechanism for enabling user accounts (the msds-userAccountDisabled attribute in AD LDS rather than the userAccountControl attribute in Active Directory Domain Services), while the SAM store requires you to call an ADSI interface member. This lack of consistency among the account management features of the three directory stores was one of the key design challenges that the AccountManagement namespace had to overcome. Now, by simply changing the PrincipalContext used to create the UserPrincipal object, you can easily switch between directory stores and get one consistent set of account management features.

The Protocols namespace introduced in the .NET Framework 2.0 doesn't address any of these needs. Its purpose is to provide systems-level LDAP programmers with a more powerful and flexible API for building LDAP-based applications. However, it provides even less abstraction over the LDAP model than System.DirectoryServices, and it does nothing to simplify the differences among various directories. In addition, it is not intended for use with the local SAM database (which is not an LDAP directory). The online code samples that accompany this article include a sample similar to the AccountManager example following Figure 3. It performs the same task but takes three times as many lines of code.

The PrincipalContext in Figure 5 shows the ContextType enumeration referring to a machine in order to target a SAM store. The next parameter targets the actual machine by name (or IP address), and the last two values provide the credentials of an account that can perform the account creation.

Figure 5 Creating a SAM Database User Account (Close Figure 5)
PrincipalContext principalContext = new PrincipalContext(
    ContextType.Machine. "computer01", "adminUser", "adminPassword");

UserPrincipal user = new UserPrincipal(principalContext,
    "User1Acct", "pass@1w0rd01", true);

//Note the difference in attributes when accessing a different store
//the attributes appearing in IntelliSense are not derived from the
//underlying store
user.Name = "User One";
user.Description = "User One";
user.ExpirePasswordNow();
user.Save();

We set the Name and Description properties because the SAM store does not contain givenName, sn, or displayName attributes, as in Active Directory Domain Services. Even though AccountManagement attempts to provide a consistent experience across all three directory stores, there are differences in the underlying models. However, it will throw an InvalidOperationException if you attempt to get an attribute that is not available in the underlying store.

Figure 3 and Figure 5 are two examples of creating user accounts; they show the consistent programming model inherent in the AccountManagement namespace when operating against any store. The PrincipalContext in Figure 6 uses ContextType.ApplicationDirectory to target an AD LDS store, as was shown in Figure 3. The next parameter shows the AD LDS server. In this case, sea-dc-02.fabrikam.com is the Fully Qualified Domain Name (FQDN) of the server hosting the AD LDS instance, and the instance is listening on port 50001 for SSL communications. Note that the code download uses non-SSL communications over port 50000. This is not secure but is fine for your own testing.

Figure 6 Creating an ADAM or AD LDS User Account (Close Figure 6)
PrincipalContext principalContext = new PrincipalContext(
    ContextType.ApplicationDirectory,
    "sea-dc-02.fabrikam.com:50001",
    "ou=ADAM Users,o=microsoft,c=us",
    ContextOptions.SecureSocketLayer | ContextOptions.SimpleBind,
    "CN=administrator,OU=ADAM Users,O=Microsoft,C=US",
    "P@55w0rd0987");

UserPrincipal user = new UserPrincipal(principalContext,
    "User1Acct", "pass@1w0rd01", true);

user.GivenName = "User";
user.Surname = "One";

user.Save();

The next parameter designates the container where you want to perform a CRUD (Create/Read/Update/Delete) operation. This example specifies a user stored in AD LDS for performing our CRUD operations, so it must use an LDAP simple bind, and it's combined with SSL for security. Even though AD LDS supports secure DIGEST authentication natively, ADSI itself does not. Once again, our example is practically identical to the two previous examples with only the PrincipalContext being significantly different.

The AccountManagement namespace provides a comprehensive set of account management features such as password expiration and account unlock. We don't have space here to demonstrate all of them, but the bottom line is that they work consistently and reliably across directory stores, and they take the hassle out of implementing such capabilities.

Creating Groups and Computers

You've seen that creating a user account is simple and consistent from one DirectoryServices store to another. This consistency extends to creating the other two supported DirectoryServices objects, groups, and computers. Like the UserPrincipal class, the GroupPrincipal and ComputerPrincipal classes inherit from the Principal abstract class and operate similarly. For example, to create a group named Group01 in Active Directory Domain Services, AD LDS, or the SAM account database, you can use this code:

GroupPrincipal group = new GroupPrincipal(principalContext,
    "Group01");
group.Save();

In each case, differences are contained in the PrincipalContext class that establishes context with the different stores. The code used to create a computer object follows a similar pattern of creating the principal object using a context and then saving the object to the target of the principal context:

ComputerPrincipal computer = new ComputerPrincipal(domainContext);
computer.DisplayName = "Computer1";
computer.SamAccountName = "Computer1$";
computer.Enabled = true;
computer.SetPassword("p@ssw0rd01");
computer.Save();

Once again, AccountManagement takes care of unifying the interaction model for all of the supported identity stores. This sample shows the proper syntax for creating a computer object that can be joined to the domain (which requires a trailing $ in the sAMAccountName attribute) but sets the display name and common name so the $ is not included. Note that since the SAM database and AD LDS do not contain the computer class, AccountManagement will only allow you to create this type of object inside a domain-based PrincipalContext. Additionally, only Active Directory Domain Services contains a variety of group scopes—global, universal, and domain local—and contains both security and distribution groups. Therefore, the GroupPrincipal class provides nullable properties that allow you to set these values when necessary.

Managing Group Membership

The AccountManagement namespace also simplifies managing group membership. Before AccountManagement, there were lots of idiosyncrasies and inconsistencies among managing groups in different stores. Managing groups with many members was programmatically difficult. In addition, you had to use COM interop to manage SAM group membership and use LDAP attributes to manage Active Directory Domain Services and AD LDS groups. But now, the Members property of the GroupPrincipal class lets you enumerate a group's membership and manage its members. It all just works.

Another seemingly simple operation that is actually difficult to achieve is getting all the groups to which a user belongs. AccountManagement provides several methods to help you. The Principal base class includes two GetGroups methods and two IsMemberOf methods that, respectively, get the group membership of any Principal type and check to see if that principal is a member of a group. Also, UserPrincipal provides a special GetAuthorizationGroups method that returns the fully expanded security group membership of any UserPrincipal type. Figure 7 shows how you can use the GetAuthorizationGroups method.

Figure 7 Using the GetAuthorizationGroups Method (Close Figure 7)
string userName = "user1Acct";

// find the user in the identity store
UserPrincipal user =
    UserPrincipal.FindByIdentity(
        adPrincipalContext,
        userName);

// get the groups for the user principal and
// store the results in a PrincipalSearchResult object
PrincipalSearchResult<Principal> results =
    user.GetAuthorizationGroups();

// display the names of the groups to which the
// user belongs
Console.WriteLine("groups to which {0} belongs:", userName);
foreach (Principal result in results)
{
    Console.WriteLine("name: {0}", result.Name);
}

Yet another tricky operation made simple with AccountManagement is the task of expanding group membership across trusted domains or with foreign security principals. The GetGroups(PrincipalContext) method on the Principal class does the heavy lifting for you.

Finding Ourselves

Another task programmers often struggle with is finding objects in the directory. While LDAP is not a particularly complex query language compared to the syntaxes that developers tackle routinely, it is nonetheless unfamiliar. Additionally, even if you know the rudiments of LDAP, it is often difficult to figure out how to use it to perform common tasks.

Once again, AccountManagement makes these tasks a breeze by helping you find objects with the FindByIdentity method. This method is part of the Principal abstract class from which the UserPrincipal, GroupPrincipal, and ComputerPrincipal classes inherit. Therefore, whenever you need to search for one of these principal types, FindByIdentity is a good friend to have.

FindByIdentity contains two overloads, both of which take a PrincipalContext and value to find. For the value, you can specify any of the supported identity types: SamAccountName, Name, UserPrincipalName, DistinguishedName, Sid, or Guid. The second overload also allows you to explicitly define the identity type you will specify as the value.

Starting with the simpler overload, here's how you can go about using FindByIdentity to return the user account we created in the previous examples:

UserPrincipal user = UserPrincipal.FindByIdentity(principalContext, "user1Acct");

Once you have a context (stored in the principalContext object), you use the FindByIdentity method to retrieve the principal object, in this case a UserPrincipal. After establishing context to any supported identity store, the code for finding the identity is always the same.

The second FindByIdentity constructor allows you to be explicit about the identity format you will specify. When you use this constructor, you must match the value's format to the identity type you specify. For instance, this code will properly return a UserPrincipal using its distinguished name provided the object exists in the directory and in the specified location:

UserPrincipal user = UserPrincipal.FindByIdentity(
    adPrincipalContext,
   IdentityType.DistinguishedName,
   "CN=User1Acct,OU=TechWriters,DC=FABRIKAM,DC=COM");

In contrast, this code will not return a UserPrincipal since the IdentityType enumeration specifies a DistinguishedName format, but the value is not in this format:

UserPrincipal user = UserPrincipal.FindByIdentity(
    adPrincipalContext,
   IdentityType.DistinguishedName,
   "user1Acct");

Format is important. For example, if you decide to use the GUID or SID IdentityTypes, you must use the standard COM GUID format and the Security Descriptor Description Language (SDDL) format, respectively, for the value. The code download for this article provides two methods (FindByIdentityGuid and FindByIdentitySid) that show the proper format. Note that you must change the GUID or SID values in these methods in order to find a match in your directory store. Obtaining either format is easy using the PrincipalSearcher class, as we'll show you in just a moment.

Now that you've found a Principal object and bound to it, you can easily perform operations on it. For example, you can add a user to a group, like so:

// get a user principal
UserPrincipal user =
    UserPrincipal.FindByIdentity(adPrincipalContext, "User1Acct");
// get a group principal
GroupPrincipal group =
    GroupPrincipal.FindByIdentity(adPrincipalContext, "Administrators");
// add the user
group.Members.Add(user);
// save changes to directory
group.Save();

Here, we use the FindByIdentity method to first find a user and then a group. Once the code obtains these principal objects, we call the Add method of the group's Members property to add the user principal to the group. Finally, we call the group's Save method to save the change to the directory.

Finding Matches

You can also use the powerful Query by Example (QBE) facility and the PrincipalSearcher class to find an object based on defined criteria. We will explain more about QBE and the PrincipalSearcher class, but first we want to examine a simple search example. Figure 8 shows how you can find all user accounts beginning with a name/cn prefix of "user" that are disabled.

Figure 8 Using PrincipalSearcher (Close Figure 8)
// create a principal object representation to describe
// what will be searched
UserPrincipal user = new UserPrincipal(adPrincipalContext);

// define the properties of the search (this can use wildcards)
user.Enabled = false;
user.Name = "user*";

// create a principal searcher for running a search operation
PrincipalSearcher pS = new PrincipalSearcher();

// assign the query filter property for the principal object
// you created
// you can also pass the user principal in the
// PrincipalSearcher constructor
pS.QueryFilter = user;

// run the query
PrincipalSearchResult<Principal> results = pS.FindAll();

Console.WriteLine("Disabled accounts starting with a name of 'user':");
foreach (Principal result in results)
{
    Console.WriteLine("name: {0}", result.Name);
}

The PrincipalContext variable, adPrincipalContext, points to an Active Directory domain, but it could just as easily point to an AD LDS application partition. After establishing context, notice that the code creates a new UserPrincipal object. This is an in-memory representation of the principal for the search operation. Once you've created this principal, you then set the properties that limit the search results. The next two code lines demonstrate some limits you can set—all disabled user accounts where the user name begins with some value. Note that the property value for the Name attribute supports wildcards.

If you're already familiar with the LDAP dialect for setting up search filters, you'll immediately appreciate why QBE is a novel and more intuitive alternative. With QBE, you set up an example object that you then use for the query operation. To clearly demonstrate that QBE is simpler than the typical DirectoryServices search dialect, here's the LDAP dialect for setting up a filter equivalent to the QBE object created in Figure 8:

(&(objectCategory=person)(objectClass=user)(name=user*)(userAccount
Control:1.2.840.113556.1.4.803:=2))

As you can see, the LDAP dialect is quite a bit more complicated, and it won't work for AD LDS because the Active Directory LDS user schema uses the msDS-UserAccountDisabled attribute instead of the userAccountControl attribute shown in the LDAP dialect. Once again, AccountManagement handles these differences for us behind the scenes.

After setting up the QBE object shown in Figure 8, we create a PrincipalSearcher object and assign its QueryFilter property that the Principal object created earlier in the code. Note that you can also pass the user principal in the PrincipalSearcher constructor, rather than setting the QueryFilter property. We then run the query, calling the FindAll method of the PrincipalSearcher and assigning the returned results to the PrincipalSearchResult generic list. The PrincipalSearchResult list stores the returned Principal objects. Lastly, the code enumerates the list of principals and displays the Name attribute of each returned principal.

Note that QBE does not work for referential attributes. That is, attributes that are not owned by the QBE object cannot be used to configure your in-memory representation of the object.

You can do a lot more in the foreach loop. For example, you can enable the disabled user accounts or delete them. If you're after just read operations, keep in mind that if you do point to some other identity store, the attributes you return must exist in that store. For example, since an AD LDS user doesn't contain the sAMAccountName attribute, it wouldn't make sense to try to return this attribute in the results.

Difficult Search Operations Made Easy

There are other powerful FindBy methods that, when coupled with the PrincipalSearchResult class, can retrieve information about user and computer principals that is otherwise difficult to retrieve. Figure 9 demonstrates how to retrieve the name of each user who changed his password today. This example uses the FindByPasswordSetTime method and the PrincipalSearchResult class. Without AccountManagement, this operation is complicated because the underlying pwdLastSet attribute is stored in the directory as a Large Integer.

Figure 9 Retrieving Users Who Reset Their Password Today (Close Figure 9)
// get today's date
DateTime dt = DateTime.Today;

// run a query
PrincipalSearchResult<Principal> results =
    UserPrincipal.FindByPasswordSetTime(
        adPrincipalContext,
        dt,
        MatchType.GreaterThanOrEquals);

Console.WriteLine("users whose password was set on {0}",
    dt.ToShortDateString());
foreach (Principal result in results)
{
    Console.WriteLine("name: {0}", result.Name);
}

The code download for this article contains examples of using other FindBy methods. They all operate similarly to that which we've shown you in Figure 9.

FindBy methods are convenient shortcuts to information that is otherwise hard to retrieve. However, they are not appropriate if you need to further filter the results using the QBE facility. An important nuance here is that the associated attribute is read-only and therefore cannot be set on a QBE object, just as it cannot be set by a user on the object to which the QBE refers. To use the QBE, you use the equivalent read-only property in your example principal object combined with the AdvancedSearchFilter property. More about that will come later. Figure 10 lists more FindBy methods and shows the equivalent read-only properties that you can use instead of the FindBy method in a search.

Figure 10 Other FindBy Methods (Close Figure 10)

Method Name | Read-Only Property | Description
FindByLogonTime | LastLogonTime | Accounts that have logged on within the specified time.
FindByExpirationTime | AccountExpirationDate | Expired accounts within the specified time.
FindByPasswordSetTime | LastPasswordSetTime | Accounts whose password was set within the specified time.
FindByLockoutTime | AccountLockoutTime | Accounts locked out within the specified time.
FindByBadPasswordAttempt | LastBadPasswordAttempt | Bad password attempts within the specified time.
No equivalent method | BadLogonCount | Accounts that have attempted to logon the specified number of times but have failed to logon.

You can't set a value on a read-only property when configuring a QBE. So how can you work with the property in a search operation? You can retrieve a result set and then perform a conditional test using the read-only property when enumerating the result set. Just keep in mind that this approach is not advisable for potentially large resultsets since the code must first retrieve results unfiltered for the read-only property and then filter the returned resultset by the read-only property. The PrincipalSearchEx6v2 method in the code download demonstrates this less-than-ideal approach.

The Directory Services team addressed this QBE limitation by adding the AdvancedSearchFilter property to the AuthenticablePrincipal class. AdvancedSearchFilter allows you to search based on the read-only properties and then combine them with other properties you can set using the QBE mechanism. Figure 11 demonstrates how you can use the LastBadPasswordAttempt read-only property of the UserPrincipal class to return a list of users who had a bad password attempt today.

Figure 11 AdvancedSearchFilter with a Read-Only Property (Close Figure 11)
DateTime dt = DateTime.Today;

// create a principal object representation to describe
// what will be searched
UserPrincipal user = new UserPrincipal(adPrincipalContext);

user.Enabled = true;

// define the properties of the search (this can use wildcards)
user.Name = "*";

//add the LastBadPasswordAttempt >= Today to the query filter
user.AdvancedSearchFilter.LastBadPasswordAttempt
    (dt, MatchType.GreaterThanOrEquals);

// create a principal searcher for running a search operation
// and assign the QBE user principal as the query filter
PrincipalSearcher pS = new PrincipalSearcher(user);

// run the query
PrincipalSearchResult<Principal> results = pS.FindAll();

Console.WriteLine("Bad password attempts on {0}:",
    dt.ToShortDateString());
foreach (UserPrincipal result in results)
{
    Console.WriteLine("name: {0}, {1}",
           result.Name,
           result.LastBadPasswordAttempt.Value);
}

Authenticating Users

Developers who build directory-based applications often need to authenticate the credentials of users stored in the directory, especially when using AD LDS. Before the .NET Framework 3.5, programmers accomplished this task using the DirectoryEntry class in System.DirectoryServices to force an LDAP bind operation under the hood. However, be careful: it is exceedingly easy to write poor versions of this code that are not secure, that are slow, or that are just plain clunky. Additionally, ADSI itself is not designed for this type of operation and can fail under high-use conditions due to the way it caches LDAP connections internally.

As we've already discussed, the System.DirectoryServices.Protocols assembly in the .NET Framework 2.0 contains lower-level LDAP classes that use a connection-based programming metaphor. This design allows you to overcome the inherent limitations in ADSI but at the expense of having to write more complicated code.

In the .NET Framework 3.5, AccountManagement delivers both the power and ease of use offered by the ActiveDirectoryMembershipProvider implementation in ASP.NET to programmers working in any environment. Additionally, the AccountManagement namespace allows you to authenticate credentials against the local SAM database if needed.

The two ValidateCredentials methods on the PrincipalContext class provide credential validation. You first create an instance of a PrincipalContext using the directory you wish to validate against and specify the appropriate options. After getting context, you test whether ValidateCredentials returns true or false based on the supplied user name and password values. Figure 12 shows an example of authenticating a user in AD LDS.

Figure 12 Authenticating a User in AD LDS (Close Figure 12)
// establish context with AD LDS
PrincipalContext ldsContext =
    new PrincipalContext(
        ContextType.ApplicationDirectory,
        "sea-dc-02.fabrikam.com:50000",
        "ou=ADAM Users,O=Microsoft,C=US");

// determine whether a user can validate to the directory
Console.WriteLine(
    ldsContext.ValidateCredentials(
        "user1@adam",
        "Password1",
        ContextOptions.SimpleBind +
        ContextOptions.SecureSocketLayer));

This approach is most useful when you want to validate many different sets of user credentials quickly and efficiently. You create a single PrincipalContext object for the directory store in question and reuse that object instance for each call to ValidateCredentials. The PrincipalContext can reuse the connection to the directory, which results in good performance and scalability. And calls to ValidateCredentials are thread-safe, so your instance can be used across threads for this operation. It's important to note that the credentials used to create the PrincipalContext are not changed by calls to ValidateCredentials—the context and method call maintain separate connections.

By default, AccountManagement uses secure Windows Negotiate authentication and attempts to use SSL when performing a simple bind against AD LDS. We recommend that you always be explicit with the type of authentication you want to perform and the connection protection you wish to use (if applicable), but at least the defaults err on the side of caution.

Active Directory Domain Services in Windows Server® 2003 and later and AD LDS both include fast concurrent binding, which is designed for high-performance authentication operations. It validates a user's password without actually building a security token for the user. Unlike in a normal bind operation, with fast concurrent binding the state of the LDAP connection remains unbound. You can use fast concurrent binding to perform bind operations on the same connection repeatedly and simply check for a failed password attempt. This feature is not an available option through ADSI or System.DirectoryServices, but it is exposed as an option in the Protocols namespace.

AccountManagement uses fast concurrent binding whenever possible and enables this option automatically. This is the reason that the AccountManagement layer also appears above the Protocols layer in Figure 1. Note that it only works in simple bind mode, which passes plain text credentials on the network. Therefore, fast concurrent binding should always be combined with SSL for security.

Extensibility Model
Directory Services Resources

  • .NET Framework 3.5 Beta 2 Download
  • System.DirectoryServices.AccountManagement Namespace Overview
  • System.DirectoryServices.AccountManagement Namespace Documentation
  • About Active Directory Lightweight Directory Services
  • Windows Server 2003 Active Directory Application Mode
  • Introduction to System.DirectoryServices.Protocols
  • Introduction to System.DirectoryServices.ActiveDirectory

Another area where AccountManagement really shines is its extensibility model. Many developers will choose to use the various Principal-derived classes for building custom provisioning systems for both Active Directory Domain Services and AD LDS. In many cases (especially with AD LDS), an organization will add custom schema extensions to the directory to support its own metadata for users and groups.

Using the .NET Framework object-oriented design and attribute-based extensible metadata, AccountManagement makes it easy to create custom security principal classes that support your custom schema. By simply inheriting from one of the Principal-derived classes and marking your class and properties with the appropriate attributes, your custom principal class can read and write these directory attributes as well as the attributes already supported by the built-in types.

An important nuance worth noting is that the extensibility mechanism provided by AccountManagement is designed for use by security principals stored in Active Directory Domain Services or AD LDS. It doesn't have a focus on non-Microsoft LDAP directories. If you wish to build a framework for provisioning in non-Microsoft LDAP directories, you should use the lower-level classes in the Protocols namespace. (In addition, the extensibility model is not intended for use with local SAM accounts, as the SAM schema is not extensible.)

Consider an AD LDS directory that uses the standard LDAP user class for storing security principals for an application. In addition, the LDAP directory schema is extended to support a special attribute for identifying user objects called msdn-subscriberID. Figure 13 demonstrates how to create a custom class that can provision user objects and also provide create, read, and write operations against this attribute.

Figure 13 Our Sample MsdnUser Class (Close Figure 13)
[DirectoryObjectClass("user")]
[DirectoryRdnPrefix("CN")]
class MsdnUser : UserPrincipal
{
    public MsdnUser(PrincipalContext context)
        : base(context) { }

    public MsdnUser(
        PrincipalContext context,
        string samAccountName,
        string password,
        bool enabled
        )
        : base(
           context,
           samAccountName,
           password,
           enabled
           )
    {
    }

    [DirectoryProperty("msdn-subscriberID")]
    public string MsdnSubscriberId
    {
        get
        {
            object[] result = this.ExtensionGet("msdn-subscriberID");
            if (result != null) {
                return (string)result[0];
            }
            else {
                return null;
            }
        }
        set { this.ExtensionSet("msdn-subscriberID", value); }
    }
}

Notice that the code inherits from the UserPrincipal class and is decorated with two attributes: DirectoryObjectClass and DirectoryRdnPrefix. Both of these attributes are required for principal extension classes. The DirectoryObjectClass attribute determines the value the supported store (Active Directory Domain Services or AD LDS) uses for the objectClass directory attribute when creating instances of this object in the directory. Here, this is still the default AD LDS user class, but in reality it could be anything. The DirectoryRdnPrefix attribute determines the RDN (relative distinguished name) attribute name to use for naming objects of this class in the directory. Under Active Directory Domain Services, you cannot change the RDN prefix—it is always CN for security principal classes. Under AD LDS, however, there is more flexibility and you can use a different RDN if desired.

Our class has a property called MsdnSubscriberID that returns a string. This class is marked with the DirectoryProperty attribute, specifying the LDAP schema attribute used to store the property value. The underlying framework uses this value for optimizing search operations against this Principal type.

Our property get and set implementations use the protected ExtensionGet and ExtensionSet methods of the Principal base class to read and write values to the underlying property cache. These methods support storing values in memory for objects that have not yet been persisted to the database/identity store. In addition, these methods support reading and writing values from existing objects. Since LDAP directories support attributes of various types and also allow an attribute to contain multiple values, these methods use the object[] type for reading and writing values. This flexibility is nice, but if you want to provide a strongly typed scalar string value on top of an array of object types, you have to do a little extra work, as our implementation demonstrates. The result for consumers of our custom MsdnUser class is an interface that is very easy to program.

The ability to provide strongly typed values on top of our directory schema is one of the most useful features of this extensibility model. Beyond simple string types, you can also use the rich type system offered by the .NET Framework to do such things as represent the Active Directory Domain Services jpgPhoto attribute as a System.Drawing.Image or a System.IO.Stream instead of the default byte[] that you would usually get by reading the value from System.DirectoryServices.

The code download for this article provides a few more samples to demonstrate these capabilities. It also has some schema extensions (via a standard LDIF formatted file, msdnschema.ldf) that you can use to extend your test directory with the MsdnUser class. We also provided some valuable links in the "Directory Services Resources" sidebar.

Final Thoughts

AccountManagement is a much-needed managed code addition to the rich directory services programming model offered by Microsoft. With the AccountManagement namespace, developers now have a set of strongly typed principals for common CRUD and search operations.

The namespace encapsulates directory service programming best practices to help you write secure and high-performance managed code. In addition, AccountManagement is extensible, allowing you to fully interact with your custom directory objects in Active Directory Domain Services and AD LDS.

Joe Kaplan works in Accenture's internal IT organization building enterprise applications using the .NET Framework. He specializes in application security, federated identity management, and directory services programming, for which he has been recognized as a Microsoft MVP. He is the co-author of The .NET Developer's Guide to Directory Services Programming (Addison-Wesley, 2006).

Ethan Wilansky is a Microsoft Directory Services MVP and an EDS enterprise architect. As part of the EDS Innovation Engineering practice, he leads a development team currently focused on building custom SharePoint applications, and he advises EDS on directory services programming solutions.


Editor's Note: A New Year—A Fresh Perspective

Editor's Note
A New Year—A Fresh Perspective
Howard Dierking

One of the most important things I've learned from working at Microsoft for a while is how much we love metrics. In any endeavor of a non-trivial size, having some sort of yardstick enables you to define what success or failure looks like before you even begin to work on the project. Think of it as Test-Driven Development for running a business.

Well, being a part of Microsoft, MSDN Magazine is no different in this respect. We recently received the results from our annual reader survey, which gives us an excellent opportunity to see how well we are doing in serving the developer community. While there's still quite a bit of analysis to perform, along with Excel pivot tables and PowerPoint decks to create, I wanted to take this opportunity to share with you some of the data points that jumped out at me. After all, the survey reflects your collective opinion—and that opinion is what we will use to help us determine what topics to focus on in the coming year.

The first question asked about what technologies you are most interested in exploring. The vast majority of you expressed interest in exploring software design in more depth. This happens to be my area of interest as a developer. Because of that, I have been a little hesitant to steer the magazine in this direction for fear of making a decision that would impact you based on my own personal preferences. As I'm sure you can imagine, this data point is a great personal validation for something I already wanted to do.

Other top-rated technology interests indicated in the survey were Web services development, database development (which we currently do not plan as much content on as the survey results indicate we should), and Web client development.

That question also had a field for direct text entry, which offered the funniest repeatable response: "I am not a programmer."

A second interesting data point related to the desired extent of coverage for technologies at different levels of maturity. An overwhelming majority of you said that you were most interested in seeing us focus on technologies that have been released over the last year. Looks like you prefer that over coverage of the other options, older technologies and pre-released technologies.

Finally, a data point that may not be as easy to act on as the first two—but that I found interesting nonetheless—is the number of you with computer science degrees. I have a minor in CS, and, after being a professional developer for the majority of my career, I have often wondered how relevant this kind of training is for the modern professional developer—and by extension, you as the reader of MSDN Magazine. In fact, in a recent interview that we conducted with the person who runs recruiting for Microsoft worldwide, we asked about how important Microsoft considers a CS degree (I'm sorry, but you'll have to wait until next month for the answer to that). However, according to the annual reader survey results, the majority of you with a bachelor's degree have that degree in computer science.

If you didn't participate in this year's survey, I hope this note has shown that we pay especially close attention to your thoughts and opinions through these surveys, and I sincerely hope that you'll participate in the next survey opportunity. Also, the ability to make good decisions based on this kind of data is predicated on asking the right questions in the survey. If you have ideas about topics you'd like to see us cover in 2008 or if you have opinions about anything we're doing here, please let us know. You can always write to us at mmeditor@microsoft.com. In the end, we want to continue working to make MSDN Magazine more relevant to you—but we can only do this if you tell us what that means.

Thanks to the following Microsoft technical experts for their help with this issue: Shawn Burke, Ben Carter, Steve Lasker, Bertrand Le Roy, Eilon Lipton, Thomas Marquardt, Diana Milirud, Larry Olson, Ladi Prosek, Travis Querec, Dave Reed, Jeff Schwartz, Varun Sekhri, Bill Staples, Chris Tavares, Stephen Toub, Varun Vaswani, and Buck Woody.


Toolbox: Advanced Unit Testing, Object Mocking, Profiling Your Apps and More

Toolbox
Advanced Unit Testing, Object Mocking, Profiling Your Apps and More
James Avery

Contents
Advanced Unit Testing
Better Testing with Mocks
Quantifying App Performance
Implementing a Continuous Integration Environment

Advanced Unit Testing

Unit testing has gradually moved from a fringe technology to standard practice on many Microsoft® .NET Framework projects. Many .NET developers have at least some experience working with nUnit, the predominant unit-testing framework for .NET. While nUnit covers most required scenarios for unit testing .NET applications, you can take unit testing to the next level with MbUnit 2.4. MbUnit is an open-source unit-testing framework started by Jonathan "Peli" de Halleux (who has since gone on to work at Microsoft Research) and is currently led by Andy Stopford along with a team of talented developers.

MbUnit starts with the general functionality that most xUnit-testing frameworks include: test fixtures with setup and teardown attributes, the familiar test attribute, a number of assertions, a console test runner, and an easy-to-use windows test runner. But MbUnit differentiates itself from other unit-testing frameworks with some very useful and innovative extras.

Row testing is a feature that allows you to pass in parameters to your unit tests. Instead of writing separate tests for each different scenario you want to examine, you can write one test with parameters and then pass in data using convenient attributes. For instance, you might have a method that accepts a book name and ISBN number and then checks if that book is in stock. Instead of writing different tests for books in stock, books out of stock, and books that aren't in the database, you can write one test method that accepts a book name and ISBN number as parameters. Then, using the Row attribute, you can specify three different sets of values for each different scenario as well as what the expected result would be. Not only does this reduce the amount of code you have to write for your tests, it also makes those tests more readable and maintainable in the future.

A similar feature, pair-wise testing, allows you to use factories to feed data into your unit test. Factories make it easy to test a very large number of data scenarios without a lot of code. Continuing the example from above, you could create a factory that returns hundreds, even thousands, of books and ISBN numbers. MbUnit is even smart enough to eliminate combinations in testing data that would be redundant based on earlier combinations.

MbUnit also includes a number of performance-oriented test attributes. The duration attribute can be used to find the amount of time it takes for your test to execute. MbUnit also gives you the ability to write tests that can read and make an assertion based on the value of a performance counter using the PerfCounter attribute.

Finally, MbUnit includes another useful feature called the Rollback attribute, which will wrap your test in a transaction and then roll back that transaction when the test is done. This way, you don't end up with a database full of test data. This feature works in both the .NET Framework 1.1 (using COM+ transactions) and in the .NET Framework 2.0 (using the new built-in transaction support).

Price: Free; source code available.

www.mbunit.com

MbUnit Provides Advanced Unit-Testing Support for .NET Developers
MbUnit Provides Advanced Unit-Testing Support for .NET Developers (Click the image for a smaller view)
MbUnit Provides Advanced Unit-Testing Support for .NET Developers
MbUnit Provides Advanced Unit-Testing Support for .NET Developers (Click the image for a larger view)

Better Testing with Mocks

The more unit tests you create, the longer it takes to run those unit tests. Ideally, unit tests should take only seconds to run so developers can run them every time they build the application. However, most applications today rely heavily on databases, Web services, and myriad other external dependencies. There is no way a large set of unit tests that use one of these dependencies, can run in seconds.

Actually, the only way your unit tests can run that quickly is if you use something to simulate the external dependency. This is where mocks come into the picture. Mocks are objects that pretend to be your external dependency so that your code can run normally using the mock object instead of the external dependency. Not only does this ensure that your tests run quickly, but perhaps more importantly, it allows you to focus on testing the functionality of your code instead of the external dependency. Moreover, you also get the ability to test the interaction between your code and the external dependencies. Instead of just testing that your business layer can successfully return an object, you can test that it will call a specific method on your data access layer to retrieve that object.

There are a number of different mocking frameworks available for .NET, and each one handles the problem of defining what a mock should expect and return a little bit differently. Rhino.Mocks, an open-source framework developed by Oren Eini (also known as Ayende Rahien), stands out because of its ability to provide strongly typed mocked objects. Most of the available mocking frameworks require the use of hard-coded strings to define method names and parameters, which means you won't know about potential issues until you actually try to run your tests at run time. By allowing for strongly typed mock objects, Rhino.Mocks gives you all the benefits of compilation—helping you catch errors earlier.

Rhino.Mocks is also flexible in the types of mocking allowed. A normal mock object will throw an exception if an unexpected method is called. But Rhino.Mocks offers a dynamic mock object that, instead of throwing an exception, will return null or 0 for any unexcepted method calls. Finally, Rhino.Mocks allows a partial mock, which also avoids throwing an exception if there is no expectation defined, but instead of returning 0 or null, it will call the original method on the object and return that value. This lets you mock only the methods needed on an object.

Rhino.Mocks also offers a number of other useful features, including support for generic types, callbacks, and the ability to mock delegates and events.

Price: Free; source code available.

www.rhinomocks.com

Quantifying App Performance

One of the main goals when writing any application is performance, yet most developers work on a daily basis with almost no way to quantify this goal. Typically, it is left to users to identify which screens or pages are running too slow, followed by a guessing game on the developer's part as to what is causing the performance issues. What developers really need is a way to quantify how an application is performing and quickly identify bottlenecks. dotTrace is an application profiler from JetBrains that makes accomplishing both tasks quick and easy.

Using dotTrace, you can profile Windows® applications, Web applications (using either IIS or the Visual Studio® Development Server), and Windows services. dotTrace offers both performance profiling, which allows you to view the amount of time each of your methods and lines of code takes to execute, as well as memory profiling, which shows the amount of memory currently being used and by what objects. Performance profiling is available for applications built on the 1.1 and 2.0 versions of the .NET Framework; memory profiling is available only for applications built using version 2.0.

Performance profiling with dotTrace shows you how many times each of your methods was called, how much total time was spent in those methods, and what percentage of the overall time a particular method consumes. These stats can be easily examined to find the parts of your application that aren't performing well. An even easier way is to use the Hot Spots feature, which shows you the 100 worst-performing parts of your application.

dotTrace includes two modes for memory profiling. You can get a dump of the memory, which lets you view the objects currently in memory and the amount of space they occupy. Or you can use the Differences mode, which lets you capture memory during a particular interval and then shows you both new objects and dead objects and the difference between the two during a profiling run. Both ways of memory profiling can be very valuable when trying to track down a memory leak.

One feature that makes dotTrace stand out is its integration with Visual Studio, which lets you easily start profiling a solution with just a couple of mouse clicks. With dotTrace, the next time you need to speed up all or part of your application, you will know exactly what needs to be optimized to gain the maximum performance improvement for the least amount of work.

Price: $499.

www.jetbrains.com/profiler

dotTrace Offers a Number of Ways to View Profiling Data that is Stored in Snapshots
dotTrace Offers a Number of Ways to View Profiling Data that is Stored in Snapshots (Click the image for a smaller view)
dotTrace Offers a Number of Ways to View Profiling Data that is Stored in Snapshots
dotTrace Offers a Number of Ways to View Profiling Data that is Stored in Snapshots (Click the image for a larger view)

Implementing a Continuous Integration Environment

There is plenty of information on the Internet and scattered throughout different books and articles on setting up and configuring a continuous integration environment. However, Continuous Integration: Improving Software Quality and Reducing Risk by Paul M. Duvall with Steve Matyas and Andrew Glover (Addison-Wesley Professional, 2007) is the first attempt I'm aware of to collect in one place all of the great reasons to use continuous integration as well as the patterns and best practices for setting up and using continuous integration on a software project.

The book starts with easy-to-understand scenarios that underscore the benefits of continuous integration—and the dangers of not using it. Anyone trying to convince his team or manager to incorporate continuous integration should find more than enough information to make a compelling case. The first section also includes some best practices concerning your actual build, like being sure you have a dedicated build server and build metrics.

The book goes on to identify best practices for continuous integration, with chapters on database integration, running and providing feedback from automated tests, code inspection and analysis, deployment, and providing feedback. These chapters are, for the most part, technology-neutral. While specific tools and code samples are used in each section, most of the advice is applicable regardless of your continuous integration platform.

The book concludes with a pair of useful appendices. The first presents a good list of tools for building your continuous integration environment. The second appendix describes the best way to choose the tools to include in your continuous integration environment and includes a more in-depth guide to the major continuous integration servers.

Price: $44.99.

www.integratebutton.com

Send your questions and comments for James to toolsmm@microsoft.com.

James Avery runs his own .NET consulting practice, Infozerk Inc. His most recent book is Windows Developer Power Tools (O'Reilly, 2006). You can e-mail him at javery@infozerk.com and read his blog at www.dotavery.com/blog.


CLR Inside Out: Marshaling between Managed and Unmanaged Code

CLR Inside Out
Marshaling between Managed and Unmanaged Code
Yi Zhang and Xiaoying Guo

Code download available at: CLRInsideOut2008_01.exe (1269KB)
Contents
[InAttribute] and [OutAttribute]
Keywords Out and Ref and Passing by Reference
Return Values
StringBuilder and Marshaling
Copying and Pinning
Memory Ownership
Reverse P/Invoke and Delegate Lifetime
P/Invoke Interop Assistant
Give It a Try

L
et's face it. The world is not perfect. Very few companies are developing entirely in managed code, and on top of that there's a lot of legacy unmanaged code out there you need to work with. How do you integrate your managed and unmanaged projects? Does it take the form of calling into unmanaged code from a managed application or calling into managed code from an unmanaged application?

Fortunately, Microsoft® .NET Framework interop opens a channel between managed and unmanaged code, and marshaling plays a very important role in that connection in that it allows for data exchange between the two (see Figure 1). There are many factors that affect the way the CLR marshals data between the unmanaged and managed worlds, including attributes such as [MarshalAs], [StructLayout], [InAttribute], and [OutAttribute] as well as language keywords such as out and ref in C#.

Figure 1 Bridging the Gap between Managed and Unmanaged Code
Figure 1 Bridging the Gap between Managed and Unmanaged Code(Click the image for a smaller view)
Figure 1 Bridging the Gap between Managed and Unmanaged Code
Figure 1 Bridging the Gap between Managed and Unmanaged Code (Click the image for a larger view)

Because of the sheer number of these factors, it can be a challenge to marshal correctly, which requires you to understand many details in both unmanaged and managed code. In this column, we'll cover the basic yet confusing topics you'll encounter in your everyday marshaling attempts. We won't cover custom marshaling, marshaling complex structures, or other advanced topics, but with a solid understanding of the basics, you'll be ready to begin tackling those as well.

[InAttribute] and [OutAttribute]

The first marshaling topic we'll discuss concerns the use of InAttribute and OutAttribute, attribute types located in the System.Runtime.InteropServices namespace. (When applying these attributes in your code, C# and Visual Basic® allow you to use the abbreviated forms [In] and [Out], but we will stick to the full names to avoid confusion.)

When applying to method parameters and return values, these attributes control marshaling direction, so they're known as directional attributes. [InAttribute] tells the CLR to marshal data from the caller to the callee at the beginning of the call, while [OutAttribute] tells the CLR to marshal back from the callee to the caller upon return. Both the caller and the callee can be either unmanaged or managed code. For example, in a P/Invoke call, managed code is calling unmanaged code. However, in a reverse P/Invoke, unmanaged code could call managed code through a function pointer.

There are four possible combinations in which [InAttribute] and [OutAttribute] can be used: [InAttribute] only, [OutAttribute] only, [InAttribute, OutAttribute], and neither. When neither attribute is specified, you are telling the CLR to figure out directional attributes by itself, with [InAttribute] usually applied by default. However, in the case of the StringBuilder class, both [InAttribute] and [OutAttribute] are applied when neither attribute is specified. (See the later section on StringBuilder for details.) Also, the use of out and ref keywords in C# could change the attributes that are applied, as shown in Figure 2. Note that if no keyword is specified for a parameter, it means that it is an input parameter by default.

Figure 2 Out and Ref and Their Associated Attributes (Close Figure 2)

C# Keyword | Attribute
(none specified) | [InAttribute]
out | [OutAttribute]
ref | [InAttribute], [OutAttribute]

Consider the code in Figure 3. There are three native C++ functions, and they are all making the same change to arg. Also, note that the use of strcpy for string manipulation is for illustrative purposes only—production code should instead use the safe versions of these functions, which can be found at msdn.microsoft.com/msdnmag/issues/05/05/SafeCandC.

Figure 3 Trying Out Directional Attributes (Close Figure 3)
MARSHALLIB_API void __stdcall Func_In_Attribute(char *arg)
{
    printf("Inside Func_In_Attribute: arg = %s\n", arg);
    strcpy(arg, "New");
}

MARSHALLIB_API void __stdcall Func_Out_Attribute(char *arg)
{
    printf("Inside Func_Out_Attribute: arg = %s\n", arg);
    strcpy(arg, "New");
}

MARSHALLIB_API void __stdcall Func_InOut_Attribute(char *arg)
{
    printf("Inside Func_InOut_Attribute: arg = %s\n", arg);
    strcpy(arg, "New");
}

The only thing that is different is how we will invoke them, using directional attributes in the P/Invoke signatures, as shown in the following C# code:

[DllImport(@"MarshalLib.dll")]
public static extern void Func_In_Attribute([In]char[] arg);
[DllImport(@"MarshalLib.dll")]
public static extern void Func_Out_Attribute([Out]char[] arg);
[DllImport(@"MarshalLib.dll")]
public static extern void Func_InOut_Attribute([In, Out]char[] arg);

If you call these functions from managed code by P/Invoke and pass in "Old" as a char array to the functions, you'll get the following output (abridged for demonstration purposes):

Before Func_In_Attribute: arg = Old
Inside Func_In_Attribute: arg = Old
After Func_In_Attribute: arg = Old

Before Func_Out_Attribute: arg = Old
Inside Func_Out_Attribute: arg =
After Func_Out_Attribute: arg = New

Before Func_InOut_Attribute: arg = Old
Inside Func_InOut_Attribute: arg = Old
After Func_InOut_Attribute: arg = New

Let's look at the results more closely. In Func_In_Attribute, the original value is passed in, but the change made inside Func_In_Attribute is not propagated back. In Func_Out_Attribute, the original value is not passed in and the change made inside Func_Out_Attribute is propagated back. In Func_InOut_Attribute, the original value is passed in and the change made inside Func_Out_Attribute is propagated back. However, things could be totally different if you make a slight change. This time, let's change the native function to use Unicode, as shown below:

MARSHALLIB_API void __stdcall Func_Out_Attribute_Unicode(wchar_t *arg)
{
    wprintf(L"Inside Func_Out_Attribute_Unicode: arg = %s\n", arg);
    printf("Inside Func_Out_Attribute_Unicode: strcpy(arg, \"New\")\n");
    wcscpy(arg, L"New");
}

Here we declare the C# function, apply only [OutAttribute], and change CharSet to CharSet.Unicode:

[DllImport(@"MarshalLib.dll", CharSet=CharSet.Unicode)]
public static extern void Func_Out_Attribute_Unicode([Out]char[] arg);

Here's the output:

Before Func_Out_Attribute_Unicode: arg = Old
Inside Func_Out_Attribute_Unicode: arg = Old
After Func_Out_Attribute_Unicode: arg = New

Interestingly, the original value is passed even though there is no [InAttribute]. The [DllImportAttribute] tells the CLR to marshal Unicode, and, because char type in the CLR is also Unicode, the CLR sees an opportunity to optimize the marshaling process by pinning the char array and passing the address of the char directly. (You'll see a detailed discussion of copying and pinning later.) However, this doesn't mean you should rely on this behavior. Instead, the correct marshaling directional attributes should always be used when not relying on the CLR's default marshaling behavior. A typical example of this default behavior is the case of an int argument; specifying [InAttribute] int arg isn't necessary.

There are cases when [OutAttribute] will be ignored. For example, [OutAttribute]int doesn't make any sense, so the [OutAttribute] is simply ignored by the CLR. The same is true of [OutAttribute] string because string is immutable.

Interface definition (IDL) files also have [in] and [out] attributes, which can be considered the same as [InAttribute] and [OutAttribute] in the CLR.

Keywords Out and Ref and Passing by Reference

Previously, we've shown that C# out and ref keywords can be directly mapped to [InAttribute] and [OutAttribute]. As a matter of fact, out and ref can also change what data type the CLR will marshal into or from. Passing data as out or ref is the same as passing by reference. If you examine the corresponding function signature in intermediate language (IL) using ILDASM, you can see there is an ampersand character (&) next to the type, which means that the argument should be passed by reference. When passing by reference, an extra level of indirection will be added by the CLR. Figure 4 shows some examples.

Figure 4 Marshaling Results (Close Figure 4)

C# Signature | Unmanaged Signature | MSIL Signature | Actual MSIL Signature Seen by the CLR
Basic Types |  |  |
int arg | int arg | int | [in] int
out int arg | int *arg | [out] int & | [out] int &
ref int arg | int *arg | int & | [in, out] int &
Structs |  |  |
MyStruct arg | MyStruct arg | MyStruct | [in] MyStruct
out MyStruct arg | MyStruct *arg | [out] MyStruct & | [out] MyStruct &
ref MyStruct arg | MyStruct *arg | MyStruct & | [in, out] MyStruct &
Strings |  |  |
string arg | char *arg | string | [in] string
out string arg | char **arg | [out] string & | [out] string &
ref string arg | char **arg | string & | [in, out] string &
Classes |  |  |
MyClass arg | MyClass *arg | MyClass | [in] MyClass
out MyClass arg | MyClass **arg | [out] MyClass & | [out] MyClass &
ref MyClass arg | MyClass **arg | MyClass & | [in, out] Myclass &

Let's summarize what we discussed for out and ref in the table shown in Figure 5.

Figure 5 Default Attributes (Close Figure 5)

C# Signature | MSIL Signature | Default Direction Attributes
<type> | type | [InAttribute]
out <type> | [OutAttribute] type & | [OutAttribute]
ref <type> | type & | [InAttribute, OutAttribute]

Please note that when passing by reference, if no directional attributes are specified, the CLR will apply [InAttribute] and [OutAttribute] automatically, which is why there is only "string &" in the Microsoft Intermediate Language (MSIL) signature in Figure 4. If any of these attributes are specified, the CLR will follow them instead of the default behavior, as shown in this example:

public static extern void
      PassPointerToComplexStructure(
        [In]ref ComplexStructure
        pStructure);

The signature above will override the default directional behavior of ref, making it [InAttribute]-only. In this particular case, if you are doing a P/Invoke, a pointer to ComplexStructure (which is a value type) is being passed from the CLR side to the native side, but the callee can't make any changes visible to the ComplexStructure pointed to by pStructure pointer. Figure 6 shows other examples of directional attributes and keyword combinations.

Figure 6 More Attributes and Keywords (Close Figure 6)

C# Signature | Unmanaged IDL Signature | MSIL Signature
Out |  |
[InAttribute] out int arg | Compiler error CS0036. An out parameter cannot have the In attribute. | N/A
[OutAttribute] out int arg | [out] int *arg | [out] int &
[InAttribute, OutAttribute] out int arg | Compiler error CS0036. An out parameter cannot have the In attribute. | N/A
Ref |  |
[InAttribute] ref int arg | [in] int *arg | [in] int &
[OutAttribute] ref int arg | Compiler error CS0662 cannot specify only Out attribute on a ref parameter. Use both In and Out attributes, or neither. | N/A
[InAttribute, OutAttribute] ref int arg | [in, out] int *arg | [in] [out] int &

Return Values

So far we have only been discussing arguments. What about values returned from functions? The CLR automatically treats a return value as if it is a normal argument using the [OutAttribute]. Also, the CLR can make the transformation to the function signature, a process controlled by the PreserveSigAttribute. If [PreserveSigAttribute] is set to false when applied to a P/Invoke signature, the CLR will map HRESULT return values to managed exceptions, and it will map [out, retval] parameters to the function's return value. So the following managed function signature

public static string extern GetString(int id);

would become the unmanaged signature:

HRESULT GetString([in]int id, [out, retval] char **pszString);

If [PreserveSigAttribute] is set to true (the default for P/Invoke), this transformation won't happen. Note that with COM functions, [PreserveSigAttribute] is usually set to false by default, though there are a number of ways to change that. For details, please consult the MSDN® documentation for TlbExp.exe and TlbImp.exe.

StringBuilder and Marshaling

The CLR marshaler has built-in knowledge of the StringBuilder type and treats it differently from other types. By default, StringBuilder is passed as [InAttribute, OutAttribute]. StringBuilder is special because it has a Capacity property that can determine the size of the required buffer at run time, and it can be changed dynamically. Therefore, during the marshaling process, the CLR can pin StringBuilder, directly pass the address of internal buffer used in StringBuilder, and allow the contents of this buffer to be changed by native code in place.

To take full advantage of StringBuilder, you'll need to follow all of these rules:

  1. Don't pass StringBuilder by reference (using out or ref). Otherwise, the CLR will expect the signature of this argument to be wchar_t ** instead of wchar_t *, and it won't be able to pin StringBuilder's internal buffer. Performance will be significantly degraded.
  2. Use StringBuilder when the unmanaged code is using Unicode. Otherwise, the CLR will have to make a copy of the string and convert it between Unicode and ANSI, thus degrading performance. Usually you should marshal StringBuilder as LPARRAY of Unicode characters or as LPWSTR.
  3. Always specify the capacity of StringBuilder in advance and make sure the capacity is big enough to hold the buffer. The best practice on the unmanaged code side is to accept the size of the string buffer as an argument to avoid buffer overruns. In COM, you can also use size_is in IDL to specify the size.

Copying and Pinning

When the CLR performs data marshaling, it has two options: copying and pinning (see msdn2.microsoft.com/23acw07k).

By default, the CLR will make a copy that will be used during marshaling. For example, if managed code is passing a string as an ANSI C-String to unmanaged code, the CLR will make a copy of the string, convert it to ANSI, and then pass the temporary's pointer to unmanaged code. That copying can be quite slow and can create performance problems.

In certain cases, the CLR is able to optimize the marshaling process by directly pinning the managed object in the Garbage Collector (GC) heap so that it cannot be relocated during the call. The pointer to the managed object (or to somewhere inside the managed object) will be passed directly to unmanaged code.

Pinning is performed when all of the following conditions are met: First, managed code must be calling native code, not the other way around. Second, the type must be blittable or must be capable of becoming blittable under certain conditions. Third, you're not passing by reference (out or ref), and fourth, the caller and callee are in the same thread context or apartment.

The second rule deserves further explanation. A blittable type is one that has a common representation in both managed and unmanaged memory. As such, blittable types do not require conversion when marshaling. A typical example of a type that is non-blittable, but can become blittable, is the char type. By default, it is non-blittable since it can be mapped to either Unicode or ANSI. However, because char is always Unicode in the CLR, it does become blittable when you specify [DllImportAttribute(CharSet= Unicode)] or [MarshalAsAttribute(UnmanagedType.LPWSTR)]. In the following example, arg can be pinned in PassUnicodeString, but it cannot be pinned in PassAnsiString:

[DllImport(@"MarshalLib.dll", CharSet = CharSet.Unicode)]
public static extern string PassUnicodeString(string arg);

[DllImport(@"MarshalLib.dll", CharSet = CharSet.Ansi)]
public static extern string PassAnsiString(string arg);

Memory Ownership

During a function call, a function can make two kinds of changes to its arguments: a reference change or an in-place change. A reference change involves changing where a pointer points; if the pointer already points to a piece of allocated memory, that memory may first need to be freed before the pointer to it is lost. An in-place change involves changing the memory at the location pointed to by the reference.

Which of these changes is made depends on the type of the argument and, most importantly, the contract between the callee and caller. However, since the CLR cannot figure out the contract automatically, it has to rely on common knowledge about types, as illustrated in Figure 7.

Figure 7 CLR Type Knowledge (Close Figure 7)

IDL Signature | Change Type
[In] Type | No change allowed
[In] Type * | No change allowed
[Out] Type * | In-place change
[In, Out] Type * | In-place change
[In] Type ** | No change allowed
[Out] Type ** | Reference change
[In, Out] Type ** | Reference change or in-place change

As previously discussed, only reference types have two levels of indirection when passing by reference (there are a few exceptions, however, such as "[MarshalAs(UnmanagedType.LPStruct)]ref Guid"), so only the pointer or reference to a reference type can be changed, as shown in Figure 8.

Figure 8 Type Change Rules (Close Figure 8)

C# Signature | Change Type
int arg | No change allowed
out int arg | In-place change
ref int arg | In-place change
string arg | No change allowed
out string arg | Reference change
ref string arg | Reference change or in-place change
[InAttribute, OutAttribute] StringBuilder arg | In-place change
[OutAttribute] StringBuilder arg | In-place change

You don't need to worry about memory ownership for an in-place change because the caller has allocated memory for the callee and the caller owns the memory. Let's take "[OutAttribute] StringBuilder" as an example here. The corresponding native type is char * (assuming ANSI), since we are not passing by reference. Data is marshaled out, not in. Memory is allocated by the caller (in this case the CLR). The size of the memory is determined by the capacity of the StringBuilder object. The callee doesn't need to concern itself with the memory.

To change the string, the callee will make the change directly to the memory itself. However, when making a reference change, it is critical to recognize who owns which memory, otherwise any number of unexpected outcomes can occur. Regarding ownership issues, the CLR follows COM-style conventions:

  • Memory passed as [in] is owned by the caller and should be both allocated by the caller and freed by the caller. The callee should not try to free or modify that memory.
  • Memory allocated by the callee and passed as [out] or returned is owned by the caller and should be freed by the caller.
  • The callee can free memory passed as [in, out] from the caller, allocate new memory for it, and overwrite the old pointer value, thereby passing it out. The new memory is owned by the caller. This requires two levels of indirection, such as char **.

In the interop world, caller/callee becomes CLR/native code. The rules above imply that in the unpinned case, if when in native code you receive a pointer to a block of memory passed to you as [out] from the CLR, you need to free it. On the other hand, if the CLR receives a pointer that is passed as [out] from native code, the CLR needs to free it. Clearly, in the first case, native code needs to do the de-allocation and in the second case, managed code needs to do de-allocation.

Since this involves memory allocation and de-allocation, the biggest problem is what function to use. There are many to choose from: HeapAlloc/HeapFree, malloc/free, new/delete, and so on. However, since the CLR uses CoTaskMemAlloc/CoTaskMemFree in the non-BSTR case and SysStringAlloc/SysStringAllocByteLen/SysStringFree in the BSTR case, you'll have to use those functions. Otherwise, it is likely that you'll get a memory leak or a crash in certain versions of Windows®. We have seen cases where malloc'd memory was passed to the CLR and the program didn't crash in Windows XP but did crash in Windows Vista®.

Besides those functions, a system-implemented IMalloc interface returned from CoGetMalloc also works fine because internally they are using the same heap. However, it is best to always stick with CoTaskMemAlloc/CoTaskMemFree and SysStringAlloc/ SysStringAllocByteLen/SysStringFree, since CoGetMalloc is subject to future change.

Let's look at an example. The GetAnsiStringFromNativeCode takes a char ** argument as [in, out] and returns a char * as [out, retval]. For the char ** argument, it can choose to call CoTaskMemFree to free the memory that is allocated by the CLR, then allocate new memory by using CoTaskMemAlloc and overwrite the pointer with a new memory pointer. Later, the CLR will free the memory and create a copy for the managed string. As for the return value, it only needs to allocate a new piece of memory by using CoTaskMemAlloc and return it to the caller. After return, the newly allocated memory is now owned by the CLR. The CLR will first create a new managed string from it and then call CoTaskMemFree to free it.

Let's consider the first option (see Figure 9). The corresponding C# function declaration is as follows:

Figure 9 Using Pointers (Close Figure 9)
MARSHALLIB_API char *__stdcall GetAnsiStringFromNativeCode(char **arg)
{
    char *szRet = (char *)::CoTaskMemAlloc(255);
    strcpy(szRet, "Returned String From Native Code");

    printf("Inside GetAnsiStringFromNativeCode: *arg = %s\n", *arg);
    printf("Inside GetAnsiStringFromNativeCode: CoTaskMemFree(*arg);
        *arg = CoTaskMemAlloc(100); strcpy(*arg, \"Changed\")\n");

    ::CoTaskMemFree(*arg);
    *arg = (char *)::CoTaskMemAlloc(100);
    strcpy(*arg, "Changed");

    return szRet;
}

class Lib
{
    [DllImport(@"MarshalLib.dll", CharSet= CharSet.Ansi)]
    public static extern string GetAnsiStringFromNativeCode(
        ref string inOutString);
}

When the following C# code makes a call to GetAnsiStringFromNativeCode

string argStr = "Before";
Console.WriteLine("Before GetAnsiStringFromNativeCode : argStr = \"" +
    argStr + "\"");
string retStr = Lib.GetAnsiStringFromNativeCode(ref argStr);
Console.WriteLine("AnsiStringFromNativeCode() returns \"" + retStr +
    "\"" );
Console.WriteLine("After GetAnsiStringFromNativeCode : argStr = \"" +
    argStr + "\"");

the output is:

Before GetAnsiStringFromNativeCode : argStr = "Before"
Inside GetAnsiStringFromNativeCode: *arg = Before
Inside GetAnsiStringFromNativeCode: CoTaskMemFree(*arg); *arg = CoTaskMemAlloc(100); strcpy(*arg, "Changed")
AnsiStringFromNativeCode() returns "Returned String From Native Code"
After GetAnsiStringFromNativeCode : argStr = "Changed"

If the native function you are going to call doesn't follow this convention, you'll have to do the marshaling by yourself to avoid memory corruption. This could easily happen because the function for an unmanaged function could return whatever it wants; it can return the same piece of memory every time or return a new block of memory allocated by malloc/new, and so forth, again depending on the contract.

Besides memory allocation, the size of the memory being passed in or out is also very important. As discussed in the StringBuilder case, it is very important to change the Capacity property so that the CLR can allocate a piece of memory big enough to hold the results. In addition, marshaling a string as [InAttribute, OutAttribute] (without out or ref and any other attribute) is a bad idea because you don't know whether the string will be big enough. You can use SizeParamIndex and SizeConst fields in MarshalAsAttribute to specify the size of the buffer. However, these attributes cannot be used when passing by reference.

Reverse P/Invoke and Delegate Lifetime

The CLR allows you to pass a delegate to the unmanaged world so that it can call the delegate as an unmanaged function pointer. In fact, what's happening is that the CLR creates a thunk, which forwards the calls from native code to the actual delegate, then to the real function (see Figure 10).

Figure 10 Using a Thunk
Figure 10 Using a Thunk(Click the image for a smaller view)
Figure 10 Using a Thunk
Figure 10 Using a Thunk (Click the image for a larger view)

Usually, you won't have to worry about the lifetime of delegates. Whenever you are passing a delegate to unmanaged code, the CLR will make sure the delegate is alive during the call.

However, if the native code keeps a copy of the pointer beyond the span of the call and intends to call back through that pointer later, you might need to use GCHandle to explicitly prevent the garbage collector from collecting the delegate. We must warn you that a pinned GCHandle could have a significantly negative impact on program performance. Fortunately, in this case, you don't need to allocate a pinned GC handle, because the thunk is allocated in the unmanaged heap and is referencing the delegate indirectly through a reference known to the GC. Therefore, it is not possible for the thunk to move around, and native code should always be able to call the delegate through the unmanaged pointer if the delegate itself is alive.

Marshal.GetFunctionPointerForDelegate can convert a delegate to a function pointer, but it doesn't do anything to guarantee the lifetime of the delegate. Consider the following function declaration:

public delegate void PrintInteger(int n);

[DllImport(@"MarshalLib.dll", EntryPoint="CallDelegate")]
public static extern void CallDelegateDirectly(
    IntPtr printIntegerProc);

If you call Marshal.GetFunctionPointerForDelegate for it and store the returned IntPtr, then pass the IntPtr to the function you are going to call, like so:

IntPtr printIntegerCallback = Marshal.GetFunctionPointerForDelegate(
    new Lib.PrintInteger(MyPrintInteger));

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

CallDelegateDirectly(printIntegerCallback);

It is possible that the delegate will be collected before you call CallDelegateDirectly, and you will get an MDA error that CallbackOnCollectedDelegate was detected. To fix that, you can either store a reference to the delegate in memory or allocate a GC handle.

If native code returns an unmanaged function pointer to the CLR, it is the responsibility of native code to keep the actual function code around. Usually this isn't a problem unless the code is in a dynamically loaded DLL or generated on the fly.

P/Invoke Interop Assistant

Understanding and remembering all the attributes and rules described so far may seem a bit daunting. After all, most developers of managed code just need to be able to quickly figure out the P/Invoke signature for a Win32® API function, paste it in their code, and be done with it. That's where the P/Invoke Interop Assistant (available at the MSDN Magazine Web site) can help. This tool can efficiently assist with conversions from C++ to managed P/Invoke signatures as well as those in the opposite direction. It even comes with a database of Win32 functions, data types, and constants, so the common task of adding a Win32 P/Invoke to your C# or Visual Basic source file is made very easy. The tool package includes two command-line tools, SigImp and SigExp, which can be used for batch file processing. A GUI tool is also found in the package, which includes the functionalities of both tools.

The GUI tool is handy for doing simple conversions. It contains three tabs: SigExp, SigImp Search, and SigImp Translate Snippet.

SigExp converts managed signature to an unmanaged signature. It reflects over managed assemblies to find all P/Invoke declarations and COM imported types. From this input it produces the corresponding native C signatures (see Figure 11).

Figure 11 P/Invoke Interop Assistant GUI Tool—SigExp
Figure 11 P/Invoke Interop Assistant GUI Tool—SigExp(Click the image for a smaller view)
Figure 11 P/Invoke Interop Assistant GUI Tool—SigExp
Figure 11 P/Invoke Interop Assistant GUI Tool—SigExp (Click the image for a larger view)

SigImp Search and SigImp Translate Snippet convert unmanaged signatures to managed signatures. They generate managed signatures and definitions in either C# or Visual Basic from native types, functions, constants, and snippets of manually entered native function signatures.

SigImp Search allows users to choose the managed code language in which they want code generated and then select a native type, procedure, or constant to perform the generation from. The tool will display a list of supported types, methods, and constants collected from Windows SDK header files (see Figure 12).

Figure 12 P/Invoke Interop Assistant GUI Tool—SigImp Search
Figure 12 P/Invoke Interop Assistant GUI Tool—SigImp Search(Click the image for a smaller view)
Figure 12 P/Invoke Interop Assistant GUI Tool—SigImp Search
Figure 12 P/Invoke Interop Assistant GUI Tool—SigImp Search (Click the image for a larger view)

SigImp Translate Snippet allows users to write their own native code snippet in the tool. The tool will then generate and display the managed code equivalent in the main window, as you see in Figure 13.

Figure 13 P/Invoke Interop Assistant GUI Tool—SigImp Translate Snippet
Figure 13 P/Invoke Interop Assistant GUI Tool—SigImp Translate Snippet(Click the image for a smaller view)
Figure 13 P/Invoke Interop Assistant GUI Tool—SigImp Translate Snippet
Figure 13 P/Invoke Interop Assistant GUI Tool—SigImp Translate Snippet (Click the image for a larger view)

For details of the GUI tool or the command-line tools in the P/Invoke Interop Assistant, please refer to the documentation packed with the tool.

Give It a Try

As you know, marshaling is a complex topic, and there are many techniques you can use to change the marshaling process to adapt to your own needs. We suggest you try some of the ideas presented here. They're sure to help you find your way through the maze we know as marshaling.

Send your questions and comments to clrinout@microsoft.com.

Yi Zhang is a Software Development Engineer on the Silverlight Shanghai team, which is part of the Server and Tool Business in China. He is currently focusing on interop and the CLR.

Xiaoying Guo is the Program Manager on the Silverlight Shanghai team, which is part of the Server and Tools Business in China. She is currently focusing on the areas of unmanaged/managed Interop within the CLR, as well as the CoreCLR on the Mac. Xiaoying is also engaged in Silverlight local customer connection.


Data Points: Accessing Data from a Mobile Application

Data Points
Accessing Data from a Mobile Application
John Papa

Code download available at: DataPoints2008_01.exe (179KB)
Contents
Mobile Development
Getting Started
Automatic Binding
Binding the Fields
Mobile Binding Options
Fine-Tuning Your Mobile App
Final Thoughts

D
ata access is an important aspect of developing applications with the Microsoft® .NET Compact Framework for Windows Mobile® devices. By using the existing architecture to send and receive data between your mobile application and your application server, you can pass data with either DataSets, custom objects, or scalar values.

In this installment of DataPoints, I want to discuss best practices for data access strategies when developing mobile applications for Smartphones. I will demonstrate how to design a data-driven Smartphone application that can display and save data over the Internet. I will show you how to communicate with a server application that exposes methods to get data using a DataSet, custom objects, and a generic List<T> of entities.

Mobile Development
Essential Tools for Mobile Development
  • .NET Compact Framework 2.0 SP2
  • Windows Mobile 6 Professional and Standard SDKs
  • Windows Mobile 5.0 SDK for Smartphone
  • Windows Mobile 5.0 SDK for Pocket PC
  • ActiveSync 4.5 (Windows XP)
  • Windows Mobile Device Center (Windows Vista)
  • Microsoft Device Emulator 2.0
  • Windows Mobile 6 Emulators
  • Windows Mobile 5.0 Smartphone Emulators
  • Windows Mobile 5.0 Smartphone Emulator with Landscape Screen
  • Windows Mobile 5.0 Pocket PC Emulators
  • Motorola Q Emulator

To develop mobile applications, you must first install the .NET Compact Framework (and its latest service pack). The latest update is the .NET Compact Framework 2.0 with SP2. Before diving in, you must make some decisions about the platform (Windows Mobile 5.0 or Windows Mobile 6) and the device (or devices) you are going to target for your application. These decisions will affect which SDKs you must install.

The Windows Mobile 5.0 SDK is available in different versions, depending on whether you are targeting a Smartphone, a Pocket PC, or a Pocket PC Phone. The Windows Mobile 6 SDK is also available in different versions for different target platforms, but the naming conventions have changed and are now a bit less intuitive. Figure 1 lists the various SDK versions and which devices they can target. It goes without saying that if you want to target each type of device, you should install each of the SDKs.

Figure 1 Windows Mobile SDKs (Close Figure 1)

Target Device | Required Windows Mobile 5.0 SDK | Required Windows Mobile 6 SDK
Smartphone | Windows Mobile 5.0 SDK for Smartphone | Windows Mobile 6 Standard SDK
Pocket PC | Windows Mobile 5.0 SDK for Pocket PC | Windows Mobile 6 Professional SDK
Pocket PC Phone | Windows Mobile 5.0 SDK for Pocket PC | Windows Mobile 6 Professional SDK

For this discussion, I am using Windows Mobile 5 for the Smartphone. The version is not critical, though, as the data-access concepts presented remain the same regardless of version.

After installing the .NET Compact Framework 2.0 with SP2 and the appropriate Windows Mobile SDK, you must also install ActiveSync (for Windows® XP) or Windows Mobile Device Center (for Windows Vista®). These are needed for the mobile device emulator to communicate with your network or the Internet.

You should also download and install an emulator that closely resembles the device you are targeting. For this column, I have opted for a landscape 320x240 screen. If your target devices have different screen sizes, you should test your app for each using all of the appropriate emulators.

If you are using Windows Vista or the Windows Mobile 6 SDK, you need to install the Device Emulator 2.0. I know that this sounds like a lot of prerequisites to put in place, so I've included a list of links with this column in the download and on the MSDN Magazine Web site at msdn.microsoft.com/msdnmag/issues/08/01/datapoints.

Getting Started

I highly recommend testing on both the emulator and the device itself to avoid those situations when the application works fine on the emulator but not on the device. My approach is to use the emulator throughout the process of developing a mobile application, then deploy the application to a real mobile device at certain checkpoints.

Before debugging, make sure you run the Device Emulator from the Tools menu in Visual Studio®. Once you choose the emulator you want, right-click on it and select Connect. The emulator will then begin to boot the Smartphone. Once the Smartphone is running in the emulator, right-click the item in the Device Manager again and select Cradle. Cradling allows the emulator to communicate with the network. Finally, in Visual Studio, select an emulator from the dropdown list in the Device toolbar (see Figure 2).

Figure 2 Selecting a Target Emulator
Figure 2 Selecting a Target Emulator

To get data, my sample application communicates with a Web service on an IIS Web server. The Web service works well for mobile applications, as it can pass both ADO.NET objects—such as the DataSet—and custom entities. First, I create a Web service project and add a WebService called CustomerWebService. I add three methods to the WebService and decorate each with the WebMethod attribute:

[WebMethod]
public Customer GetCustomer(string customerID) {
  ...
}

[WebMethod]
public DataSet GetCustomerDataSet(string customerID) {
  ...
}

[WebMethod]
public List<Customer> FindCustomerList() {
  ...
}

The GetCustomer method retrieves a single customer record in the form of a custom entity. The GetCustomerDataSet method also gets a single customer record, but this method returns the customer in a DataSet. The FindCustomerList method gets a list of customers, returning a List<Customer>, thus taking advantage of customer entities and generics. The full code for this Web service is included in the downloadable code.

Once the Web service is created, I add a Windows Mobile 5.0 Smartphone application to the solution. If you install both the Windows Mobile 5.0 and Windows Mobile 6 SDKs, you can choose the target application from this dialog. For this project, I am creating a Windows Mobile 5.0 Smartphone Device Application.

The Smartphone application will have a startup form that displays a list of customers in a dropdown list. The user can select one of the customers and then view the details of the customer on another page. The list of customers is retrieved from the Web service's FindCustomerList Web method, which brings back a List<Customer> to the form. The list can then be manually or automatically bound to the dropdown list. (I will demonstrate both techniques shortly.)

Automatic Binding

The CustomerPicker form is the first screen the user will see. I add a dropdown list to this form and name it ddlCustomer. I want to take advantage of automatic binding and bind my dropdown list to the List<Customer> that I will get back from my FindCustomerList Web method. I add a new data source to the project, selecting the type of entity to which I want to bind the dropdown list. Since I reference my Web service, the application knows what types of entity that my WebMethods return. This makes it easy to bind to the dropdown list with strongly typed entities.

I go to the Data Sources window and click on Add New Data Source—this pops up the wizard. I choose to add an object type, click next, and locate my Customer entity from the provided list (see Figure 3). If you don't see your entities in the list when adding a new data source, make sure you have first successfully compiled the Web service project and then go back and try again to add the data source. When an entity is exposed by the Web service, it will not show up in the list unless the DLL has first been compiled.

Figure 3 Creating a Customer Data Source
Figure 3 Creating a Customer Data Source(Click the image for a smaller view)
Figure 3 Creating a Customer Data Source
Figure 3 Creating a Customer Data Source (Click the image for a larger view)

Once the data source has been added, I drag the new Customer Data Source (see Figure 4) from the Data Sources window to the dropdown list. This automatically creates a BindingSource control on the form named customerBindingSource, and this handles the two-way binding of the Customer entity to the controls.

Figure 4 Using a Customer Data Source
Figure 4 Using a Customer Data Source

There are two more steps to set up the automatic binding before I can test my application in the emulator. The next step is to set the DisplayMember and the ValueMember properties of the dropdown list. I set the DisplayMember property to the CompanyName and the ValueMember property to the CustomerID. Since the data source is strongly typed, these values are in a dropdown list in the properties window for the dropdown list control.

The last step is to tell the BindingSource control where to get its data. This is as simple as creating an instance of the Web service proxy class, invoking the appropriate method, and setting its returning List<Customer> to the BindingSource control's DataSource property, like so:

private void CustomerPicker_Load(object sender, EventArgs e)
{
    using (RemoteService.CustomerWebService svc =
                        new RemoteService.CustomerWebService())
        customerBindingSource.DataSource = svc.FindCustomerList();
}

At this point, you can run the application and test it to see if it displays the list of customers in the dropdown list. Make sure that you have set the target device in Visual Studio, that you are running ActiveSync® or Windows Mobile Device Center (as appropriate), and that the device you are emulating is cradled (use the Device Emulator Manager).

Binding the Fields

After creating and testing the CustomerPicker form, I then create another form called CustomerDetail. This will display the selected customer. I change the data source settings so that the details controls are created when I drag the Customer data source to the form, and then I customize which properties to display from the Data Source window. Afterwards, I drag the controls to the CustomerDetail form. Figure 5 shows the Data Source after I have customized it.

Figure 5 Customized Data Source
Figure 5 Customized Data Source

Now I must get the selected Customer record from the Web service's GetCustomer Web method. I use the CustomerID that was selected from the previous form and call the GetCustomer Web method. The Customer entity that is returned is then set to the Data source of the customerBindingSource control on the CustomerDetail form. I now test the application again in the emulator, and I see the screen shown in Figure 6.

Figure 6 Application Running in the Emulator
Figure 6 Application Running in the Emulator

Mobile Binding Options

It is very easy to implement automatic binding using the BindingSource control in a mobile device application. It works similarly to how the BindingSource control works in a Windows Forms application, with the notable exception being the lack of a BindingNavigator control. Since mobile apps don't have a BindingNavigator control, you have to write code for navigating from one record to the next if you want that functionality. I tend to avoid that type of feature since it is typically used when retrieving a large list of items. This can affect performance in a mobile application, as the mobile device usually has limited bandwidth. So the lack of a BindingNavigator is not a bad thing since it reminds you to be economical with bandwidth.

BindingSource takes the headache out of manually getting the values from the entity and stuffing them into the controls on the form. When saving a record, BindingSource also eliminates the hassle of retrieving the values from the form's controls and pulling them back into an entity. The downside of a BindingSource is that loading data can be slightly slower than when loading data manually.

I still recommend using the BindingSource control unless the performance difference is too great during your tests. I, however, have often found its performance to be adequate. The automatic binding you get when you use the BindingSource control is also very useful when you need to bind the same set of data to multiple controls.

Changing to manual binding is simple. All it requires is removing the BindingSource control, clearing the Data source settings of the controls on the form, and writing some code to manually load each value from the entity into the corresponding control, like this:

using (RemoteService.CustomerWebService svc =
  new RemoteService.CustomerWebService()) {
    RemoteService.Customer customer = svc.GetCustomer(_customerID);
    companyNameTextBox.Text = customer.CompanyName;
    customerIDLabel1.Text = customer.CustomerID;
    ...
  }

Note that when using a manual technique, you must also account for how to extract the data from the controls when the user wants to save the data. It's a simple procedure of grabbing the values from the controls and pulling them into an entity, but it is an extra step that you must handle when you want to allow users to save information. A common practice with manual binding is to store the Customer entity as a private field of the CustomerDetail form so you can readily access it when you need to pull the data out of the controls and back into the entity.

Fine-Tuning Your Mobile App

Since bandwidth may be limited and processors on mobile devices are slower than desktop computers, it is important that you only pass the minimum amount of data needed by the application. In my sample application, I retrieved a list of Customer entities, but only used two of the properties of each entity. This wasted a lot of bandwidth for the CustomerPicker screen since it had to go and get all of the data for all of the customers.

A more efficient approach would be to create a method that retrieves just the CustomerID and CompanyName for each customer. Considerations like these are important to remember when tuning your mobile application, as performance issues like these may not show up in a desktop application.

Final Thoughts

Before you dive into mobile development, it is important that you get yourself properly set up. You must make sure you have the right emulator manager version, the appropriate emulators, the correct SDKs, and the necessary version of ActiveSync or Windows Mobile Device Center. When I ran my application on Windows Vista, the emulator would not communicate with my Web service until I installed the Windows Mobile Device Center 6.1 and the Device Emulator Manager 2. These are important updates to the software that must be installed when running in Windows Vista, even when targeting Windows Mobile 5.0.

If you get an error when trying to get ActiveSync to see your emulator, make sure that ActiveSync is set to communicate through DMA. If you get an error when trying to access your Web service, make sure the Web service uses your server's name, as opposed to localhost. Localhost will work fine on your PC, but when the emulator tries to reconcile localhost, it will not know that you meant your computer's Web server. Another good tip is to make sure you connect and cradle your emulated device so it can communicate with the local PC.

If you need information about deploying your application to a real mobile device, check out my December 2006 installment of Data Points, "RSS Feeds on a Smartphone" (which is available at msdn.microsoft.com/msdnmag/issues/06/12/DataPoints).

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a Senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server. He can often be found speaking at industry conferences, such as VSLive, or blogging at codebetter.com/blogs/john.papa.


Advanced Basics: Reflection and Generics

Advanced Basics
Reflection and Generics
Ken Getz

Code download available at: AdvancedBasics2008_01.exe (181KB)
Contents
Beginning the Investigation
Examining Methods
Creating an Instance of a Type
Creating Generic Type Instances and Calling Methods
Working with Type Arguments
Generic Type Argument Constraints

K
nowing that I love to play with such things, a friend recently asked me to help write a simple tool that could export to Microsoft® Excel® a sortable list of all the members of a new assembly that she was required to document. The tool needed to provide information about the properties, methods, events, and interfaces, and list all the parameters for each of the members. Having played with reflection before, it seemed like fun.

As I began, though, I kept finding member names that were formatted oddly. For example, using my program to list out the contents of the mscorlib assembly, I found names like Dictionary`2 and ReadOnlyCollection`1.

It turns out that names like those exist for a reason. The Microsoft .NET Framework exposes information about generic types and methods differently than it does for those that are non-generic. This realization led me to investigate—and now document—reflection features as they relate to generics. While generics are new as of the .NET Framework 2.0, using reflection to investigate and fabricate or call generic types and methods isn't terribly difficult. My sample application (available from the MSDN® Magazine Web site) shows off as many of the features exposed at the intersection of generics and reflection as I could discuss.

If generics are completely new to you, see "Overview of Generics in the .NET Framework" at msdn2.microsoft.com/ms172193. Most importantly, review the terms described in the article because my goal is to show you how to extend existing reflection-based applications to handle generic types and methods.

Beginning the Investigation

Given an assembly, you can easily use reflection to investigate its members. You will need to use the Type.IsGenericType property to learn whether each type is generic in order to later create an instance of it. In my sample, the TestForGenericType procedure loads the contents of the mscorlib assembly and displays information about all the public generic types:

Dim asm As Assembly = Assembly.ReflectionOnlyLoad("mscorlib")
For Each typ As Type In asm.GetTypes()
    If typ.IsPublic AndAlso typ.IsGenericType Then
        AddToResults(typ.Name)
    End If
Next

In the sample, the AddToResults method and the DisplayResults method (not shown here) simply build output text and display it in an alert. So that you can investigate types that you created, the sample also includes a TestClasses.vb file that contains several sample classes, as shown in Figure 1.

Figure 1 Classes in TestClasses.vb (Close Figure 1)
Public Class GenericClass(Of T)
    ' This is not a generic method.
    Public Sub Swap(ByRef item1 As T, ByRef item2 As T)
        Dim temp As T = item1
        item2 = item1
        item1 = temp
    End Sub

    ' This is a generic method and a generic method definition.
    Public Sub Swap(Of U)(ByRef item1 As U, ByRef item2 As U)
        Dim temp As U = item1
        item2 = item1
        item1 = temp
    End Sub

    Public Function GenericMethod(Of M)(ByVal item1 As M) As String
        Return String.Format("You passed in: {0}", item1)
    End Function

End Class

Public Class BaseClass(Of T, U)
    Public Items As New Dictionary(Of T, U)

    Public Sub New()
        Debug.Print("In BaseClass, GetType(T) = " & GetType(T).Name)
        Debug.Print("In BaseClass, GetType(U) = " & GetType(U).Name)
    End Sub
End Class

TestForGenericType continues by testing these three classes:

AddToResults("GenericClass(Of Integer) is generic: {0}", _
  GetType(GenericClass(Of Integer)).IsGenericType)
AddToResults("NonGenericClass is generic: {0}", _
  GetType(NonGenericClass).IsGenericType)
AddToResults( _
  "BaseClass(Of Integer, String) is generic: {0}", _
  GetType(BaseClass(Of Integer, String)). _
  IsGenericType)

This code creates constructed versions of each of the generic types shown previously. (A constructed type is a generic type for which you've supplied specifics for each placeholder type.) GenericClass and BaseClass are generic types, whereas NonGenericClass is not. Note that when the .NET Framework returns the Name property of a generic type, it appends the number of generic parameters. That's why the Dictionary class appears as Dictionary`2.

Examining Methods

You can repeat the same sort of experimentation for methods. When it comes time to execute a method dynamically, you'll need to know if the method is generic so you can construct a version of it in which you've satisfied all its type placeholders. A method is considered generic if it includes two sets of type arguments: one that describes the method as a whole, and the other that supplies the method's parameters. In GenericClass, for example, you'll find two overloaded versions of the Swap method. The first one isn't generic—its parameter types are defined by the class's type placeholder. The second one is generic, however. In this case, the compiler infers the type of the placeholder from the type of the first parameter to the procedure. (If you make an attempt to call the procedure, passing in two different types, the compiler will complain.) The TestForGenericMethod procedure uses the MethodInfo.IsGenericMethod property to determine if a method is generic and calls the following code to test the property:

Dim typ As Type = GetType(GenericClass(Of Integer))

For Each method As MethodInfo In typ.GetMethods( _
  BindingFlags.Public Or BindingFlags.Instance _
  Or BindingFlags.DeclaredOnly)

  AddToResults(method.Name & " is generic: {0}", _
   method.IsGenericMethod)
Next

This code uses the BindingFlags enumeration in its call to the Type.GetMethods method, so that it only retrieves public instance methods that are declared directly in the class. The output looks like this:

Swap is generic: True
GenericMethod is generic: True
Swap is generic: False

Creating an Instance of a Type

If you want to create an instance of a type, you'll need to know if it's a generic type definition or a constructed generic type. A generic type definition has one or more of its type placeholders unsatisfied—in other words, a generic type definition isn't aware of its parameter types. A constructed generic type (as you've seen in the previous examples) has its type placeholders replaced with specific types. You can't create an instance of a non-constructed generic type—you must supply its types before you attempt to create the instance. Use the Type.IsGenericTypeDefinition property to determine the status of the type, as in the code from the TestForGenericTypeDefinition procedure shown in Figure 2.

Figure 2 Testing for Generic Type Definition (Close Figure 2)
Dim type1 As Type = GetType(Dictionary(Of ,))
Dim type2 As New Dictionary(Of String, DateTime)
Dim type3 As Type = type2.GetType()
Dim type4 As Type = type3.GetGenericTypeDefinition()

AddToResults("type1 is generic type definition: {0}", _
  type1.IsGenericTypeDefinition)
AddToResults("type3 is generic type definition: {0}", _
  type3.IsGenericTypeDefinition)
AddToResults("type4 is generic type definition: {0}", _
  type4.IsGenericTypeDefinition)
AddToResults()

AddToResults("type1 is type4: {0}", type1 Is type4)

You've already seen how to retrieve a Type object corresponding to a constructed generic type. You can also do the same with a generic type definition, as you saw in the first line of Figure 2 (make sure you include a comma separator between each of the required type placeholders):

Dim type1 As Type = GetType(Dictionary(Of ,))

Given a constructed type, you can use the Type.GetGenericTypeDefinition method to retrieve the generic type's definition. The following code creates a constructed instance of the generic Dictionary class, retrieves its type, and then retrieves the generic type definition of the constructed generic type:

Dim type2 As New Dictionary(Of String, DateTime)
Dim type3 As Type = type2.GetType()
Dim type4 As Type = type3.GetGenericTypeDefinition()

Finally, the procedure compares the references in type4 and type1; they should be the exact same reference (that is, the generic type definition of the Dictionary instance). The procedure's output looks like the following, indicating that type1 and type4 are the same generic type definition:

type1 is generic type definition: True
type3 is generic type definition: False
type4 is generic type definition: True

type1 is type4: True

Type3, a constructed generic type, isn't a generic type definition.

You can use a similar technique to determine if the type you're inspecting is a standard type, a generic type, or a generic type definition. The following illustrates the MethodInfo.IsGenericMethod and MethodInfo.IsGenericMethodDefinition properties:

Dim type1 As Type = GetType(GenericClass(Of ))

Dim method As MethodInfo = type1.GetMethod("GenericMethod")
If method.IsGenericMethodDefinition Then
    Dim miConstructed As MethodInfo = _
        method.MakeGenericMethod(GetType(Integer))
    AddToResults("Is generic method definition: {0}", _
        miConstructed.IsGenericMethodDefinition)
    AddToResults("Is generic method: {0}", _
        miConstructed.IsGenericMethod)
End If

Every generic method definition is also a generic method, but the converse is not true; every generic method is not a generic method definition. If you've supplied the type placeholder values for a method, it's a generic method but not a generic method definition. This sample also demonstrates the MethodInfo.MakeGenericMethod method, which allows you to take a generic method definition and supply types for its type placeholders, thereby constructing a generic method from the definition. The sample procedure displays the following output:

Is generic method definition: False
Is generic method: True

Again, this sample is operating on the generic type definition:

Public Function GenericMethod(Of M)(ByVal item1 As M) As String
    Return String.Format("You passed in: {0}", item1)
End Function

And by calling the MakeGenericMethod method, passing in the Integer type, the code then proceeds to create the following generic method at run time (item1 represents the type you've passed in from the calling code):

Public Function GenericMethod (ByVal item1 As Integer) As String
    Return String.Format("You passed in: {0}", item1)
End Function

Creating Generic Type Instances and Calling Methods

Assuming that one of your goals is to be able to create an instance of a generic type and perhaps to call a constructed generic method, you'll need a few more techniques. In order to create an instance of a type, there can't be generic type definitions or generic method definitions with unsatisfied type placeholders anywhere in the entire chain of dependencies for the type. Generic types for which all the type placeholders have been specified are called closed generic types (the same terminology applies to methods). You can only create an instance of a closed generic type, and you can only invoke a closed generic method.

But what if you're handed a generic type that has a complex inheritance hierarchy? How can you be sure there aren't any open types in the type's ancestry? To determine this, you can retrieve the Type.ContainsGenericParameters property, which indicates definitively if you're working with a closed or open generic type: if the property returns True, you'll know you have an open generic type, and you won't be able to create an instance of it.

The TestForOpenOrClosedGeneric procedure demonstrates this behavior using two classes in the TestClasses file, which are shown here:

Public Class BaseClass(Of T, U)
    Public Items As New Dictionary(Of T, U)
End Class

Public Class DerivedClass(Of V)
    Inherits BaseClass(Of Integer, V)

End Class

The TestForOpenOrClosedGeneric procedure calls the DisplayGenericTypeInfo procedure in Figure 3, displaying information about several different types.

Figure 3 DisplayGenericTypeInfo Procedure (Close Figure 3)
Private Sub DisplayGenericTypeInfo(ByVal t As Type)
    AddToResults("{0}", t)

    Dim indent As String = vbTab
    AddToResults("{0}Generic type definition: {1}", _
        indent, t.IsGenericTypeDefinition)

    AddToResults("{0}Generic type: {1} ", _
        indent, t.IsGenericType)

    AddToResults("{0}Unassigned generic parameters: {1}", _
        indent, t.ContainsGenericParameters)

    Dim args() As Type = t.GetGenericArguments()
    AddToResults("{0}List type arguments ({1}):", _
        indent, args.Length)
    For Each argType As Type In args
        AddToResults("{0}{0}{1}", indent, argType)
    Next

    AddToResults("{0}{1}", indent, _
        IIf(t.ContainsGenericParameters, "OPEN", "CLOSED"))
End Sub

Given a type, the DisplayGenericTypeInfo procedure first displays the type name, whether the type is a generic type definition, and whether it's a generic type. Next, it displays whether the type contains generic parameters (if it's a generic type definition, it contains generic parameters and can't be instantiated). The code in Figure 3 then calls the GetGenericArguments method to retrieve a list of all the generic arguments and displays each generic argument in the output. Finally, the code uses the ContainsGenericParameters property to indicate whether the type is open or closed. (Note that you can apply the same sort of logic to MethodInfo instances in order to determine if a particular MethodInfo represents an open or closed generic method.)

The TestForOpenOrClosedGeneric method includes the following code, which tests several of the sample methods, both as generic definitions and as constructed types. See if you can predict the output for each:

Dim baseType As Type = GetType(BaseClass(Of ,))
DisplayGenericTypeInfo(baseType)

Dim derivedType As Type = GetType(DerivedClass(Of ))
DisplayGenericTypeInfo(derivedType)
DisplayGenericTypeInfo(derivedType.BaseType)

Dim constructedType As Type = GetType(DerivedClass(Of String))
DisplayGenericTypeInfo(constructedType)

DisplayGenericTypeInfo(constructedType.BaseType)

The CreateGenericTypeGivenGenericTypeDefinition sample procedure shows how you can create a closed generic type given a generic type definition. You've already seen this technique applied to a method definition: just as you called the MethodInfo.MakeGenericMethod method to construct a generic method from its definition, you can call the Type.MakeGenericType method to construct a generic type from its definition. The sample code creates a constructed type at run time and compares it to a constructed type it created at design time, as you can see in Figure 4.

Figure 4 Comparing Constructed Types (Close Figure 4)
Dim type1 As Type = GetType(BaseClass(Of ,))
DisplayGenericTypeInfo(type1)

' Create an array of types to substitute for the type
' parameters.
Dim args() As Type = {GetType(String), GetType(DateTime)}

' Create a Type object representing the constructed generic
' type.
Dim constructed As Type = type1.MakeGenericType(args)
DisplayGenericTypeInfo(constructed)

Dim type2 As Type = GetType(BaseClass(Of String, DateTime))
AddToResults("Constructed types equal: {0}", _
   constructed Is type2)
AddToResults("Generic types equal: {0}", _
  type2.GetGenericTypeDefinition Is type1)

In order to call a generic method given its generic method definition, you must call the MethodInfo.MakeGenericMethod method, passing an array containing Type objects corresponding to each of the method's type placeholders. Given the new constructed method, you can call the MethodInfo.Invoke method, supplying a host object instance and an array of parameters for the method. Of course, the type of the parameters must correctly match the type of the placeholder type you specified when you constructed the generic method instance.

The CallGenericMethodGivenGenericMethodDefinition sample procedure in Figure 5 demonstrates this technique. This sample starts by retrieving a MethodInfo instance corresponding to the GenericMethod method in GenericClass. This method accepts a single type placeholder and simply returns a string containing the value you supply at run time:

Figure 5 CallGenericMethodGivenGenericMethodDefinition (Close Figure 5)
Dim type1 As Type = GetType(GenericClass(Of Integer))
Dim mi As MethodInfo = type1.GetMethod("GenericMethod")
DisplayGenericMethodInfo(mi)

Dim argTypes() As Type = {GetType(String)}
Dim miConstructed As MethodInfo = mi.MakeGenericMethod(argTypes)
DisplayGenericMethodInfo(miConstructed)

' Invoke the method, creating an instance of the
' generic class first.
Dim host As New GenericClass(Of Integer)

' Supply the method parameters, and then invoke the method:
Dim args() As Object = {"test value"}
AddToResults("{0}", miConstructed.Invoke(host, args))
AddToResults()

' Construct another version of the same
' generic method, this time taking an integer
' as the parameter:
argTypes(0) = GetType(Integer)
miConstructed = mi.MakeGenericMethod(argTypes)
DisplayGenericMethodInfo(miConstructed)

' Supply the method parameters, and then invoke the method:
args(0) = 13
AddToResults("{0}", miConstructed.Invoke(host, args))

Public Function GenericMethod(Of M)(ByVal item1 As M) As String
    Return String.Format("You passed in: {0}", item1)
End Function

The code does its work by constructing a specific generic class, supplying a type for the class's type placeholder:

Dim type1 As Type = GetType(GenericClass(Of Integer))
Dim mi As MethodInfo = type1.GetMethod("GenericMethod")
DisplayGenericMethodInfo(mi)

As you learned earlier, you won't be able to call a generic method if there are any unsatisfied type placeholders in its ancestry (that is, if it's not a closed generic method). To prove that point, run the procedure once as is and then modify it so that you specify a generic type definition for type1. The code will compile, but it will fail at run time:

Dim type1 As Type = GetType(GenericClass(Of ))

The sample calls the DisplayGenericMethodInfo procedure, which displays information about the generic method and about each of its generic arguments.

The code continues by creating an array of Type instances and then calls the MethodInfo.MakeGenericMethod method to create a constructed generic method, as you see here:

Dim argTypes() As Type = {GetType(String)}
Dim miConstructed As MethodInfo = mi.MakeGenericMethod(argTypes)
DisplayGenericMethodInfo(miConstructed)

Given the constructed generic method, the code creates an instance of GenericClass (you'll see exactly how to do this in the next demonstration), creates an array of parameters to pass to the method, and calls MethodInfo.Invoke in order to execute the constructed generic method:

Dim host As New GenericClass(Of Integer)

' Supply the method parameters, and then invoke the method:
Dim args() As Object = {"test value"}
AddToResults("{0}", miConstructed.Invoke(host, args))

To see this in action, you can create multiple, different constructed versions of the same generic method definition. The code will then create a new generic method definition, this time indicating that it will pass an integer (as opposed to a string). The code creates the constructed generic method and then invokes it, as before:

argTypes(0) = GetType(Integer)
miConstructed = mi.MakeGenericMethod(argTypes)
DisplayGenericMethodInfo(miConstructed)

' Supply the method parameters, and then invoke the method:
args(0) = 13
AddToResults("{0}", miConstructed.Invoke(host, args))

The CreateClosedGenericInstance sample method uses a similar technique, but creates a constructed generic type instead. In the previous example, the code created the constructed type at design time. In this example, it happens at run time.

Working with Type Arguments

Although you've already seen code that retrieved information about generic arguments, doing so involves some subtleties worth examining. The RetrieveTypeParameterInfo sample procedure works with several different generic types and generic type definitions, displaying generic argument information for each. The Type.GetGenericArguments method returns an array, and if the current type is a generic type definition, the array contains the type parameters for the generic type definition in the order in which they appear in the type definition. If the current type is a closed constructed type, the array contains the types that have been assigned to the generic type parameters. If the current type is a generic type definition, the array contains the type parameters. If the current type is an open constructed type (that is, not all the parameters have been satisfied and the ContainsGenericParameters property returns True), the array contains both types and type parameters. You can use the IsGenericParameter property to distinguish between the two.

The RetrieveTypeParameterInfo procedure contains within it the following code:

Dim type1 As Type = GetType(DerivedClass(Of String))
DisplayGenericParameterInfo(type1)

Dim type2 As Type = type1.GetGenericTypeDefinition
DisplayGenericParameterInfo(type2)

Dim type3 As Type = type2.BaseType
DisplayGenericParameterInfo(type3)

What this code displays is the parameter information for a constructed generic type, for the corresponding generic type definition, and for the base type of the generic type definition (the BaseClass sample class).

DisplayGenericParameterInfo does most of the work in this sample and includes the code in Figure 6. After displaying the type name, the code displays indications of whether the type is a generic type definition, whether it's a generic type, and whether it contains generic parameters. If it is a generic type, the code retrieves the array of generic arguments:

Figure 6 Displaying Generic Parameter Information (Close Figure 6)
AddToResults("{0}", t)

Dim indent1 As String = vbTab
Dim indent2 As String = vbTab & vbTab

AddToResults("{0}Generic type definition: {1}", _
    indent1, t.IsGenericTypeDefinition)

AddToResults("{0}Generic type: {1}", _
    indent1, t.IsGenericType)

AddToResults("{0}Unassigned generic parameters: {1} ", _
    indent1, t.ContainsGenericParameters)

If t.IsGenericType Then
    ' If a generic type, display the arguments:
    Dim args As Type() = t.GetGenericArguments()

    AddToResults("{0}List type arguments ({1}):", _
        indent1, args.Length)

    For Each argType As Type In args
        If argType.IsGenericParameter Then
            AddToResults("{0}{1} at position {2}", _
            indent2, argType, argType.GenericParameterPosition)
        Else
            AddToResults("{0}{1}", indent2, argType)
        End If
    Next argType
End If

Dim args As Type() = t.GetGenericArguments()

Then, for each Type instance in the array, the code determines if it has a generic parameter (or a Type, from a constructed type). If so, it displays the GenericParameterPosition property value; if not, it displays the type name. (Retrieving the GenericParameterPosition property for types that aren't generic parameters will trigger a run-time exception.)

Determining the Source of a Generic Parameter

If some code hands you a generic parameter, you may need to determine if it was for a generic type or a generic method. It might come from a type you are examining, from an enclosing type, or from a generic method. How can you determine the source? First, check the type's DeclaringMethod property. If the value isn't Nothing, it will be a MethodInfo instance, and you will know that the type parameter came from a generic method, and the MethodInfo's IsGenericMethodDefinition property will return True. If the source isn't a generic method, the DeclaringMethod property returns Nothing, and you can then retrieve the DeclaringType property. This type will always be a generic type definition. You can only retrieve the DeclaringMethod property if the IsGenericParameter property of the type returns True; otherwise, an exception will be raised.

Generic Type Argument Constraints

Finally, you may need to investigate generic type argument constraints. These constraints enforce rules on type arguments, ensuring that generic types adhere to the application requirements. You can add constraints that force a type parameter to inherit from a specific class, implement one or more specific interfaces, be a value type, be a reference type, and/or provide a default constructor. You might want to determine programmatically which constraints, if any, apply to a type argument, so that you can construct a type dynamically that meets the constraint requirements.

In order to examine constraints, you must take two sets of steps. In order to retrieve information about a single class or one or more interfaces that a type argument must inherit or implement, you need to call the Type.GetGenericParameterConstraints method. This method returns an array of Type objects, each of which indicates one class or interface.

To retrieve information about variance and other special constraint types, you must work with two separate bit masks. For variance constraints, you retrieve the type's GenericParameterAttributes property. Then you must apply the GenericParameterAttributes.VarianceMask bit mask to the returned value. Compare the result to the GenericParameterAttributes.Covariant or Contravariant enumerated value to determine if the type argument has a variance constraint.

To retrieve information about the special constraints, apply the GenericParameterAttributes.SpecialConstraintMask bit mask. This time, compare the results against the ReferenceTypeConstraint, NotNullableValueTypeConstraint, and the DefaultConstructorConstraint enumerated values to determine the status of each special constraint. The ExamineGenericParameterAttributes method does all this work for you, using the IsBitSet helper procedure to determine if a particular bit has been set:

Private Function IsBitSet( _
        ByVal value As Integer, ByVal bitValue As Integer) As Boolean
    Return (value And bitValue) = bitValue
End Function

The ExamineGenericParameterAttributes procedure includes the code in Figure 7, which follows the preceding steps.

Figure 7 Examine Generic Parameter Attributes (Close Figure 7)
' Select the variance flags.
Dim variance As GenericParameterAttributes =  _
     parameterAttributes And _
    GenericParameterAttributes.VarianceMask

Dim indent As String = vbTab
If variance = GenericParameterAttributes.None Then
    AddToResults("{0}No variance flag", indent)
Else
    If IsBitSet(variance, GenericParameterAttributes.Covariant) Then
        AddToResults("{0}Variance flag = Covariant", indent)
    Else
        AddToResults("{0}Variance flag = Contravariant", indent)
    End If
End If

' Select the constraint flags.
Dim constraints As GenericParameterAttributes = _
    parameterAttributes And _
    GenericParameterAttributes.SpecialConstraintMask

If constraints = GenericParameterAttributes.None Then
    AddToResults("{0}No special constraints", indent)
Else
    If IsBitSet(constraints, _
        GenericParameterAttributes.ReferenceTypeConstraint) Then
            AddToResults("{0}ReferenceTypeConstraint", indent)
    End If
    If IsBitSet(constraints, _
        GenericParameterAttributes.NotNullableValueTypeConstraint) Then
            AddToResults("{0}NotNullableValueTypeConstraint", indent)
    End If
    If IsBitSet(constraints, _
        GenericParameterAttributes.DefaultConstructorConstraint) Then
            AddToResults("{0}DefaultConstructorConstraint", indent)
    End If
End If

The sample DisplayConstraintInfo procedure includes the code in Figure 8, which displays parameter constraint information for the type it's handed.

Figure 8 Display Constraint Info (Close Figure 8)
Private Sub DisplayConstraintInfo(ByVal typ As Type)
    AddToResults("{0}", typ)

    Dim indent As String = vbTab

    Dim params() As Type = typ.GetGenericArguments()
    For Each paramType As Type In params
        ' Handle special constraints:
        ExamineGenericParameterAttributes(paramType)

        ' Loop through type constraints:
        Dim constraints As Type() = _
            paramType.GetGenericParameterConstraints()

        For Each constraintType As Type In constraints
            AddToResults("{0}{1}", indent, constraintType)
        Next constraintType
        AddToResults()
    Next
End Sub

As mentioned earlier, this procedure loops through all the parameters and, for each, first calls the ExamineGenericParameterAttributes procedure (handling variance and other special constraints) and then loops through all the types that resulted from calling the GetGenericParameterConstraints method.

The ExamineConstraints sample procedure is shown here:

DisplayConstraintInto(GetType(TestClass1(Of )))
DisplayConstraintInto(GetType(TestClass2(Of )))
DisplayConstraintInto(GetType(TestClass3(Of )))

It examines the behavior of the DisplayConstraintInfo procedure for three different classes, all of which have different argument constraints. The sample classes look like Figure 9.

Figure 9 Sample Classes (Close Figure 9)
Public Class TestClass1( _
        Of T As {IComparable, IDisposable, Class, New})

    ' T must implement IComparable and IDisposable.
    ' T must be a reference type.
    ' T must supply a default constructor.
End Class

Public Class TestClass2(Of T As {NonGenericClass})
    ' T must inherit from NonGenericClass.
End Class

Public Class TestClass3(Of T As {Structure})
    ' T must be a value type.
End Class

Your Turn

At this point, you've seen just about every method and property exposed by the .NET Framework at the intersection of reflection and generics. All you need to do now is download the sample application, follow through all the code and output, read the associated documentation topics, and you should be ready to tackle any problems related to reflection and generics.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a Senior Consultant with MCW Technologies and a courseware author for AppDev (www.appdev.com). He is coauthor of ASP .NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2002), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Cutting Edge: Customize Controls with AJAX Extenders

Cutting Edge
Customize Controls with AJAX Extenders
Dino Esposito

Contents
Getting Started
Extending a Control's Capabilities
Anatomy of an Extender Control
Input Extenders
Watermarks and Input
Confirmation and Modality
Enhanced Checkboxes
Restricted Input
Toward Better Web Pages

I
nput controls are essential in any application, but they're especially critical when your Web application is the face of your organization. These controls can be the primary measure by which users judge your application and even your credibility.

In both Windows® and Web programming, user interfaces are built by composing controls. But the array of controls available is not particularly rich. Input controls for Windows Forms are still based on Win32® controls, and Web controls are little more than wrappers around HTML <INPUT> tags. Clearly, better input controls are needed, especially for anyone who is writing ASP.NET AJAX applications, since these rely heavily on client-side user interaction.

In this two-part series, I'll focus on the input capabilities of ASP.NET 3.5, but most of the information I'll cover is also applicable to ASP.NET 2.0 with ASP.NET AJAX Extensions installed. I'll discuss how to enhance ASP.NET input controls using some of the JavaScript-powered behaviors available in the AJAX Control Toolkit, which I'll refer to here as ACT. This month I'll cover the basic set of ACT input extensions; next month I'll delve into more advanced features.

Getting Started

ACT is a shared-source library of Web controls that you can download from codeplex.com/AtlasControlToolkit. ACT extends existing ASP.NET server controls with predefined blocks of script code. ACT extenders can be applied to both built-in and custom ASP.NET controls.

Technically speaking, ACT components add an extra JavaScript layer on top of controls such as TextBox and Panel. Therefore, the use of AJAX is not a strict requirement. However, the implementation of ACT components relies largely on the JavaScript facilities in the Microsoft® AJAX Library as well as the ASP.NET AJAX Extensions server runtime.

ACT licensing regulations are covered at asp.net/ajax/ajaxcontroltoolkit. The ACT project page on CodePlex features two downloads: the ASP.NET 2.0 with AJAX Extensions and ASP.NET 3.5 (see codeplex.com/AtlasControlToolkit/Release/ProjectReleases.aspx).

To use ACT with ASP.NET applications, you need to reference the assembly in the project and then register ACT with any page that uses it. You can register ACT on a per-page basis using the @Register directive and specifying an arbitrary prefix, like so:

  <%@ Register Assembly="AjaxControlToolkit"
               Namespace="AjaxControlToolkit"
               TagPrefix="act" %>

Alternatively, you can register the library in the web.config file for all pages in the application. Here's the configuration script to use:

  <pages>
     <controls>
        <add tagPrefix="act"
             namespace="AjaxControlToolkit"
             assembly=" AjaxControlToolkit" />
     </controls>
  </pages>

ACT controls do not automatically appear in the Visual Studio toolbox, but adding a new tab is quite simple, as you can see in Figure 1.

Figure 1 ACT in Visual Studio 2008
Figure 1 ACT in Visual Studio 2008

Extending a Control's Capabilities

In an object-oriented framework like ASP.NET, each server control is a class that you can further customize by deriving new controls from it. For example, imagine you need a masked textbox control. You would start by creating a new MaskedTextBox class derived from the built-in TextBox class. In the derived class, you add the properties and client-side features required to deliver the kinds of behaviors needed for a masked input field. If, later on, you need a numeric textbox or an up/down input box, you can create another custom control class. But creating a new control for each additional feature is not the best idea. As Figure 2 shows, you'll end up with a proliferation of controls forming a flat hierarchy where each new control adds just a little behavior to the same base class.

Figure 2 Derived Controls versus Extenders
Figure 2 Derived Controls versus Extenders(Click the image for a smaller view)
Figure 2 Derived Controls versus Extenders
Figure 2 Derived Controls versus Extenders (Click the image for a larger view)

Fortunately, there is another way to augment a control: extending the behavior of a particular instance of an existing control. In this case, the additional behavior is added via an ASP.NET extender. Any number of additional behaviors can be associated with an instance of a given control to enhance its capabilities without creating new classes.

As an ASP.NET server control, an extender gets a reference to the extended control and modifies its behavior by injecting script code. The extender control may feature a few properties of its own so developers can further configure behavior as well. The association between the target ASP.NET control and its set of extenders is typically established declaratively in the page markup. The following code snippet shows how to add a numeric up/down interface to a textbox so that users can select numbers by just clicking an up/down button:

<asp:TextBox ID="Children" runat="server" />
<act:NumericUpDownExtender ID="UpDown1" runat="server"
     TargetControlID="Children"
     Width="100" />

As you can see, the page contains a regular TextBox control plus an instance of the NumericUpDownExtender control. The TargetControlID property on the extender sets the ID of the extendee control. The Width property is specific to the extender and defines the final width of the extendee control plus any additional UI elements contributed by the extender—in this case, a pair of up/down buttons (see Figure 3). The extender's properties can be set both declaratively and programmatically in the codebehind class.

Figure 3 NumericUpDown Extender in Action
Figure 3 NumericUpDown Extender in Action

Anatomy of an Extender Control

ACT contains classic server controls to serve particular user interface needs and extenders to add specific behaviors to existing controls. For example, in ACT there is a TabContainer control to set up a tab-based interface as well as a ReorderList control that provides a listbox with drag-and-drop support to move items around. Mostly, though, ACT consists of extenders.

Take a look at Figure 4 where you see that an extender inherits from the base class, ExtenderControl, defined in the AJAX extensions assembly. Internally, a custom extender control class overrides the GetScriptReferences method, which returns the list of script files to be injected into the client page to implement the behavior, and the GetScriptDescriptors method, which returns the list of mappings between client-side and server-side properties. Thus any value assigned on the client to a given property will be properly reflected on the server. Aside from this relatively standard code, an extender is just JavaScript code that uses the DOM to provide richer UI elements to the end user.

Figure 4 Add Focus Capabilities to Target Controls (Close Figure 4)
[
[TargetControlType(typeof(Control))]
[ClientScriptResource("IntroAjax.FocusBehavior"), "focusBehavior.js")]
public class FocusExtender : AjaxControlToolkit.ExtenderControlBase
{
    [ExtenderControlProperty]
    [RequiredProperty]
    public string HighlightCssClass
    {
        get { return GetPropertyValue("HighlightCssClass", ""); }
        set { SetPropertyValue("HighlightCssClass", value); }
    }

    [ExtenderControlProperty]
    public string NoHighlightCssClass
    {
        get { return GetPropertyValue("NoHighlightCssClass", ""); }
        set { SetPropertyValue("NoHighlightCssClass", value); }
    }
}

The sample extender featured in Figure 4 adds a focus behavior to virtually all ASP.NET controls; that is, to all controls that derive from Control. The focus feature is implemented in the linked focusbehavior.js file. This script file will likely be embedded in the assembly that contains the extender control and then downloaded to the client when requested. The focusbehavior.js file uses the programming model of the Microsoft AJAX Library (see my December 2007 column at msdn.microsoft.com/msdnmag/issues/07/12/CuttingEdge) and is centered around an initialization method that registers Document Object Model (DOM) handlers for focus and blur events. In the focus handler, the code sets the CSS class for the target element. In the blur handler, it resets the CSS class. For more details, check out the source code that accompanies this column. All things considered, the easiest way to create extenders is to use the facilities of the ACT. You run the installer in the toolkit download, set up the templates, then select "Add AJAX Control Extender" to obtain a scaffold for the extender and the behavior. More information is available at asp.net/AJAX/AjaxControlToolkit/Samples/Walkthrough/CreatingNewExtender.aspx.

Input Extenders

As I mentioned, ACT features several extenders that add capabilities to buttons, textboxes, panels, listboxes, and so forth. The components in ACT can be grouped in a few categories: panel, input, generic UI, animation. The full list of components with documentation and sample code can be found at asp.net/ajax/control-toolkit, and the complete list of input extenders in ACT is shown in Figure 5.

Figure 5 Input Extenders from ACT (Close Figure 5)

Input Extender | Target Controls | Primary Goal | Description
AutoComplete | TextBox | Improved user experience | Provides a list of suggestions as the user types in the edit field.
Calendar | TextBox | Restricted input | Provides client-side date-picking functionality with customizable date format.
ConfirmButton | IButtonControl | Preventing mistakes | Pops up a confirmation dialog box as the user clicks the button.
FilteredTextBox | TextBox | Restricted input | Lets users only enter text that matches a given set of characters.
HoverMenu | Control | Improved user experience | Displays the contents of an associated panel control when the mouse hovers next to a given control—like a tooltip.
ListSearch | ListControl | Improved user experience | Enables users to search for an item in a list by typing some of the characters.
MaskedEdit | TextBox | Restricted input | Lets users only enter text that fits into a given input mask.
ModalPopup | Control | Improved user experience | Implements a classic modal dialog box without using the browser's window.alert method. It basically displays the contents of a panel and prevents users from interacting with the underlying page.
MutuallyExclusiveCheckBox | CheckBox | Restricted input | Lets you define logical groups of checkboxes so that users can check only one in each group.
NumericUpDown | TextBox | Preventing mistakes | Displays up and down buttons to select the next or previous value.
Slider | TextBox | Preventing mistakes | Uses a slider-like user interface to select a numeric value in a range.
TextBoxWatermark | TextBox | Preventing mistakes | Adds watermark text that disappears when the user starts typing and reappears as the textbox becomes empty.
ToggleButton | CheckBox | Improved user experience | Enables users to display custom images to render the check buttons.
ValidatorCallout | BaseValidator | Improved user experience | Improves the user interface of validators by popping up a balloon-style callout for each error message.

Watermarks and Input

One of the most annoying elements on a Web page is an unlabeled textbox or a page in which the labels and the textboxes don't line up. This can be caused by the font size the user chose, or other settings, but the result is that the user can't be sure just what belongs in the textbox. A great solution is a text watermark—default text in the textbox that disappears when the user begins to type in the box (see Figure 6). The following code shows how to get the effect:

Figure 6 Some Input Extenders
Figure 6 Some Input Extenders(Click the image for a smaller view)
Figure 6 Some Input Extenders
Figure 6 Some Input Extenders (Click the image for a larger view)

<act:TextBoxWatermarkExtender runat="server" ID="TextBoxWatermark1"
     TargetControlID="TextBox1"
     WatermarkText="Type Last Name Here"
     WatermarkCssClass="watermarked" />

The extender injects script code that handles focus, blur, and keydown events. As the control gets focus, the watermark disappears. When the control loses focus, the watermark comes back if no text has been typed in. The watermark text is cleared when the page posts back. This is accomplished using a handler for the DOM's submit event. The watermark extender control allows you to specify the text and the CSS style to apply to it. You might want to use a special CSS style to visually differentiate regular input text from watermark text.

In HTML, there's one primary element for accepting data—the <input> element. You have to use it for numbers, strings, dates, currency values, and so forth. Often you need a numeric input to be restricted to a specific range of values. You can certainly enforce this business rule through a validation layer, but it would be better to do it right there in the input box.

The Slider and NumericUpDown extenders force users to enter only numbers that fall in a given range. The Slider extender hides its associated TextBox and replaces it with a graphical slider moving through discrete steps within a minimum and a maximum interval. The underlying TextBox can always be set programmatically to any value you wish. Note, though, that the assigned value must be compatible with the numeric range set via the slider; otherwise, the slider will silently fix it. Let's consider the following example:

<act:SliderExtender runat="server" ID="SliderExtender1"
     TargetControlID="Income"
     Minimum="0"
     Maximum="200000"
     Steps="41"
     BoundControlID="IncomeAsText" />

The slider applies to a TextBox control named Income and ensures it is visually assigned a value in the range 0-200000. The Steps property indicates how many discrete steps should be provided. The preceding setting lets the slider jump by 5000 every time. If you programmatically set the slider on the server, the value is first converted to a number and then mapped to the nearest step value. If you pass in a string, the slider will ignore it and default to the initial value of the range. If you set the value of the underlying textbox on the client via JavaScript, then the value is correctly recognized over the next postback, but it is not immediately reflected by the user interface. To programmatically change the value in the slider from within the client, use the following code:

$find("SliderExtender1").set_Value(145678);

Why should you opt for $find instead of $get? The $get function is shorthand for document.getElementById. As such, it can only look for DOM elements. The $find function stands for Sys.Application.findComponent and applies to any component of the Microsoft AJAX Library that has been programmatically created. An extender is added to the page using the following script:

Sys.Application.add_init(
  function() {
      $create(AjaxControlToolkit.SliderBehavior,
           {"BoundControlID":"IncomeAsText",
            "Maximum":200000,
            "Steps":41,
            "id":"SliderExtender1"},
           null,
           null,
           $get("Income"));
  }
);

Being a Microsoft AJAX Library component, it can only be found using the $find function.

The BoundControlID property refers to a DOM element that dynamically displays the current slider value. In most cases, you'll use a <span> or a Label for this purpose. Internally, the slider distinguishes between INPUT and any other HTML elements. It sets the value property in the former case; otherwise, it uses the innerHTML property of the matching DOM element.

The slider script hides the underlying TextBox, so the element may be flashing during the page loading. You can hide it declaratively using CSS styles. Finally, be aware that regular textboxes are displayed inline, whereas slider boxes are positioned blocks.

The NumericUpDown extender selects the next/previous value in a bound list of possible values. Despite what the name implies, NumericUpDown is not just a tool for specifying numeric quantities. The RefValues property gives you the ability to list values to move through:

<act:NumericUpDownExtender ID="UpDown1" runat="server"
     Width="120"
     RefValues="None;1;2;3;4;5;More than 5;"
     TargetControlID="Children" />

NumericUpDown has the ability to retrieve reference values from a remote service. The service will be invoked asynchronously each time the user clicks the up or down buttons. It's important to remember that the service must be script-callable and include methods with the following signature:

public int YourMethod(int current, string tag)

You configure the extender to use the remote service through the ServiceDownPath, ServiceDownMethod, ServiceUpPath, and ServiceUpMethod properties. Finally, the up and down buttons don't have to be autogenerated via script. They can be buttons already defined in the page and referenced through the TargetButtonDownID and TargetButtonUpID properties.

Confirmation and Modality

Input forms that gather data to run critical server procedures are better started after the user has had an opportunity to confirm his intention. For example, it's wise to implement a confirmation step before permanently deleting some data on the server. A client-side confirmation step can only be implemented via JavaScript, by popping up something like a modal dialog box. The simplest is via the ConfirmButton extender:

<act:ConfirmButtonExtender ID="ConfirmButtonExtender1" runat="server"
     TargetControlID="Button1"
     ConfirmText="Are you sure you want to
                 "send this data?&#10;Really sure?" />

The extender hooks up the click event of the bound button control (not just Button, but also LinkButton and ImageButton) and pops up an alert message box. If you cancel the message, the postback won't occur. The browser's window.alert method is used to show the message. Clearly, you can code the same feature manually without the extender, but the extender is simpler and more reusable.

If you want a richer, more customized user interface, then you must look elsewhere. The ModalPopup extender is an alternative that is worth exploring. ModalPopup adds modality to a piece of markup—typically, a panel. Bound to a button control, it pops up the specified DIV tag and disables the underlying page. The element in the topmost DIV tag can't be dismissed except through the user interface it provides. The action of clicking on anything other than the element in the topmost DIV is lost and never reaches the intended target. ModalPopup performs a smart trick by adding an invisible DIV to cover the entire browser window. This layer swallows any user action and stops it from reaching underlying controls. With clever CSS coding, you can add some nice effects such as graying out anything underneath the topmost DIV (see Figure 7).

Figure 7 ModalPopup Extender
Figure 7 ModalPopup Extender(Click the image for a smaller view)
Figure 7 ModalPopup Extender
Figure 7 ModalPopup Extender (Click the image for a larger view)

In the following code for the ModalPopup extender, the PopupControlID property is referring directly to the markup element that is to be popped up:

<act:ModalPopupExtender ID="ModalPopupExtender1" runat="server"
     TargetControlID="LinkButton1"
     PopupControlID="Panel1"
     BackgroundCssClass="modalBackground"
     OkControlID="Button1"
     OnOkScript="yes()"
     OnCancelScript="no()"
     CancelControlID="Button2" />

In the majority of cases, this element tends to be an ASP.NET panel control or a plain DIV element. The BackgroundCssClass property indicates the CSS style to apply to the underlying page for the duration of the modal dialog. In order to get the grayed-out effect you see in Figure 7, you should use the following attributes:

.modalBackground
{
    background-color:Gray;
    filter:alpha(opacity=70);
    opacity:0.7;
}

Nearly all modal dialog boxes have a pair of OK and Cancel buttons. These buttons can be link, push, or image buttons and can be labeled with any text. You use the OkControlID and CancelControlID properties to tell the extender which controls in the panel represent the OK and Cancel buttons. Likewise, you use OnOkScript and OnCancelScript properties to indicate JavaScript functions to be run when the user clicks to confirm or dismiss the modal dialog box.

Enhanced Checkboxes

Another extender that can help users to enter correct data and save the system some extra client validation steps is MutuallyExclusiveCheckBox. It lets you define logical groups of checkboxes such that users can check only one in each group:

<asp:CheckBox runat="server" ID="beginner" Text="Beginner" />
<asp:CheckBox runat="server" ID="intermed" Text="Intermediate" />
<act:MutuallyExclusiveCheckBoxExtender runat="server" ID="Mutual1"
     TargetControlID="beginner"
     Key="Expertise" />
<act:MutuallyExclusiveCheckBoxExtender runat="server" ID="Mutual2"
     TargetControlID="intermediate"
     Key="Expertise" />

The preceding extenders define a group of two checkboxes named Expertise that are mutually exclusive. Normally, you would use checkboxes to allow multiple choices and use radio buttons to force the user to choose just one option. So why not use a RadioButtonList control here? Well, as you may have noticed, a RadioButtonList control can have all options unselected initially, but once a selection has been made there's no way to return to a fully unselected state. In other words, you can unselect an item only by selecting another one, and one is always selected. A group of checkboxes using MutuallyExclusiveCheckbox extenders does not have this limitation.

Checkboxes are usually displayed using bitmaps embedded in the browser. Changing the bitmap programmatically is not easy because there's no exposed public property. The ToggleButton extender hides the bound checkbox element and replaces it with a piece of dynamically generated markup that emulates the layout of a checkbox except that it uses a user-defined pair of bitmaps for the checked and unchecked states:

act:ToggleButtonExtender ID="ToggleButtonExtender1" runat="server"
    TargetControlID="CheckBox1"
    ImageHeight="19"
    ImageWidth="19"
    UncheckedImageUrl="~/images/DontLike.gif"
    CheckedImageUrl="~/images/Like.gif" />

The ToggleButton extender cannot be applied to a CheckBoxList control; it only works with individual checkboxes. It's nice to have the ability to change the look of the standard checkbox.

Restricted Input

Preventing user mistakes involves controlling the input by filtering out undesired characters, invalid expressions, or data of the wrong type. Using the standard TextBox control, you would enforce proper input by validating it on the server where you could convert the string to any other data type. With AJAX, input data types are more controlled. For example, by using the calendar extender you make it virtually impossible for users to type anything other than a date. ASP.NET comes with a server Calendar control, but a calendar extender is really different because it builds its user interface entirely on the client, works entirely on the client, and generates no postbacks at all as the user navigates to find month and day. Furthermore, if a given browser doesn't support JavaScript, the old textbox is displayed:

<act:CalendarExtender ID="CalendarExtender1" runat="server"
     TargetControlID="Birth"
     Format="dd/MM/yyyy" />

The preceding code snippet is sufficient to display a popup calendar, as the associated textbox receives the focus. As an alternative, you can display the popup when the user clicks a page button. The ID of the button is set through the PopupButtonID property. The Format property indicates the format of the date as it will be written to the textbox when the user dismisses the calendar popup (see Figure 8).

Figure 8 Calendar Extender
Figure 8 Calendar Extender

The FilteredTextBox extender prevents a user from entering invalid characters into a textbox. It basically adds JavaScript that hooks up the keyboard activity and filters out undesired keystrokes. You can configure the extender to refuse certain characters or to ensure that only specified characters are accepted. You use the FilterMode property for this setting. It determines the type of filter to apply—only numbers, only lowercase or uppercase, and a custom set of characters. It should be noted that a TextBox with a watermark can't be filtered to accept numeric values only. The filter layer, in fact, will automatically clear any watermark text you set.

Toward Better Web Pages

Extenders are a powerful and smart way to add extra behaviors to ASP.NET controls. ACT comes with more than 30 extenders to enrich built-in controls. In ACT, you find extenders to add animation, drag-and-drop functionality, autocompletion, and, more importantly, enhance input capabilities. ACT can be downloaded with or without the original source code. Additionally, most complex extenders have cross-dependencies. For example, the Calendar extender requires the Popup extender.

In this column, I've reviewed the simplest and most commonly used extenders that can make your pages more effective. In the next column, I'll go on with more advanced functionality such as masked editing and autocompletion.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Introducing ASP.NET AJAX (Microsoft Press, 2007). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can reach him at cutting@microsoft.com or join his blog at weblogs.asp.net/despos.


Wicked Code: Drag and Drop with ASP.NET AJAX

Wicked Code
Drag and Drop with ASP.NET AJAX
Jeff Prosise

Code download available at: WickedCode2008_01.exe (296KB)
Contents
Drag-Drop in Action
DragDropManager
Building Drag Sources
Building Drop Targets
Creating Drag Sources and Drop Targets
Beyond Drag and Drop

A
JAX has revolutionized Web user interfaces, and ASP.NET AJAX has made AJAX available to the Visual Studio® users. It comes in three separate downloads: ASP.NET AJAX Extensions (asp.net/ajax/downloads), which provides the core, fully tested set of AJAX functionality; ASP.NET AJAX Futures (asp.net/downloads/futures), which contains experimental features on which the product group wants feedback; and the ASP.NET AJAX Control Toolkit (asp.net/ajax/ajaxcontroltoolkit/samples), which provides a grab bag of AJAX controls as well as an SDK for building controls of your own.

Of the three, the Futures release has garnered the least attention from the developer community. That's unfortunate because, more than providing a glimpse into what future versions of ASP.NET AJAX might look like, the Futures Community Technology Preview (CTP) is chock full of features that can be used to build cutting-edge Web apps today. A case in point is drag-and-drop.

Hidden away inside the Futures PreviewDragDrop.js file lies support for rich, browser-based drag-and-drop user interfaces. The model it uses is patterned after the old OLE drag-drop model, in which drag sources implement the IDragSource interface, drop targets implement the IDropTarget interface, and the system provides a drag-drop manager to connect drag sources to drop targets. The Futures drag-drop manager is an instance of a JavaScript class named Sys.Preview.UI._DragDropManager, which is automatically instantiated and made available through a global variable named Sys.Preview.UI.DragDropManager.

For months now, I've been meaning to write a sample showing how to use PreviewDragDrop.js to implement real drag-drop, featuring custom drag sourcing and custom drop targeting. I finally got around to it, and the results are pretty cool. I learned quite a lot about DragDropManager in the process, including how to enhance it by adding support for custom drag visuals. Once you're familiar with the model (and comfortable with the concept of deriving classes and implementing interfaces in JavaScript), DragDropManager opens up a whole new world of possibilities for Web UIs.

Drag-Drop in Action

Before I dive into the code, take a moment to download the sample project that accompanies this article and give it a quick run-through. Open it using the Visual Studio Open Web Site command and then view the home page, DragDropDemo.aspx, in your browser. To run this sample, you must have ASP.NET AJAX Extensions installed. You don't have to download and install the Futures CTP because it's already in the site's Bin folder.

You'll see five color swatches at the top of the page and an empty box labeled "Drop It Here" at the bottom (see Figure 1). The color swatches are drag sources, and the empty box is the drop target. Grab one of the color swatches with your mouse and drag it down to the box. Observe how the box changes color from white to light gray when the cursor enters it, a feature known as drop-target highlighting. Now drop the color swatch over the box. The box changes color to match the swatch.

Figure 1 DragDropDemo.aspx in Action
Figure 1 DragDropDemo.aspx in Action(Click the image for a smaller view)
Figure 1 DragDropDemo.aspx in Action
Figure 1 DragDropDemo.aspx in Action (Click the image for a larger view)

Grab another color swatch and move it around the screen. Notice that the swatch can only be dropped over the drop target, and that the cursor indicates whether a drop would be accepted if it occurred right now. Also notice that as you drag a color swatch, a partially transparent rendition of the swatch—a "drag visual"—follows the cursor so you can see what you're dragging. As you'll see shortly, you write the code to create the drag visual, but ASP.NET AJAX handles everything else, including moving the drag visual with the cursor and updating the cursor with each mouse movement.

The custom logic that drives this page is located in a script file named ColorDragDrop.js in the Web site's Scripts folder. ColorDragDrop.js is a custom script file that's loaded by ScriptManager. In fact, open DragDropDemo.aspx in source view and you'll see that ScriptManager loads three script files: PreviewScript.js, which contains a key ASP.NET AJAX base class used in this demo; PreviewDragDrop.js, which contains the drag-drop support used by ColorDragDrop.js; and ColorDragDrop.js itself.

DragDropManager

The first step in building rich drag-drop user interfaces with ASP.NET AJAX is getting acquainted with the Sys.Preview.UI._DragDropManager class (hereafter referred to as DragDropManager). Under the hood, it delegates most operations to an instance of Sys.Preview.UI.IEDragDropManager or Sys.Preview.UI.GenericDragDropManager, depending on the host browser type. Its public interface consists of about a dozen methods, three of which play a vital role in drag-drop implementations:

  • startDragDrop—initiates a drag-drop operation
  • registerDropTarget—registers an object as a drop target
  • unregisterDropTarget—unregisters a drop target

DragDropManager's primary job, once its startDragDrop method has been called, is to monitor ongoing mouse events, fire pertinent notifications to the drag source and the drop target (if any) beneath the cursor, and shape the cursor to provide a visual indication of what would happen if a drop occurred at the current location. To do this, it registers its own handlers for key events such as mousemove and mouseup. From these event handlers, it calls IDragSource methods on the source being dragged and IDropTarget methods on any drop targets below.

When a cursor carrying a payload moves within the boundaries of a drop target, for example, DragDropManager calls the drag source's get_dragDataType, get_dragMode, and getDragData methods to get information about the payload. Then it passes the results to the drop target's canDrop method to determine whether the drag source and drop target are compatible. If a drop occurs, DragDropManager notifies the drop target by calling its drop method, once more passing in data obtained from the drag source.

DragDropManager can determine whether the cursor is over a drop target because it maintains an internal array of references to objects registered as drop targets. Objects are added to the array by calling DragDropManager's registerDropTarget method and removed by calling unregisterDropTarget. It's important to unregister the drop targets you register so DragDropManager can unregister the associated event handlers it registers on behalf of those drop targets. Failure to do so can result in memory leaks.

DragDropManager is the glue that binds drag sources and drop targets together. Without it, you'd be writing a ton of code to make drag-and-drop user interfaces work—especially if they had to work in a variety of browsers.

Building Drag Sources

Before you can begin dragging, you must have something to drag—that is, a drag source. To ASP.NET AJAX, a drag source is simply an object that implements the IDragSource interface defined in PreviewDragDrop.js. Figure 2 lists the methods that belong to that interface.

Figure 2 IDragSource Methods (Close Figure 2)

Method | Description
get_dragDataType | Returns the type of data being dragged.
getDragData | Returns the data being dragged.
get_dragMode | Returns the drag mode (move or copy).
onDragStart | Called when dragging of the drag source begins.
onDrag | Called repeatedly as the drag source is dragged.
onDragEnd | Called when dragging ends.

It may feel strange to implement interfaces in JavaScript given that JavaScript doesn't explicitly support interfaces. Nor, for that matter, does it explicitly support classes, inheritance, namespaces, and other tenets of object-oriented programming (OOP). And yet the client half of ASP.NET AJAX, the Microsoft® AJAX Library, is a class library implemented entirely in JavaScript. It fakes OOP using popular JavaScript programming techniques, which you can use to create classes of your own and even to implement interfaces in those classes and derive one class from another.

ColorDragDrop.js contains a pair of JavaScript classes named ColorDragSourceBehavior and ColorDropTargetBehavior. The former implements IDragSource, and both belong to a namespace named Custom.UI.

To create a JavaScript class in the style of the Microsoft AJAX Library, you begin by defining a function whose name is the same as the class name. This function serves as the class constructor:

Custom.UI.ColorDragSourceBehavior = function(element, color)
{
    Custom.UI.ColorDragSourceBehavior.initializeBase(this, [element]);
    this._mouseDownHandler = Function.createDelegate(this,
        this.mouseDownHandler);
    this._color = color;
    this._visual = null;
}

You then use the JavaScript prototype property to define methods to be included in every instance of the class:

Custom.UI.ColorDragSourceBehavior.prototype =
{
    // Methods here
}

JavaScript doesn't support strong typing or type reflection, so the Microsoft AJAX Library does it for you. Methods of the Type class (found in MicrosoftAjax.js, which is part of the ASP.NET AJAX core) such as registerClass and registerNamespace allow you to register the classes you build and assign them to namespaces. Later, you can use methods such as getTypeName, which is added to JavaScript's built-in Object type by the Microsoft AJAX Library, to reflect upon the types you registered.

Another essential piece of infrastructure provided by the Type class is a mechanism for calling a base class method from the corresponding method in a derived class. From a class constructor, you call initializeBase to call the base class's constructor, and from a method override in a derived class, you call callBaseMethod to invoke the equivalent method in the base class.

The ColorDragSourceBehavior class shown in Figure 3 uses all these tools to encapsulate the logic needed to convert an ordinary DOM element into a drag source in one easy-to-use class. From the call to registerClass, you can see that ColorDragSourceBehavior derives from the IDragSource interface as well as from an ASP.NET AJAX base class named Sys.UI.Behavior, which is defined in MicrosoftAjax.js. The former allows ColorDragSourceBehavior to function as a drag source; the latter enables instances of ColorDragSourceBehavior to be attached to DOM elements to modify their behavior.

Figure 3 ColorDragSourceBehavior (Close Figure 3)
Custom.UI.ColorDragSourceBehavior = function(element, color)
{
    Custom.UI.ColorDragSourceBehavior.initializeBase(this, [element]);
    this._mouseDownHandler = Function.createDelegate(this,
        this.mouseDownHandler);
    this._color = color;
    this._visual = null;
}

Custom.UI.ColorDragSourceBehavior.prototype =
{
    // IDragSource methods
    get_dragDataType: function()
    {
        return 'DragDropColor';
    },

    getDragData: function(context)
    {
        return this._color;
    },

    get_dragMode: function()
    {
        return Sys.Preview.UI.DragMode.Copy;
    },

    onDragStart: function() {},

    onDrag: function() {},

    onDragEnd: function(canceled)
    {
        if (this._visual)
            this.get_element().parentNode.removeChild(this._visual);
    },

    // Other methods
    initialize: function()
    {
        Custom.UI.ColorDragSourceBehavior.callBaseMethod(this,
            'initialize');
        $addHandler(this.get_element(), 'mousedown',
            this._mouseDownHandler)
    },

    mouseDownHandler: function(ev)
    {
        window._event = ev; // Needed internally by _DragDropManager

        this._visual = this.get_element().cloneNode(true);
        this._visual.style.opacity = '0.4';
        this._visual.style.filter =
          'progid:DXImageTransform.Microsoft.BasicImage(opacity=0.4)';
        this._visual.style.zIndex = 99999;
        this.get_element().parentNode.appendChild(this._visual);
        var location =
            Sys.UI.DomElement.getLocation(this.get_element());
        Sys.UI.DomElement.setLocation(this._visual, location.x,
            location.y);

        Sys.Preview.UI.DragDropManager.startDragDrop(this,
            this._visual, null);
    },

    dispose: function()
    {
        if (this._mouseDownHandler)
            $removeHandler(this.get_element(), 'mousedown',
                this._mouseDownHandler);
        this._mouseDownHandler = null;
        Custom.UI.ColorDragSourceBehavior.callBaseMethod(this,
            'dispose');
    }
}

Custom.UI.ColorDragSourceBehavior.registerClass
    ('Custom.UI.ColorDragSourceBehavior', Sys.UI.Behavior,
    Sys.Preview.UI.IDragSource);

Most of the methods that ColorDragSourceBehavior implements—get_dragDataType, getDragData, and so on—are members of the IDragSource interface. But ColorDragSourceBehavior implements a few methods that don't come from IDragSource. The initialize method, for example, registers a handler for mousedown events so the drag source can call DragDropManager.startDragDrop to begin a drag-drop operation. The dispose method deregisters the event handler to prevent memory leaks.

The IDragSource methods themselves are rather straightforward. get_dragDataType returns the string "DragDropColor" to identify the type of data that ColorDragSourceBehavior offers to drop targets—in this case, color data. getDragData returns a string defining the color represented by the drag source. get_dragMode returns Sys.Preview.UI.DragMode.Copy, which indicates that if a drop occurred right now, the drag source would be copied rather than moved. (The alternative is Sys.Preview.UI.DragMode.Move, which is defined along with Sys.Preview.UI.DragMode.Copy in PreviewDragDrop.js.)

Another item to note in ColorDragSourceBehavior is how it handles drag visuals. The mousedown event handler initiates a drag-drop operation by calling startDragDrop on DragDropManager. Before it calls startDragDrop, however, the handler calls cloneNode to clone the DOM element represented by the drag source. Then it sets the cloned node's opacity to 40 percent, inserts the node into the browser DOM, and passes a reference to the new node to DragDropManager as a parameter to startDragDrop. DragDropManager now assumes responsibility for the drag visual, animating its movement so that it follows the cursor.

Building Drop Targets

The ColorDropTargetBehavior class in ColorDragDrop.js (included in the code download) encapsulates the logic that converts DOM elements into drop targets. It derives from Sys.UI.Behavior, so it can be attached to DOM elements, and it implements the IDropTarget interface, so it can function as a drop target. Figure 4 lists the members of IDropTarget.

Figure 4 IDropTarget Methods (Close Figure 4)

Method | Description
get_dropTargetElement | Returns a reference to the DOM element that is the drop target.
canDrop | Returns true or false indicating whether a drop would be accepted.
drop | Called when a drop occurs over the drop target.
onDragEnterTarget | Called when a cursor carrying a payload enters the drop target.
onDragInTarget | Called when a cursor carrying a payload moves within the drop target.
onDragLeaveTarget | Called when a cursor carrying a payload leaves the drop target.

ColorDropTargetBehavior's onDragEnterTarget and onDragLeaveTarget methods perform the drop-target highlighting demonstrated earlier. When a cursor carrying a payload enters the drop target, DragDropManager calls onDragEnterTarget. If the payload is of the correct type (DragDropColor), onDragEnterTarget saves the target's current background color in a field and changes the background color to light gray. If the cursor leaves the drop target still carrying a payload, DragDropManager calls the drop target's onDragLeaveTarget method. ColorDropTargetBehavior's implementation of that method restores the drop target's original background color.

Each time a cursor carrying a payload moves over the drop target, DragDropManager calls the drop target's canDrop method to determine whether the target would accept a drop. ColorDropTargetBehavior.canDrop checks the payload type and returns true if it's DragDropColor and false if it's not. This process allows the DragDropManager class to provide the necessary visual feedback to the user via the cursor.

If a drop occurs, the drop target's drop method is called. The ColorDropTargetBehavior.drop method extracts the color from the drag source and sets its own background color accordingly (actually, the color of the DOM element that serves as the drop target).

The dragMode parameter that is passed to the drop method specifies the drag mode, which is either move or copy. Ultimately, it is up to the drop target to manipulate the browser's DOM to honor the drag mode. DragDropDemo.aspx only supports copy mode; therefore the drop target "copies" the drag source by setting its own background color to the color contained in the drag source. If move mode were supported also, the drop target would need to check the drag mode following the completion of a successful drop and would then have to delete (or reposition) the DOM element associated with the drag source if dragMode were set to Sys.Preview.UI.DragMode.Move.

Creating Drag Sources and Drop Targets

Once ColorDragSourceBehavior and ColorDropTargetBehavior are implemented, building pages containing drag sources and drop targets is easy. Figure 5 shows the relevant markup and code from DragDropDemo.aspx.

Figure 5 Excerpt from DragDropDemo.aspx (Close Figure 5)
<asp:ScriptManager ID="ScriptManager1" runat="server">
    <Scripts>
        <asp:ScriptReference Name="PreviewScript.js"
            Assembly="Microsoft.Web.Preview" />
        <asp:ScriptReference Name="PreviewDragDrop.js"
            Assembly="Microsoft.Web.Preview" />
        <asp:ScriptReference Path="~/Scripts/ColorDragDrop.js" />
    </Scripts>
</asp:ScriptManager>

<fieldset>
    <legend>Pick a Color</legend>
    <table cellpadding="8">
        <tr>
            <!-- Drag sources -->
            <td><div id="RedDragSource"
    style="width: 32px; height: 32px; background-color: red" /></td>
            <td><div id="YellowDragSource"
    style="width: 32px; height: 32px; background-color: yellow" /></td>
            <td><div id="GreenDragSource"
    style="width: 32px; height: 32px; background-color: green" /></td>
            <td><div id="MagentaDragSource"
    style="width: 32px; height: 32px; background-color: magenta" /></td>
            <td><div id="BlueDragSource"
    style="width: 32px; height: 32px; background-color: blue" /></td>
        </tr>
    </table>
</fieldset>

<div style="width: 100%; height: 128px">&nbsp;</div>

<fieldset>
    <legend>Drop It Here</legend>
    <!-- Drop target -->
    <div id="DropTarget" style="width: 100%; height: 196px">
        &nbsp;
    </div>
</fieldset>

<script type="text/javascript">
function pageLoad()
{
    //
    // Make the "DragSource" DIVs drag sources.
    //
    var source1 = new Custom.UI.ColorDragSourceBehavior
        ($get('RedDragSource'), 'red');
    var source2 = new Custom.UI.ColorDragSourceBehavior
        ($get('YellowDragSource'), 'yellow');
    var source3 = new Custom.UI.ColorDragSourceBehavior
        ($get('GreenDragSource'), 'green');
    var source4 = new Custom.UI.ColorDragSourceBehavior
        ($get('MagentaDragSource'), 'magenta');
    var source5 = new Custom.UI.ColorDragSourceBehavior
        ($get('BlueDragSource'), 'blue');

    source1.initialize();
    source2.initialize();
    source3.initialize();
    source4.initialize();
    source5.initialize();

    //
    // Make the "DropTarget" DIV a drop target.
    //
    var target = new Custom.UI.ColorDropTargetBehavior
        ($get('DropTarget'));
    target.initialize();
}
</script>

The color swatches that serve as drag sources are nothing more than DIVs with their background colors set to red, yellow, green, and so on. The JavaScript pageLoad function converts the DIVs into drag sources by instantiating five ColorDragSourceBehavior objects and assigning each to one of the DIVs. The calls to the drag source's initialize methods provide each drag source the opportunity to register its mousedown event handler.

The drop target, too, is a DIV. One statement instantiates a ColorDropTargetBehavior object and associates it with the DIV to convert the DIV into a drop target. A second statement calls the ColorDropTargetBehavior's initialize method so it can register as a drop target with DragDropManager.

ColorDragSourceBehavior and ColorDropTargetBehavior are specifically built to enable colors to be dragged and dropped. You can build classes like them to support other drag-drop scenarios. For example, imagine you were building a photo-sharing site and wanted to allow users to organize photos by dragging them between folders. You could use DIVs to represent the folders and write a PhotoDropTargetBehavior class to convert the DIVs into drop targets. Correspondingly, you could build a PhotoDragSourceBehavior class to turn DOM image elements into draggable photos. The classes you build tend to be very application-specific, but they all follow the same pattern exemplified by the drag-source and drop-target classes presented in this column.

Beyond Drag and Drop

ASP.NET AJAX is a wonderful tool for adding AJAX magic to Web pages, and the ASP.NET AJAX Futures release adds useful enhancements to the core platform. Support for drag-and-drop user interfaces is one example; other examples include animations, client-side data binding, and other handy DOM abstractions. Regardless of whether these features make it into core someday, they're available for use today. Consider using them in your next ASP.NET AJAX application to make a good app even better.

For another cool example of how to add fancy animations to Web pages by using the classes in the Futures PreviewGlitz.js file, see my blog post titled "Creating Sophisticated Animations with the Microsoft AJAX Library" at wintellect.com/cs/blogs/jprosise/archive/2007/04/04/Creating-Sophisticated-Animations-with-the-Microsoft-AJAX-Library.aspx.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect (www.wintellect.com), a software consulting and education firm that specializes in the Microsoft .NET Framework. Contact Jeff at wicked@microsoft.com.


Foundations: Templates for Uncommon Controls

Foundations
Templates for Uncommon Controls
Charles Petzold

Code download available at: Foundations2008_01.exe (784KB)
Contents
Elements and Controls
Default Templates and DLLs
Control Derivation Basics
A Hierarchy of Elements and Controls
Code and XAML
Generating Commands
Replacing the Template

F
or programmers who enjoy turning common controls into uncommon visual objects, Windows® Presentation Foundation (WPF) offers an exciting feature known as the template. Traditionally, the functionality of a control and its visual appearance were bound together deep within the control's code. In WPF, the functionality of a control is still implemented in code, but the visuals are decoupled from that code and take the form of a template defined in XAML. By creating a new template—most often in XAML without a lick of code—programmers and designers can completely revise the visual appearance of the control without having to modify the control's code.

In my inaugural column one year ago, I showed how to design templates for ScrollBar, ProgressBar, and Slider controls. But the templating feature has a flip side: when designing a new custom control, you should provide a default template for the control's visual appearance and also allow that template to be replaced by programmers using your control. You aren't strictly required to structure your control like this—indeed, none of the custom controls in my book Applications = Code + Markup (Microsoft Press®, 2006) define replaceable templates—but people who need to use that control (including you) will be a lot happier if you do.

The purpose of this column is not to create fully functional and picturesque controls but to establish the mechanics for defining default replaceable templates for controls distributed in dynamic link libraries. Many of the templating techniques I discuss here I learned by examining templates on existing WPF controls. If you want to do the same, the DumpControlTemplate program in chapter 25 of Applications = Code + Markup lets you extract the default templates from all the standard WPF controls in convenient XAML form.

Elements and Controls

Programmers with experience in previous Windows client programming environments soon discover a curiosity in the WPF class hierarchy. In the native Windows API, for example, everything that has an on-screen visual appearance is categorized as a "window," while in Windows Forms, everything is a "control." But in WPF, the Control class derives from FrameworkElement, as do many other visual objects—most notably TextBlock, Image, Decorator, and Panel. What exactly, then, is the difference between an element and a control?

First, the Control class adds a collection of very handy properties to the FrameworkElement class, including Foreground, Background, and five font-related properties. Control doesn't use these properties directly—they're for the convenience of classes that derive from Control.

Second, the Control class adds IsTabStop and TabIndex properties, implying that controls often need to be stops in the tab-key navigation chain, while elements do not. In short, elements are for looking and controls are for interaction (but elements can still obtain focus and respond to keyboard, mouse, and stylus input).

Third, the Control class defines the Template property of type ControlTemplate. This template is mostly a visual tree of elements and other controls that make up the control's visual appearance and also often contains triggers that change this visual appearance based on property changes and events.

This third feature means that classes that derive from Control have a customizable visual appearance, while other classes that derive from FrameworkElement do not. Certainly TextBlock and Image have a visual appearance, but customizing these visuals doesn't make sense because these elements don't add anything to the formatted text or the bitmap they display. A ScrollBar, on the other hand, can have a variety of appearances and still be functionally the same. That's what the template is for.

For a programmer, here's perhaps the biggest difference between elements and controls: if you derive from FrameworkElement, you'll very likely need to override MeasureOverride, ArrangeOverride, and OnRender to render the element's visual elements and its children on the screen. If you derive from Control, you do not usually need to override these methods because the control's visuals are defined by the visual tree in the Template property's ControlTemplate object.

WPF includes a class named UserControl that derives from Control by way of ContentControl. This UserControl is often recommended as a base class for simple custom controls, and it serves fine for many purposes. The DatePicker control in Chapter 25 of my book derives from UserControl, for example. But keep in mind this important distinction between Control and UserControl: when deriving from UserControl, you can define a visual tree in XAML, but this visual tree is a child of UserControl's Content property. UserControl has its own simple default template that you probably won't replace because it simply nests a ContentPresenter inside a Border.

The visual tree of a class that derives from UserControl is not meant to be replaced, so the code of that class and the visual tree can be more tightly coupled. Conversely, when you intend to derive from Control and supply a replaceable default template, the interaction between the code and visual tree should instead be both simple and well-documented.

Default Templates and DLLs

For this column, I decided to take another crack at calendar controls. The source code for this column is a single Visual Studio® solution named CalendarTemplateDemo. It contains a library project that creates a DLL named CalendarControls and four demonstration programs that use these controls.

The CalendarControls library contains code and default templates for three controls in the CalendarControls namespace named CalendarMonth, CalendarDay, and CalendarDayNotes. Each of these classes is defined in its own C# file. The first two classes derive from Control, and CalendarDayNotes derives from CalendarDay.

There are very strict rules for defining default templates in a DLL so they can be replaced by applications using the controls. If you don't follow these rules, you're likely to create a control that doesn't have a default template or one that has a template that can't be replaced by an application. The DLL project must have a directory named Themes containing a file named generic.xaml with a root element of ResourceDictionary. The resources are Style elements targeting the controls in your DLL. The generic.xaml file in my CalendarControls library might look something like Figure 1.

Figure 1 Generic.xaml for CalendarControls (Close Figure 1)
<ResourceDictionary
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:cc="clr-namespace:CalendarControls">
    <Style TargetType="cc:CalendarMonth" />
        ...
    </Style>

    <Style TargetType="cc:CalendarDay" />
        ...
    </Style>

    <Style TargetType="cc:CalendarDayNotes" />
        ...
    </Style>
</ResourceDictionary>

Each of the Style elements in this file has at least one Setter element for setting the control's default template:

<Setter Property="Template">
    <Setter.Value>
        <ControlTemplate TargetType="cc:CalendarMonth">
            ...
        </ControlTemplate>
    </Setter.Value>
</Setter>

The ControlTemplate contains the visual tree for the control. Optionally, it contains a Resources section for defining some resources used within the template. Very often, there will be a Triggers section that defines how the template reacts to changes in other properties or events.

Alternatively, you can separate these default templates into separate files, which is what I did. The Themes directory of the CalendarControls project contains three files named CalendarMonthStyle.xaml, CalendarDayStyle.xaml, and CalendarDayNotesStyle.xaml. Each of these files has a root element of ResourceDictionary and a single child of type Style targeting the particular control. The generic.xaml file references these three resource files, as shown in Figure 2.

Figure 2 Referencing Themes from Generic.xaml (Close Figure 2)
<ResourceDictionary
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:cc="clr-namespace:CalendarControls">
    <ResourceDictionary.MergedDictionaries>
        <ResourceDictionary
            Source="CalendarControls;Component/themes/
                CalendarMonthStyle.xaml" />
        <ResourceDictionary
            Source="CalendarControls;Component/themes/
                 CalendarDayStyle.xaml" />
        <ResourceDictionary
            Source="CalendarControls;Component/themes/CalendarDayNotesStyle.xaml" />
    </ResourceDictionary.MergedDictionaries>
</ResourceDictionary>

To ensure that these styles are successfully united with objects of type CalendarMonth, CalendarDay, and CalendarDayNotes, these classes must return a correct value from their DefaultStyleKey properties. For this reason, the static constructors of these classes alter the default value of that property to return the type of the class. This is accomplished by overriding the metadata of the DefaultStyleKey dependency property:

static CalendarMonth()
{
    DefaultStyleKeyProperty.OverrideMetadata(typeof(CalendarMonth),
        new FrameworkPropertyMetadata(typeof(CalendarMonth)));
    ...
}

The first argument to OverrideMetadata is the type of the class modifying the metadata; the argument to the FrameworkPropertyMetadata constructor indicates the new default value of the DefaultStyleKey property, which is also the type of the class.

The assembly information file for the DLL (a file usually named AssemblyInfo.cs) is a very good place to put the following assembly attribute:

[assembly: ThemeInfo(ResourceDictionaryLocation.None,
                     ResourceDictionaryLocation.SourceAssembly)]

This means that there is no theme-specific set of templates, but generic templates for the controls are located in the same assembly (the DLL) as the controls themselves.

Control Derivation Basics

When deriving from Control, you'll likely need to define a few new public properties. In most cases, you should back these properties with dependency properties so they can be targets of data bindings and animations. Even if they are get-only properties and can't be binding targets, the use of dependency properties provides a notification mechanism so that other objects can be informed when these properties change.

If you need to define a property that is already defined by some other class but that you do not inherit from Control, don't entirely redefine the property. Instead, call AddOwner on the existing dependency property. Save the return value from AddOwner as a public static read-only field and use that value in the CLR property.

If you need to change the default value of an inherited property, you have a couple of options. Simply setting the property's value in the class's constructor is not a good idea because this so-called local setting has a very high precedence and can't be overridden by a style or property. The low-precedence approach is calling OverrideMetadata on the dependency property, as I showed earlier with the DefaultStyleKey property. The Control class itself sets a default true value of IsTabStop in this manner.

Another low-precedence approach is setting a property with a Setter in the same Style element as the control's default template. Many of the WPF controls use this technique to set properties that relate to the visual appearance of the control, such as SnapsToDevicePixels, MinWidth, or MinHeight. If your class needs to be notified whenever an inherited property's value changes, you can use OverrideMetadata to install an additional callback method.

One of the biggest challenges in deriving from Control is defining the interactions between the code and the template in a way that provides for all of your needs but won't baffle someone trying to write an alternative template. Generally, your control code accommodates the template in several ways:

  • The code defines properties that can be accessed by the template through the TemplateBinding markup extension.
  • The code defines properties and events that the template can use as triggers.
  • The code defines RoutedCommand properties or fields that buttons within the template can fire.
  • The code assumes that certain helper elements are present in the template; sometimes it references these elements through pre-defined names.

I'll have examples of all these techniques.

In some cases, your Control derivative class will need to be aware of when the template is replaced so that it can access the new template and establish links to it. The Control class defines a virtual OnTemplateChanged method that you can override to be notified when the Template property has changed. However, it is my experience that the OnTemplateChanged method is not really a very good place to perform any required linking because the template has not yet been applied (to use the WPF jargon). A much better strategy to employ is overriding the OnApplyTemplate method that, oddly enough, is defined by FrameworkElement but has no default implementation.

A Hierarchy of Elements and Controls

The MonthCalendar control in the CalendarControls library displays a single month of the year. The TwoCalendars program displays two instances of MonthCalendar, one in English and the other in French, as shown in Figure 3.

Figure 3 Two Instances of MonthCalendar
Figure 3 Two Instances of MonthCalendar(Click the image for a smaller view)
Figure 3 Two Instances of MonthCalendar
Figure 3 Two Instances of MonthCalendar (Click the image for a larger view)

It is in keeping with the WPF design philosophy that default templates look just like their Win32® counterparts—which are usually somewhat bland, and the calendars you see here certainly are. But one of my goals from the beginning of this project was to correctly implement calendars that don't necessarily begin with Sunday (or dimanche, in French). However, I didn't venture beyond the Gregorian calendar.

In order to keep these examples relatively simple, I decided that I would not implement any concept of selecting dates. As you'll see, however, there is a property named IsToday that causes a particular day to be highlighted.

One of the big lessons I've learned with WPF is that complex controls can be built from simpler controls and elements. None of what you see in Figure 3 is defined by the code part of CalendarMonth. It's all part of the default XAML template: a border surrounds the whole thing and provides a background. The buttons on top are RepeatButton controls; they navigate backward and forward by one month or one year. Between the buttons is a TextBlock. The days of the week are StatusBarItem objects on a StatusBar using a UniformGrid as an ItemsPanel. The days of the month are also displayed in a UniformGrid.

For reasons that will become apparent, I decided that the individual days should be a separate control that I called CalendarDay. CalendarDay also derives from Control; the default template is a Border containing a TextBlock. The CalendarMonth control generates 28, 29, 30, or 31 CalendarDay objects when it displays a month. The TwoCalendars.xaml file that displays these two calendars is shown in Figure 4.

Figure 4 TwoCalendars.xaml (Close Figure 4)
<!-- TwoCalendars.xaml by Charles Petzold, Sept. 2007 -->
<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:cc="clr-namespace:CalendarControls;assembly=CalendarControls"
  x:Class="Petzold.TwoCalendars.TwoCalendars"
  Title="Two Calendars Demonstration">

  <Window.Resources></Window.Resources>

  <Grid>
    <Grid.ColumnDefinitions>
      <ColumnDefinition Width="Auto" />
      <ColumnDefinition Width="Auto" />
    </Grid.ColumnDefinitions>

    <Grid.RowDefinitions>
      <RowDefinition Height="Auto" />
    </Grid.RowDefinitions>

    <cc:CalendarMonth Grid.Column="0" Margin="24" />

    <cc:CalendarMonth Grid.Column="1" Margin="24"
      Culture="fr-FR" />
  </Grid>
</Window>

Code and XAML

Because CalendarDay is much simpler than CalendarMonth and short enough to show in its entirety, that's the part of this project I'd like to focus on first. Figure 5 shows CalendarDay.cs. Notice that this is not a partial class. The default template is not part of the CalendarDay class; instead, it is automatically applied to objects of type CalendarDay.

Figure 5 CalendarDay.cs (Close Figure 5)
// CalendarDay.cs by Charles Petzold, Sept. 2007
using System;
using System.Windows;
using System.Windows.Controls;

namespace CalendarControls
{
    public class CalendarDay : Control
    {
        // Static constructor: Change defaults.
        static CalendarDay()
        {
            DefaultStyleKeyProperty.OverrideMetadata(typeof(CalendarDay),
                new FrameworkPropertyMetadata(typeof(CalendarDay)));

            IsTabStopProperty.OverrideMetadata(typeof(CalendarDay),
                new FrameworkPropertyMetadata(false));
        }

        // Date dependency property and property.
        public static readonly DependencyProperty DateProperty =
            DependencyProperty.Register("Date",
                typeof(DateTime),
                typeof(CalendarDay),
                new PropertyMetadata(DateChangedCallback));

        public DateTime Date
        {
            set { SetValue(DateProperty, value); }
            get { return (DateTime)GetValue(DateProperty); }
        }

        // IsToday dependency property and property.
        static readonly DependencyProperty IsTodayProperty =
            DependencyProperty.Register("IsToday",
                typeof(bool),
                typeof(CalendarDay),
                new PropertyMetadata(false));

        public bool IsToday
        {
            set { SetValue(IsTodayProperty, value); }
            get { return (bool)GetValue(IsTodayProperty); }
        }

        // Day read-only dependency property and property.
        static readonly DependencyPropertyKey DayKey =
            DependencyProperty.RegisterReadOnly("Day",
                typeof(string),
                typeof(CalendarDay),
                new PropertyMetadata());

        public static readonly DependencyProperty DayProperty =
            DayKey.DependencyProperty;

        public string Day
        {
            protected set { SetValue(DayKey, value); }
            get { return (string)GetValue(DayProperty); }
        }

        // DateChangedCallback method
        static void DateChangedCallback(DependencyObject obj,
           DependencyPropertyChangedEventArgs args)
        {
            CalendarDay calday = obj as CalendarDay;
            calday.Day = calday.Date.Day.ToString();
        }
    }
}

The class defines three new properties, all backed by dependency properties: Date of type DateTime, IsToday of type bool, and a get-only property named Day of type string. The Day property is entirely for the benefit of the template; whenever the Date property changes, the class converts the Day property of the DateTime object to a string and sets it to its own Day property.

The CalendarDayStyle.xaml file contains the template for the CalendarDay control. A template often begins with a Border element whose properties are set through TemplateBinding extensions to the same-named properties defined by the Control class. If these properties aren't explicitly referenced in the template, they have no effect on the visual appearance of the control!

The Control properties you do not need to reference in the template are those that are inherited through the visual tree. These inherited properties are Foreground and the five font-related properties. Any change to those properties will be automatically inherited by the TextBlock, which displays the Day property defined by the CalendarDay class. The HorizontalContentAlignment and VerticalContentAlignment properties determine how that text is aligned in the cell. The template concludes with a Triggers section, as shown below:

<Trigger Property="IsToday" Value="True">
  <Setter Property="Background"
    Value="{DynamicResource
    {x:Static SystemColors.HighlightBrushKey}}" />
  <Setter Property="Foreground"
    Value="{DynamicResource
    {x:Static SystemColors.HighlightTextBrushKey}}" />
</Trigger>

When the IsToday property defined by CalendarDay is true, then the Background and Foreground properties are set to system brushes for a highlighted item. This Triggers section can also contain MultiTrigger elements and EventTrigger elements, and the EventTrigger elements can trigger animations.

Although CalendarDay contains a Border element, there is no visible border. The template for CalendarMonth is defined similarly, and there's no visible border around the entire month either. The border is not visible because Control defines the default value of BorderBrush to be null and the default value of BorderThickness to be zero, just like the Border element itself. You might feel very strongly that each day should be enclosed in a visible box. Why not just set the properties in the template to more sensible values?

<Border BorderThickness="1"
        BorderBrush="Black" ...

The problem with this approach is that any future programmer working with the control will have to alter the default template to change these values. The template should have a minimum of hardcoded property values. If you're the designer of the control and its default template, you can use the Style element in the resource file to set these properties to better defaults. If you're a consumer of the control, you can set these properties when using the control.

For example, in the TwoCalendars.xaml file, you can change the first calendar like this:

<cc:CalendarMonth BorderThickness="1"
                  BorderBrush="Black" ...

That will put a visible border around the whole calendar. But how do you set the CalendarDay border? The CalendarDay element doesn't even show up in the TwoCalendars.xaml file because CalendarMonth is generating all the CalendarDay objects internally!

The solution is to use a style that targets CalendarDay. Try inserting this in the Resources section of TwoCalendars.xaml:

<Style TargetType="cc:CalendarDay">
    <Setter Property="BorderThickness" Value="1" />
    <Setter Property="BorderBrush"
            Value="{DynamicResource
                {x:Static SystemColors.ControlTextBrushKey}}" />
    <Setter Property="HorizontalContentAlignment"
            Value="Center" />
    <Setter Property="VerticalContentAlignment"
            Value="Center" />
    <Setter Property="FontStyle" Value="Italic" />
</Style>

This style not only sets the border but also the content-alignment properties to center the dates in their cells, and it makes the numbers italic as well.

The default CalendarMonth template is contained in the file CalendarMonthStyle.xaml. It begins with a Border, then a Grid of three vertical cells for the buttons and month name, the days of the week, and the calendar grid itself. The top cell of the Grid is another Grid with five horizontal cells for the buttons and month name.

The CalendarMonth class makes available several properties that the template can use to display textual information, such as MonthName, AbbreviatedMonthName, and a formatted AbbreviatedYearMonth. (CalendarMonth obtains much of this information from DateTimeFormatInfo.) CalendarMonth also stores the names of the days of the week in two arrays available as the DayNames and AbbreviatedDayNames properties. These are a little different from the same-named properties in the DateTimeFormatInfo class because the arrays in DateTimeFormatInfo always begin with Sunday.

The FirstDayInWeek property defined by CalendarMonth is based on the day of the week of the first day in the month and the FirstDayOfWeek property from DateTimeFormatInfo.

The template found in CalendarMonthStyle.xaml contains its own Resources section. This is handy for setting styles within the template, particularly for element types that don't explicitly show up in the markup. For example, notice how the days of the week are displayed. The AbbreviatedDayNames property is assigned to a StatusBar that has a UniformGrid as its ItemsPanel. Internally, the strings in the AbbreviatedDaysNames array become the Content properties of StatusBarItem controls. StatusBarItem does not actually show up in this markup, but a Style can target StatusBarItem controls to center the content and insert some space between the names.

Accessing Named Elements

So far you've seen examples of how a class can define properties that the XAML file references through TemplateBinding extensions and triggers. But sometimes it is more convenient for a class to access a particular element in the template. In this example, the CalendarMonth class needs to generate all the CalendarDay objects for a particular month, and these objects must end up in some kind of panel. It seemed most convenient to assign this panel a name that the code references.

As you'll note, the second UniformGrid in the CalendarMonth template is given a name of PART_Panel. This name is similar to the names defined in the default templates for several WPF controls. The CalendarMonth class definition is preceded by an attribute indicating this name and the type of the element that the code expects it to be identifying:

[TemplatePart(Name = "PART_Panel", Type = typeof(Panel))]

This attribute is for the benefit of visual designers; it is not required. But notice that I've implied that this element in the template can be any derivative of Panel.

In response to a call to ApplyTemplate, this statement in CalendarMonth.cs obtains this panel and stores it as a field:

pnl = Template.FindName("PART_Panel", this) as Panel;

Notice that this is not the normal FindName method. It's the FindName method defined by FrameworkTemplate, from which ControlTemplate (and ItemsPanelTemplate and DataTemplate and HierarchicalDataTemplate) is derived, so it's accessed through the Template property of the CalendarMonth object.

What happens if this pnl object is null? It means the template does not include anything named PART_Panel, or perhaps an element named PART_Panel does not derive from Panel. If a class can't find a named element in its template, it should silently just continue on the best it can.

When associating names and types with helper elements in the template, try to be as general as possible. In this case, MonthCalendar needs an element with a Children property, so Panel seems reasonable. MonthCalendar does not require a certain type of panel, although it's obvious that a UniformGrid works quite well.

Generating Commands

The template for CalendarMonth must contain buttons to navigate forward and backward. How do you write code that responds to these button presses? The most convenient method is for the class to define public static read-only fields or get-only properties of type RoutedCommand. The existing WPF controls are inconsistent in defining these RoutedCommand objects as fields or properties. The ScrollBar class defines them as read-only fields and names them LineDownCommand, LineLeftCommand, and so forth. The Slider class defines them as get-only properties named DecreaseLarge, DecreaseSmall, IncreaseLarge, and IncreaseSmall. I chose the ScrollBar approach and defined my RoutedCommand objects as fields, like this:

public static readonly RoutedCommand NextMonthCommand =
    new RoutedCommand("NextMonth",
    typeof(CalendarMonth));

Notice that these are static! The template refers them with fully qualified names assigned to the Command property of the control:

<ToggleButton Command="
CalendarMonth.NextMonthCommand" ...

Not very many controls define Command properties that you can set to an object of type RoutedCommand. The only ones are ButtonBase, MenuItem, and Hyperlink.

The CalendarMonth constructor links these RoutedCommand objects with an execution method by adding the objects to its CommandBindings collection:

CommandBindings.Add(new
  CommandBinding(NextMonthCommand,
  NextMonthExecuted));

In addition, you can also specify a can-execute method that allows the code to set a Boolean indicating whether the control is valid. If not, it is automatically disabled.

The NextMonthExecuted method in CalendarMonth.cs looks like this:

void NextMonthExecuted(object sender,
ExecutedRoutedEventArgs args)
{
    Date = Date.AddMonths(1);
}

One advantage of using RoutedCommand objects is that you can trigger them from code fairly easily by calling the Execute method. You might want to trigger the command from some keyboard input, for example. But in many cases you can accomplish this job without any explicit keyboard handling by associating objects of type InputGesture (an abstract class from which KeyGesture and MouseGesture derive) with the RoutedCommand. CalendarMonth's static constructor adds KeyGesture objects to all four RoutedCommand fields, like this:

NextMonthCommand.InputGestures.Add(
    new KeyGesture(Key.PageDown));

These gestures could have been added in the original field definition for the RoutedCommand objects, but it's a little clumsy because you need to provide an entire InputGestureCollection in the constructor.

Replacing the Template

The real test of this entire exercise comes when you write an application that attempts to replace the default template and make the calendar look completely different. The NewCalendar program defines a new template for CalendarMonth that displays the days of the month vertically. The program creates 12 of these monthly calendars and displays them side-by-side(see Figure 6).

Figure 6 The NewCalendar Display
Figure 6 The NewCalendar Display

It is also desirable to enhance the calendar by deriving new classes from CalendarDay, but at first this doesn't seem possible: CalendarMonth is responsible for generating the 28, 29, 30, or 31 instances of CalendarDay. It seems that if you wish to derive from CalendarDay, you must also derive from CalendarMonth to change that logic. To avoid that, I defined yet another property of CalendarMonth called DayType of type Type. By default, this property is equal to typeof(CalendarDay), but it can also be set to the type of a class that derives from CalendarDay.

CalendarDayNotes derives from CalendarDay and expects its template to contain a control of type TextBox with a name of PART_TextBox. Whatever you type into those TextBox controls is saved by CalendarDayNotes in little files identified by a file name derived from the date. The next time you run the program, it loads them all in. CalendarDayNotes has a rather bland default template, but the ReminderCalendar program includes a slightly fancier template that displays the day of the month in an elongated shape and gradiated with color, as shown in Figure 7.

Figure 7 The ReminderCalendar
Figure 7 The ReminderCalendar

It's not even necessary to derive from CalendarDay to make the days of the month a little fancier. The CalendarControls library also has a class named MoonDisk that derives from Viewport3D and has a DateTime property to orient lighting on a sphere of the moon. The MoonPhaseCalendar doesn't create the speediest calendar in the world—it needs to generate up to 31 Viewport3D objects for each month—but it does show a little more than the average calendar, as Figure 8 demonstrates.

Figure 8 The MoonPhaseCalendar Display
Figure 8 The MoonPhaseCalendar Display

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a Contributing Editor to MSDN Magazine. His most recent book is 3D Programming for Windows (Microsoft Press, 2007).


Extreme ASP.NET: Encapsulate Silverlight with ASP.NET Controls

Extreme ASP.NET
Encapsulate Silverlight with ASP.NET Controls
Fritz Onion

Code download available at: ExtremeASPNET2008_01.exe (700KB)
Contents
Using Silverlight
Building a Custom Silverlight Control
ASP.NET AJAX and IScriptControl
The asp:Xaml and asp:Media Controls

M
any ASP.NET developers around the world are wondering how and where they should incorporate SilverlightTM into their applications. The answer isn't obvious because there is a wide range of approaches you can take. You can make your entire page a Silverlight control and do everything there or, the more likely approach, you can identify portions of your pages that would benefit from Silverlight and integrate rich UI elements where they add the most benefit, leaving the underlying application structure the same.

The latter approach is sometimes referred to as "adding islands of richness" to your pages. And with Silverlight, those islands have many possible bridges to the surrounding content via scriptable methods and events.

In this month's column, I'm going to explore techniques for building custom server controls that encapsulate Silverlight content. Wrapping Silverlight content into a custom ASP.NET control has several advantages that make it an especially appealing technique. It makes integrating Silverlight content as easy as using any other server-side control, which greatly increases the chances of adoption. The process of adding handlers, setting properties, and calling methods on the Silverlight control is exactly the same as with any other control. And this approach keeps pages free from the clutter of Silverlight-specific JavaScript, making it easier to maintain and deploy.

Using Silverlight

Before I jump into the details of building a custom control to host Silverlight content, I should describe precisely what needs to be in place for Silverlight content to render to the client. So I will begin by showing you how to host a XAML file in an ASP.NET page and add interaction between the page and the rendered Silverlight content. It will be clearer how to put the pieces of the control together once I have a page that exhibits the behavior I ultimately want to encapsulate with a control.

The first step is to write a XAML file to be rendered by Silverlight. My goal is to keep the XAML simple enough to be unobtrusive in presenting the control but complex enough to be interesting. So I have chosen to render a single sphere with a title. The XAML shown in Figure 1 defines a TextBlock used as a title and an Ellipse painted with a radial gradient brush. This XAML also defines two Storyboards in the resources portion of the Canvas, describing two animations that will cause the sphere to grow and shrink, respectively, over durations of two seconds each. Ultimately, I want to host this XAML in Silverlight and then use JavaScript handlers to set the title text and invoke the animations as the user clicks on the rendered sphere.

Figure 1 Sphere.xaml File (Close Figure 1)
<!-- File: Sphere.xaml -->
<Canvas
    xmlns="http://schemas.microsoft.com/client/2007"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Width="300" Height="300"
    Background="White"
    >
  <Canvas.Resources>
    <Storyboard x:Name="growAnimation">
      <DoubleAnimation
        Storyboard.TargetName="ellipse"
        Storyboard.TargetProperty="(Ellipse.Width)"
        To="250" Duration="0:0:2" />
      <DoubleAnimation
        Storyboard.TargetName="ellipse"
        Storyboard.TargetProperty="(Ellipse.Height)"
        To="250" Duration="0:0:2" />
    </Storyboard>
    <Storyboard x:Name="shrinkAnimation">
      <DoubleAnimation
        Storyboard.TargetName="ellipse"
        Storyboard.TargetProperty="(Ellipse.Width)"
        To="200" Duration="0:0:2" />
      <DoubleAnimation
        Storyboard.TargetName="ellipse"
        Storyboard.TargetProperty="(Ellipse.Height)"
        To="200" Duration="0:0:2" />
    </Storyboard>
  </Canvas.Resources>

  <TextBlock x:Name="titleText" Width="200" Height="24"
        Canvas.Left="94" Text="[Title]" TextWrapping="Wrap" />

  <Ellipse Width="200" Height="200"
     x:Name="ellipse" Canvas.Left="47" Canvas.Top="41">
    <Ellipse.Fill>
      <RadialGradientBrush GradientOrigin="0.75,0.25"
         Center="0.5,0.5" RadiusX="0.5" RadiusY="0.5">
        <GradientStop Color="Yellow" Offset="0" />
        <GradientStop Color="Green" Offset="1" />
      </RadialGradientBrush>
    </Ellipse.Fill>
  </Ellipse>

</Canvas>

To incorporate this XAML into a Web page, I first need to create the Silverlight plug-in. The simplest way to do this is to add a reference to the Silverlight.js script file that is included with the Silverlight 1.0 SDK. This file defines a method called Silverlight.createObject, which you can use to create the plug-in. The Silverlight plug-in needs to be associated with an HTML element on the page (typically a <div>), so it is common practice to define the <div> element and place the call to createObject within a script block embedded within the <div>.

Figure 2 shows how to create an instance of the Silverlight control and associate with a <div> element. This page assumes that the Sphere.xaml file shown in Figure 1 is in the same directory as the page itself. When viewed in your Web browser, this page displays a lovely green sphere, as shown in Figure 3.

Figure 2 Create and Associate Silverlight Control (Close Figure 2)
<%@ Page Language="C#" %>

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <script type="text/javascript" src="silverlight.js"></script>
</head>
<body>
    <form id="form1" runat="server">
    <div id="slControlHost">
    <script type="text/javascript">
        Silverlight.createObject(
        "Sphere.xaml",
        document.getElementById('slControlHost'), "slControl",
        { width:'300', height:'300', version:'1.0' },
        { onError:null, onLoad:null },
        null);
    </script>
    </div>
    </form>
</body>
</html>
Figure 3 Control Displaying a Green Sphere
Figure 3 Control Displaying a Green Sphere

The next task is to interact with the XAML so that I can programmatically change the title text and initiate the grow and shrink animations I described earlier. The simplest way to do this is to declaratively assign handlers to events on objects defined in the XAML. In this example, the grow and shrink animations occur in response to the user clicking the sphere—which animation it runs depends on the current size of the sphere.

To begin, I add a handler to the MouseLeftButtonDown event of the Ellipse that defines the sphere, and then I place the corresponding method in JavaScript on the page. Similarly, I handle the Loaded event of the TextBlock to set the text displayed above the sphere to a custom string. The handlers that I add to the XAML look like this:

<Ellipse MouseLeftButtonDown="javascript:onSphereButtonDown" ...
<TextBlock Loaded="javascript:onTextLoaded" ...

And the corresponding JavaScript methods that I add to a new script block on the ASP.NET page are shown in Figure 4.

Figure 4 JavaScript Methods Added to New Script Block (Close Figure 4)
<script type="text/javascript">
    function onSphereButtonDown(sender, args)
    {
        // Run the grow or shrink animation,
        // depending on whether it is currently
        // 'grown' or 'shrunk'
        var animationName = (sender.Width==200) ?
              "growAnimation" : "shrinkAnimation";

        var sl = sender.getHost();
        var animation = sl.content.findName(animationName);
        if (animation)
          animation.begin();
    }

    function onTextLoaded(sender, args)
    {
      sender.Text = "My Growing Sphere";
    }
</script>

That is all it takes to manually incorporate Silverlight content into an ASP.NET page. When this page is displayed in a browser, the title text says "My Growing Sphere". When the user clicks on the sphere, it grows on the first click and shrinks on the second. As you can see, it is relatively easy to integrate Silverlight content into an ASP.NET page, and it is easy to manipulate since its entire object model is exposed to script on the page.

For a simple element like my sphere, however, you may want to package it as a custom control that someone can incorporate into an application simply by dragging the control onto the design surface of a page. This is where encapsulating Silverlight comes into play.

Building a Custom Silverlight Control

I begin the task of encapsulating Silverlight by creating a new custom ASP.NET control:

namespace MsdnMagazine
{
    public class SilverlightSphere  : Control
    {
    }
}

In this sample, I derive from System.Web.UI.Control. I've chosen not to derive from WebControl since the additional style properties defined by that class are not applicable to this control.

Next, I need to render the JavaScript handlers that I included manually. The easiest way to incorporate JavaScript in an ASP.NET control is to define a JavaScript file and embed it as a resource in the compiled assembly. I can do this by moving our two event handlers into a separate JavaScript file, which I call SilverlightSphere.js:

   // File: SilverlightSphere.js
   function onSphereButtonDown(sender, args)
   {
       // Same implementation as before
   }

   function onTextLoaded(sender, args)
   {
      // Same implementation as before
   }

Next, I embed my new SilverlightSphere.js file and the original Silverlight.js file as resources in the control project. This is so that the control can be deployed as a single assembly without any dependent files. It also makes sense to embed the XAML file as a resource and reference both it and the two JavaScript files using the WebResource.axd handler mechanism in ASP.NET for extracting embedded resources.

After I set the Build Action to Embedded Resource for all three files in Visual Studio® (/res from the command line compiler), I then use the assembly-level attribute System.Web.UI.WebResource to grant permission for these resources to be served by WebResource.axd and to associate a MIME type for the response. I use the following declarations to accomplish this (where SlSphere is the name of the control project):

[assembly: WebResource("SlSphere.SilverlightSphere.js",
                       "text/javascript")]
[assembly: WebResource("SlSphere.Silverlight.js",
                       "text/javascript")]
[assembly: WebResource("SlSphere.Sphere.xaml", "text/xml")]

The JavaScript files are now compiled into my assembly as embedded resources. So now I can use the RegisterClientScriptResource method of the ClientScriptManager class (accessed through Page.ClientScriptManager) to cause the rendered page to include references to the files. I call this method for each file in an override of the OnInit method inherited from the Control base class:

protected override void OnInit(EventArgs e)
{
  Page.ClientScript.RegisterClientScriptResource(this.GetType(),
                      "SlSphere.Silverlight.js");
  Page.ClientScript.RegisterClientScriptResource(this.GetType(),
                      "SlSphere.SilverlightSphere.js");

  base.OnInit(e);
}

The last and most important piece of the control is to implement the virtual Render method. There are two things I have to accomplish in this method. First, I need to render a <div> tag to act as the host HTML element for the Silverlight plug-in. Second, I must render embedded script to create the Silverlight plug-in.

The reference to the XAML file is generated with a call to the ClientScriptManager class's GetWebResourceUrl. This will ensure that the Silverlight control is initialized with the content of the XAML file I embedded as a resource in the assembly. Finally, I need to specify a unique identifier for the Silverlight plug-in itself. In this case, I do this by taking the ID of the control (this.ClientID) and concatenating "_ctrl" to the end (see Figure 5).

Figure 5 Implement the Virtual Render Method (Close Figure 5)
Const string _silverlightCreateScript = @"Silverlight.createObject(
  '{0}, document.getElementById('{1}'), '{2},
  {{ width:'300', height:'300', version:'1.0' }},
  {{ onError:null, onLoad:null }},
  null);";

protected override void Render(HtmlTextWriter writer)
{
  string script = string.Format(_silverlightCreateScript,
      Page.ClientScript.GetWebResourceUrl(GetType(),
               "SlSphere.Sphere.xaml"),
               this.ClientID, this.ClientID + "_ctrl");

  writer.AddAttribute(HtmlTextWriterAttribute.Id,
                      this.ClientID);
  writer.RenderBeginTag(HtmlTextWriterTag.Div);
  writer.AddAttribute(HtmlTextWriterAttribute.Type,
                      "text/javascript");
  writer.RenderBeginTag(HtmlTextWriterTag.Script);

  writer.Write(script);

  writer.RenderEndTag();
  writer.RenderEndTag();

  base.Render(writer);
}

I now have a server-side control (complete with embedded JavaScript and XAML resources) that encapsulates the pieces necessary to deploy Silverlight on an ASP.NET page. I can create an instance of this control on a page with the following declarations:

<%@ Register Assembly="SlSphere" Namespace="MsdnMagazine"
             TagPrefix="csc" %>
...
<csc:SilverlightSphere ID="_silverlightSphere" runat="server" />

What I am able to do here is compelling—any ASP.NET application can easily incorporate a growing and shrinking sphere hosted in Silverlight, yet there's no need to worry about the details of hosting Silverlight content.

You could, of course, build more interesting and complex controls with sophisticated XAML renderings. However, when you start to think about things you might want to add to your controls, you quickly encounter the problem of how to instrument the XAML with property values from the server control. For example, say I want to expose a Title property in my control and have that map to the Text property of the titleText element in my XAML file. There is no obvious way to accomplish this cleanly with the current implementation.

Another drawback to the current implementation is that the XAML is fixed as an embedded resource. While this is convenient for deployment, one of the main benefits of Silverlight and XAML is the separation of behavior from design, which allows the XAML to be replaceable. I could define a XamlUrl property on the control to allow overriding of the embedded XAML content. This approach wouldn't be difficult, but it would introduce fragility in exposing the names of the JavaScript handlers in the XAML, which would be required to ensure that the designers have the latest version of the XAML.

ASP.NET AJAX and IScriptControl

The crux of the problem is figuring out how to introduce server-side properties into client-side script. Fortunately, ASP.NET AJAX extensions provide a clean way of tying server-side control properties to client-side elements through the IScriptControl interface, which is defined in the System.Web.Extensions assembly. This interface defines two methods: GetScriptReferences and GetScriptDescriptors. When implemented and tied to the page's ScriptManager control, these methods generate references to JavaScript files and declare instances of JavaScript classes, respectively:

public interface IScriptControl
{
  IEnumerable<ScriptDescriptor> GetScriptDescriptors();
  IEnumerable<ScriptReference> GetScriptReferences();
}

You can think of a descriptor as a client-side surrogate of the server-side control, with properties initialized to those defined by the server control. This provides a link between server control properties and client properties which can be consumed by Silverlight and JavaScript handlers. Descriptors also provide a clean scoping mechanism for isolating all of the control's client-side features under a namespace and class (at least as ASP.NET AJAX client libraries define namespaces and classes in JavaScript).

When building a custom ASP.NET AJAX control that hosts Silverlight content, I first create the client-side class definition with methods and properties defined to interact with the Silverlight control. It makes sense to add both a Title and a XamlUrl property to this control, so I start by defining the class with these properties, naming the class MsdnMagazine.SilverlightAjaxSphere:

// File: SilverlightAjaxSphere.js
Type.registerNamespace('MsdnMagazine');

MsdnMagazine.SilverlightAjaxSphere = function(element) {
    MsdnMagazine.SilverlightAjaxSphere.initializeBase(
               this, [element]);

    this._title = null;
    this._xamlUrl = null;
}

Now I need to populate the methods for the prototype of the class, beginning with initialize (see Figure 6). Instead of embedding script inside a <div> element, as I did in the first control implementation, this time I create the Silverlight plug-in within the initialize method of the client-side class. Note that this is only possible since the _xamlUrl property I defined in the class will be populated with the server-side value prior to this method being called. I can also access the identifier of the <div> element to be rendered by the server-side control with the get_id method inherited from the Sys.UI.Control base class.

Figure 6 Populate the Methods for Class Prototype (Close Figure 6)
// File: SilverlightAjaxSphere.js
MsdnMagazine.SilverlightAjaxSphere.prototype = {
    initialize : function() {
      MsdnMagazine.SilverlightAjaxSphere.callBaseMethod(
             this, 'initialize');
      var hostId = this.get_id() + 'Host';

      Silverlight.createObject(this._xamlUrl,
        $get(this.get_id()), hostId,
        { width:'300', height:'300', version:'1.0' },
        { onError:null,
          onLoad:Function.createDelegate(this, this._onXamlLoaded) },
        null);

    },

    dispose : function() {
        MsdnMagazine.SilverlightAjaxSphere.callBaseMethod(this,
              'dispose');
    },

    // Once called, Silverlight control is initialized and
    // Xaml elements can be accessed.
    _onXamlLoaded : function(root) {
      var root = $get(this.get_id() + 'Host');
      var sphere = root.content.findName('ellipse');

      if (sphere) {
        sphere.addEventListener('MouseLeftButtonDown',
                    Function.createDelegate(this,
                     this._onSphereButtonDown));
      }
      else
        throw Error.invalidOperation(
              "You must have an ellipse element.");

      var title = root.content.findName('titleText');
      if (title)
        title.Text = this._title;
    },

     _onSphereButtonDown : function(sender, args)
    {
      // same implementation as before
    },

    // property accessors
    //
    get_title : function() {
        return this._title;
    },

    set_title : function(value) {
        if (this._title !== value)
            this._title = value;
    },

    get_xamlUrl : function() {
      return this._xamlUrl;
    },

    set_xamlUrl : function(value) {
      if (this._xamlUrl !== value)
        this._xamlUrl = value;
    }
}

MsdnMagazine.SilverlightAjaxSphere.registerClass(
      'MsdnMagazine.SilverlightAjaxSphere', Sys.UI.Control);

if (typeof(Sys) !== 'undefined')
  Sys.Application.notifyScriptLoaded();

When the initialize method is called on my class, the Silverlight plug-in is just being created and the XAML has not yet been evaluated. I therefore use the onLoad event handler defined in the createObject method of Sys.Silverlight to specify a callback to be invoked once the XAML is loaded—this allows me to wire up the event handlers and initialize properties on individual XAML elements. This handler, which I have called _onXamlLoaded, is passed a reference to the root Silverlight element, which I can use to retrieve the named elements in the XAML.

You should be as accommodating as possible when interacting with named XAML elements, which entails checking to see if they exist before actually modifying their content or wiring up event handlers. If the element is missing and is not crucial to the operation of the control (like the titleText element in my control's XAML), just skip the initialization steps for that element. This way, a designer can elect to remove certain elements from the XAML and the control will still work.

The rest of the class's implementation defines the property accessors for title and xamlUrl and implements the _onSphereButtonDown function to initiate the grow and shrink animations, as I did before. This time, however, the method is encapsulated in a class.

The final step in the client-side script is to register the new class, specifying the base class as Sys.UI.Control.

The remaining task is to build the server-side control that uses this class via the IScriptControl interface (see Figure 7). Like the client-side class, the server control exposes two properties: Title and XamlUrl. (These are backed by ViewState, as all server-side control properties should always be.) I initialize the XamlUrl property to the URL, referencing the embedded XAML file using the same GetWebResourceUrl method I used in the last control implementation. By setting this property in an override of OnInit, I ensure that the property will default to this URL, but any overridden version specified by the client will always take priority since state is loaded later in the lifecycle.

Figure 7 Server-Side Control (Close Figure 7)
// File: SilverlightAjaxSphere.cs
//
using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.Web.UI;

[assembly: WebResource("SlAjaxSphere.SilverlightAjaxSphere.js",
                       "text/javascript")]
[assembly: WebResource("SlAjaxSphere.Silverlight.js",
                       "text/javascript")]
[assembly: WebResource("SlAjaxSphere.Sphere.xaml", "text/xml")]

namespace MsdnMagazine
{
  public class SilverlightAjaxSphere  : Control, IScriptControl
  {
    public string Title
    {
      get { return (string)(ViewState["title"] ?? "[title]"); }
      set { ViewState["title"] = value; }
    }

    public string XamlUrl
    {
      get { return (string)(ViewState["xamlurl"] ??
                            string.Empty); }
      set { ViewState["xamlurl"] = value; }
    }

    protected override void OnInit(EventArgs e)
    {
      // Default the XamlUrl to my embedded resource.
      // If this is set in the
      // markup explicitly, it will be overridden.
      XamlUrl = Page.ClientScript.GetWebResourceUrl(GetType(),
                   "SlAjaxSphere.Sphere.xaml");

      base.OnInit(e);
    }

    public IEnumerable<ScriptReference> GetScriptReferences()
    {
      ScriptReference sr1 = new ScriptReference(
        " SlAjaxSphere.Silverlight.js",
        GetType().Assembly.FullName);
      ScriptReference sr2 = new ScriptReference(
        " SlAjaxSphere.SilverlightAjaxSphere.js",
        GetType().Assembly.FullName);

      return new ScriptReference[] { sr1, sr2 };
    }

    public IEnumerable<ScriptDescriptor> GetScriptDescriptors()
    {
      ScriptControlDescriptor scd = new ScriptControlDescriptor(
                 "MsdnMagazine.SilverlightAjaxSphere",
                 this.ClientID);
      scd.AddProperty("title", this.Title);
      scd.AddProperty("xamlUrl", this.XamlUrl);

      return new ScriptDescriptor[] { scd };
    }

    protected override void OnPreRender(EventArgs e)
    {
      if (!DesignMode)
      {
        ScriptManager sm = ScriptManager.GetCurrent(Page);
        if (sm == null)
          throw new HttpException(
              "A ScriptManager control must exist on the current page.");
        sm.RegisterScriptControl(this);
        sm.RegisterScriptDescriptors(this);
      }

      base.OnPreRender(e);
  }
  protected override void Render(HtmlTextWriter writer)
  {
      writer.AddAttribute(HtmlTextWriterAttribute.Id,
                this.ClientID);
      writer.RenderBeginTag(HtmlTextWriterTag.Div);
      writer.RenderEndTag();

      base.Render(writer);
    }
  }
}

The next two methods, GetScriptDescriptors and GetScriptReferences, contain the implementation of the IScriptControl interface. Since I am embedding the JavaScript files as resources in the assembly, I use the overloaded constructor of ScriptReference, which takes the resource name and assembly name. This generates a reference to the embedded script resources in much the same way RegisterClientScriptResource did in the previous control implementation. GetScriptDescriptors is the key to creating an instance of the client-side MsdnMagazine.SilverlightAjaxSphere class initialized with properties from the server-side control.

The ScriptControlDescriptor class takes the name of the client-side class and the ID of the associated client-side element in its constructor. I then use the AddProperty method of the class to specify the initial value of each property of the new client-side class. By passing in the current property values of my control, I ensure that the client-side class will be created with the current property values defined in my control.

I still need to take care of one last bit of housekeeping in an override of OnPreRender, where I register the control as a script control with the current ScriptManager on the page (throwing an exception if there is no ScriptManager) as well as with the script descriptors. This causes the ScriptManager to invoke the two methods of IScriptControl on the control to pull in the script references and create the initial client-side class. The Render method remains identical to the previous control implementation, simply creating the <div> element to act as the host to the Silverlight plug-in.

I now have a complete server-side control that encapsulates Silverlight. In this implementation, the control exposes properties that are used to initialize elements of the XAML (or even replace the entire XAML if need be). The following declarations will create an instance of my new control on a page with the title text set to a custom string:

<%@ Register Assembly="SlAjaxSphere" Namespace="MsdnMagazine"
             TagPrefix="sas" %>
...
<sas:SilverlightSphere ID="_silverlightSphere" runat="server"
                       Title="My custom title!" />

The asp:Xaml and asp:Media Controls

Two new ASP.NET AJAX controls were introduced in the May 2007 ASP.NET Futures release: asp:Xaml and asp:Media. (Note that asp:Media inherits from asp:XAML.) Both these controls provide their own encapsulation of Silverlight, and both are built in much the same way as the sample I presented in the previous section. In fact, these were the inspiration for my control.

The asp:Xaml control lets you associate a XAML file with the control. It includes properties for the Silverlight plug-in creation, handling all the details of creating the plug-in and loading the XAML file. For example, I can use the XAML file to display my Sphere.xaml file with the following declaration:

<asp:Xaml runat="server" ID="_sphereXaml" Windowless="true"
          Width="300px" Height="300px"
          XamlUrl="~/Sphere.xaml"  />

The asp:Media control makes it easy to embed media content (video or audio) into an ASP.NET page, and it really exemplifies the possibilities when creating server-side ASP.NET controls that encapsulate Silverlight. One of the intriguing features of the Media control is that it provides eight different skins (which are XAML files) that you can choose from to present your media—or you can design your own skin. Figure 8 shows an example of the Media control displaying a video using the Expression skin, like so:

Figure 8 Media Control Displays Video
Figure 8 Media Control Displays Video

<asp:Media runat="server" ID="_butterflyVideo"
           MediaSkin="Expression" MediaUrl="~/Butterfly.wmv" />

Another option for creating your own instrumented Silverlight server-side control is to derive from the asp:Xaml control directly and augment it with properties and methods in much the same way as I did in the second control implementation I described. In the code download for this column, I've included another version of the SilverlightSphere control that inherits from asp:Xaml.

Send your questions and comments for Fritz to xtrmasp@microsoft.com.

Fritz Onion is a co-founder of Pluralsight, a Microsoft .NET training provider, where he heads the Web development curriculum. Fritz is the author of Essential ASP.NET (Addison Wesley, 2003) and Essential ASP.NET 2.0 (Addison Wesley, 2006). You can reach him at pluralsight.com/fritz.


{ End Bracket }: Passion for the Craft

{ End Bracket }
Passion for the Craft
Scott Hanselman

A recent discussion in the blogosphere stirred my interest more than usual. The term "5:01 Developer" was used disparagingly to refer to a kind of developer who basically turns off his brain at 5:01 P.M. and heads home. Many developers took umbrage at this term and began to defend their positions by declaring how many hours per day they work. Others suggested terms like "uninspired developers," and there were those who rejected labels of any kind.

To me, the label "5:01 Developer" has nothing to do with time on the job or hours logged. It refers to the lack of passion. Labels or not, I believe that to really excel in the field of software—or any field, for that matter—requires a high degree of passion/inspiration/concern/commitment/enthusiasm/excitement. Your job might involve turning a crank or designing a great new crank that will take the world by storm, but if you want to succeed—whether you're compared to your peers or not—then passion can go a long way.

I'm lousy with wood. Seriously, I'm not handy. If I attempted to use a power saw, I would very likely hurt myself. More importantly, I have no passion for woodworking. I enjoy looking at other people's work and can appreciate the effort to a point, but if I were forced to make my living creating things with wood, I'd certainly be punching out at 5:01. This lack of satisfaction creates the anti-craftsman, who is just slapping two pieces of wood together and hoping the chair won't break. And a sense of craftsmanship is just as important as passion for a job. There's a big difference in semantics between the word "job" and the word "craft." Craftspeople begin as apprentices, learning alongside a master, working their way through as journeymen. When software is built by a craftsman—a person with passion, talent, and a sense of stewardship—you can tell right away. A software steward takes personal responsibility for another person's affairs; in the case of software, this is the user's experience from install to run.

I posted my thoughts on this topic on my blog, and, as is common on blogs, the real gems were not in my own post but in the comments of others. Max Pool nailed it immediately when he said:

Some people, when asked what they do for a living, say, "I AM a software developer," and proudly interlace their persona with their career. This simple reply tells a lot about the amount of passion that someone carries with them every day.

Paul Brazelton shared his thoughts on balance:

I'm a father, a runner, a rower, an environmentalist, a human being, and, yes, a programmer. I'm a lifelong learner, and I love to learn everything, not just programming languages. I love technology, but there are so many other things in this world to love, and we have such a short amount of time.

Clearly, passion, craftsmanship, and stewardship should be part of a larger work ethic for any software engineer (or tester, or manager). Sometimes folks blame their lack of passion on the number of years on the job, or the kind of work, or the quality of the tools. Undoubtedly, when the tools get in the way, the work suffers. I'm particularly excited about software right now because it's becoming easier and easier to get more done faster than before.

I think what keeps things fresh for me when working as a developer are the choices. Not only is there always something new coming out, but there's always better techniques, new thinking, even new people exploring classic technologies. In the coming months, I'm going to dig into IronRuby and the flood of new Dynamic Languages being enabled by the CLR. I'm going to look at Model-View-Controller (MVC) Web Frameworks running on ASP.NET and at code generation tools to create whole administration sites auto-magically. I'm creating games for my Xbox® and plug-ins for my Media Center. I'm even going to hack my old Ceiva Digital Picture Frame if it kills me. Of course, all this work will happen after work. Does it make me a better developer or just obsessed? Probably a little of both.

Can this kind of passion (or obsession) be taught? Should it be? I don't know if I'm a craftsman or just a geek. I do know that I'm still having fun, but I still take pride in my work, and I'm learning something new every day. If you don't have a passion for your craft, why are you doing it? I can't imagine walking into my job every day and not finding something to like about it. The basic question is do you care about your craft? Perhaps not about your job, but at least about your craft. James Curran feels this pain:

As a 5:01 Coder, I must protest—I am passionate about my CRAFT, just not about my DAY JOB. (My train ride home is spent on a laptop, working on an open-source project.)

Ultimately it's about finding your passion—and figuring out how to keep it.

Scott Hanselman works for the Developer Division of Microsoft as the DevDiv Community Liaison and aims to spread good information about developing software in any way he can. He blogs at www.hanselman.com, podcasts at www.hanselminutes.com, and contributes to www.asp.net.