 MSDN Magazine February 2005

MSDN Magazine February 2005

Office: Relive the Moment by Searching Your IM Logs with Custom Research Services

Often, IM conversations contain important information you'd like to keep and reuse. Fortunately, MSN Messenger 6.2 has a feature to keep a conversation history permanently in XML format. This article shows you how to leverage that conversation history by consolidating IM exchanges so they are indexed, searchable, and ultimately reusable using the Microsoft Office 2003 Research and Reference task pane.

John R. Durant Excel: Integrate Far-Flung Data into Your Spreadsheets with the Help of Web Services

Excel 2003 lets you dynamically integrate the data provided by different Web services. It also lets you take advantage of the latest capabilities in Office 2003 to customize list views, graphs, and charts, and to catalog bulk items online or offline. Find out how you can makle the most of the data returned from your Web services with the Office 2003 Web Services Toolkit API.

Alok Mehta SharePoint: Add a Recycle Bin to Windows SharePoint Services for Easy Document Recovery

Windows SharePoint Services helps improve collaboration and workflow while protecting documents and intellectual property. But one feature is conspicuously missing - an easy way to back up and restore deleted files from document libraries. In this article, the authors take advantage of the extensibility of WSS and its server-side and client-side object models to build a restore feature that works like the Recycle Bin in Windows Explorer.

Maxim V. Karpov and Eric Schoonover Smart Tags: Realize the Potential of Office 2003 by Creating Smart Tags in Managed Code

While you may well be excited about the prospect of building managed smart tags, there is little information available to help you create them using .NET. In this article the author fills in the blanks. Along the way he discusses the Microsoft Office Smart Tag List XML schema, advanced managed smart tags for Office 2003 and Office XP, and deploying these features in an organization.

Ben Waldron SQL Server: Display Your Data Your Way with Custom Renderers for Reporting Services

SQL Server 2005 Reporting Servicesis a great tool that offers a centralized approach to storing and rendering reports. It also lets users view and download reports without installing additional software. Plus, reports can be saved in any number of different formats using custom report renderers. In this article, the author will develop one such report renderer that outputs HTML reports, but the skills you'll learn can easily be used to create a renderer for Microsoft Word documents or any other format of your choosing.

James Yip Testing: Get Your Customers Involved in the Testing Process with Functional Tests in Excel

For specification documents to be truly valuable, they need to give an accurate picture of all the requirements of a project. This article describes how the communication value of specification documents can be improved by permitting users to test the code under construction using the Framework for Integrated Test (FIT), an open-source tool. It also explains how you can build a Windows Forms application in C# (WinFITRunnerLite) that converts functional tests, as written by your customers using Excel, into a form that allows you to run them with FIT against the code you're developing.

Will Stott Editor's Note: C++, Plus...

How long have you been reading MSDN Magazine? Did you start picking it up five years ago? Ten years ago when we were Microsoft Systems Journal? Even longer? If you've been reading our publication for a decade or more, you've seen a lot of changes in what's considered "cutting edge" development.

New Stuff: Resources for Your Developer Toolbox

Registry Medic 3.0, released by Iomatic, allows you to clean up your registry, improves PC performance, protects your machine from Trojans, and gets rid of those annoying start-up program artifacts that can be so hard to remove.

Marnie Hutcheson Web Q&A: ActiveX Privileges, Making Icon Files, Sticky Sessions, and More

Edited by Nancy Michell Inside MSDN: Designing URLs for MSDN2

This is the first installment of a new column about MSDN® projects: what we're doing, how we're doing it, and what we're learning along the way. It will be written by MSDN staff with the goal of sharing the team's experiences in solving the real-world business problems MSDN faces.

Tim Ewald Basic Instincts: Using the ReaderWriterLock Class

In several installments over the past year I have written about multithreading programming techniques in Visual Basic® . NET. In the September 2004 issue of MSDN®Magazine, I discussed the need for thread synchronization and how to write thread-safe code using monitors.

Ted Pattison Data Points: ADO.NET and System.Transactions

The Microsoft® . NET Framework versions 1. 0 and 1. 1 represented major changes in software development. However, one important thing that did not change much was support for distributed transactions.

John Papa Cutting Edge: Adding a Context Menu to ASP.NET Controls

Although the context menu is a common element of most desktop applications, it is still fairly uncommon in Web application names because it doesn't map well to a server-based technology like ASP. NET.

Dino Esposito Wicked Code: Five Undiscovered Features on ASP.NET 2.0

By now, developers everywhere have had the opportunity to download the first beta of the Microsoft® . NET Framework 2. 0. ASP. NET developers who have played with it are no doubt salivating at all the cool new features.

Jeff Prosise Service Station: SOA: More Integration, Less Renovation

Since the focus of this column is service-oriented architecture (SOA), I thought now might be a good time to step back and take a wide-angle look at the general concept and what it means to developers.

Aaron Skonnard .NET Matters: File Copy Progress, Custom Thread Pools

Stephen Toub Pure C++: Hello, C++/CLI

Welcome to my first column for MSDN®Magazine. I had actually written a completely different first column—one on the new generic programming support in Visual Studio® 2005. Reviewing it, though, I realized that it left far too many unanswered questions.

Stanley B. Lippman C++ at Work: Persisting View State Update, Using Managed Extensions in a DLL

This month marks the 11th anniversary of my column and the inauguration of a new title: C++ At Work. We're also adding a new bimonthly column, "Pure C++," by my pal Stan Lippman, one of the great C++ Masters.

Paul DiLascia { End Bracket }: C# and VBA: Like Oil and Water

Some things just don't mix as well as you would like. Take C# and Microsoft® Excel 2003 or Word 2003, for example. Not only are these applications huge productivity tools, but they both also provide access to large object models that you can program against from your own applications.

Ken Getz


Office: Relive the Moment by Searching Your IM Logs with Custom Research Services

Office

Relive the Moment by Searching Your IM Logs with Custom Research Services

John R. Durant
This article discusses:
  • Using Research Services to search IMs
  • Storing and indexing IMs
  • The Research Services Development Extras extensions
  • Fast, easy Research Service development with Visual Studio
This article uses the following technologies:
Office 2003, SQL, Visual Basic
Code download available at: ResearchServices.exe (158KB)
Contents
Building the Solution
The Windows Service
Querying via the Research Service
Modifications to the Solution
Conclusion

I
f you are like me, you communicate regularly via instant messaging (IM). Over time, my IM conversations have gone from tentative trials to vital missives that are an inexorable part of my day-to-day work. Often, my IM conversations contain important information that I want to keep and possibly reuse. Fortunately, MSN® Messenger 6.2 has a feature to keep a conversation history permanently in XML format.

This article shows you how to leverage that conversation history further by consolidating IM exchanges so they are indexed, searchable, and ultimately reusable using the Microsoft® Office 2003 Research and Reference task pane. For example, when writing in Microsoft Word or using Word as your e-mail editor, you can quickly search your conversation history for messages that contain a term you have typed in your document. Using smart tag technology in the task pane allows you to insert the text of IM conversations directly into your document.

One of the strengths of this solution is that it takes advantage of the existing capabilities of both MSN Messenger and Office 2003. For the Office side of things, the solution uses the Research task pane which is a new feature in Office 2003. The Research task pane allows users to quickly locate and use the information that they need without leaving the Office application that they are working in. It is available in the 2003 versions of Word, Excel, Outlook®, PowerPoint®, Publisher, and Visio®. You can also see the pane when working in Microsoft Internet Explorer. Because the Research feature interface is tightly integrated with these Office 2003 applications, it can both extract information from the Office application and insert content into it. In the solution we'll explore here, you will see that using the Research task pane, users can search the store of IM conversations and insert conversation text into the document a user is modifying.

Interacting with your solution using smart tag actions and the Research task pane is actually only a part of the overall solution. What sets the sample for this article apart is that it uses a scalable framework that allows for full-text indexing and greater extensibility. While you could wire up the Research task pane directly to the folder where MSN Messenger stores your conversation history, this has some distinct disadvantages. For example, MSN Messenger stores the conversation history in simple XML files, one file per contact. In order to search in these files, you would have to use an XML parser to sift through the files looking for the search term. With only a few short files, this is not a problem, but as the number of files grows and as their content lengthens, such an architecture yields ever worsening performance. Also, it makes your IM content available only as long as the files exist. What if the machine becomes disabled for some reason? The information did not instantly become useless—its value endures. These and other reasons led me to design a solution with a simple framework for central IM conversation management.

Building the Solution

Think of this framework in terms of a few main processes: data collection via conversation harvesting, conversation indexing and searching, and conversation repurposing in the task pane (see Figure 1). The first process, that of data collection, describes the action of gathering up conversations and sending them to a repository. The harvesting should occur when new conversations are created or when existing ones are updated. The second process involves indexing the harvested conversations so that different applications can search their contents and retrieve results. The final process involves presenting the retrieved results and allowing the user to repurpose the content in some way.

Figure 1 Overall Solution Architecture
Figure 1 Overall Solution Architecture

This solution uses a Windows® service to monitor the directory where MSN Messenger stores the conversation history files. As MSN Messenger creates new files or appends existing XML files, the service updates a SQL Server™ 2000 database with the new content. SQL Server is a greater indexer, and its query processor is also quite good. Once in the database, the conversation content is ready for queries and for discovery via a research service that presents the search results in the Research task pane. When the data is present in the pane, the user can select a search result and insert it into the document by executing a smart tag action. The smart tag action code uses the XML features in Word 2003 to insert the conversation text into the document.

Let's explore the solution, starting where it all begins, with MSN Messenger. After signing in to MSN Messenger, open it, choose the Tools menu, and click Options. Click on the Messages tab of the Options dialog. At the bottom, in the Message History section, check the box next to the label, "Automatically keep a history of my conversations." By default, Messenger will keep this history in a subdirectory of your My Documents folder.

You can change this to another location if desired. Remember where you put it, though, because you need to configure your Windows service to monitor the files in this directory. The XML content of a conversation history file is structured such that each conversation is assigned a unique identifier in the file. MSN Messenger also tracks when the conversation occurred as well as each message sent between the two parties in the conversation. The contents of a typical file are shown in Figure 2.

Figure 2 An MSN Messenger File (Close Figure 2)
<?xml version="1.0" ?>
<?xml-stylesheet type='text/xsl' href='MessageLog.xsl'?>
<Log FirstSessionID="1" LastSessionID="2">
    <Message Date="12/5/2004" Time="10:49:42 AM"
     DateTime="2004-12-05T17:49:42.880Z" SessionID="1">
        <From>
            <User FriendlyName="email_of_sender" />
        </From>
        <To>
            <User FriendlyName="John R. Durant" />
        </To>
        <Text Style="font-family:MS Shell Dlg;
         color:#000000;">I love reading MSDN Magazine.</Text>
    </Message>
    <Message Date="12/5/2004" Time="10:50:45 AM"
     DateTime="2004-12-05T17:50:45.441Z" SessionID="2">
        <From>
            <User FriendlyName="email_of_sender" />
        </From>
        <To>
            <User FriendlyName="John R. Durant" />
        </To>
        <Text Style="font-family:MS Shell Dlg;
         color:#000000;">Yeah, me too!</Text>
    </Message>
</Log>

The Windows service monitors the target directory, and when files are added or when new conversations are added to existing files, it extracts the new information and loads it into the database.

The Windows Service

The Windows service conveniently uses the classes in the .NET Framework to monitor the directory configured in the MSN Messenger UI. To be effective, the service needs to respond to update the database when a file is added to the directory or when an existing file is updated. To monitor the directory, it uses the FileSystemWatcher class in the System.IO namespace. Anyone still wondering if they ought to learn how to use the .NET Framework can now consider the matter resolved. Classes like these, and there are many, are what make the libraries in the .NET Framework so compelling. By declaring an instance of this class and registering with its events, you can easily detect when new files are created, deleted, changed, or renamed. Our code responds to only two of those events, and the event procedures both call the same custom procedure. The event code for the Windows service with the event procedures is shown in Figure 3.

Figure 3 Handling FileSystemWatcher Events (Close Figure 3)
Imports System.IO
Imports System.Web
Imports System.Xml
Imports System.Net
Imports System.Data
Imports System.Threading
Imports System.Configuration
Imports System.ServiceProcess

Public Class Service1
  Inherits System.ServiceProcess.ServiceBase

    Public fsw As New FileSystemWatcher

    Protected Overrides Sub OnStart(ByVal args() As String)
        fsw.Path = CType(ConfigurationSettings.AppSettings( _
            "MessengerSource"), String)
      AddHandler fsw.Changed, AddressOf ProcessUpdatedFile
      AddHandler fsw.Created, AddressOf ProcessUpdatedFile
      fsw.EnableRaisingEvents = True
  End Sub

  Public Sub UpdatedFile(ByVal s As Object, _
    ByVal e As FileSystemEventArgs)
      UploadData(e.FullPath)
  End Sub

The Windows service's custom procedure, UploadData, opens the target XML file (either added or changed) and sends its contents to the central SQL Server database. Figure 4 shows the code that performs this data handoff.

Figure 4 Uploading Conversations to SQL Server (Close Figure 4)
Private Sub UploadData(ByVal fullFilename As String)

    Dim CONNECT_STRING As String = CType( _
         ConfigurationSettings.AppSettings("ConnStr"), String)

    Dim chatDoc As XmlDocument = New XmlDocument
    Dim chatNode As XmlNode
    Dim SessionID As String
    Dim chatNodes As XmlNodeList

    Dim cn As SqlClient.SqlConnection = _
        New SqlClient.SqlConnection(CONNECT_STRING)
    Dim cmd As SqlClient.SqlCommand = _
        New SqlClient.SqlCommand("sp_AddData", cn)
    cmd.CommandType = CommandType.StoredProcedure

    Dim p_FromName As New SqlClient.SqlParameter
    p_FromName.ParameterName = "@FromName"
    p_FromName.Direction = ParameterDirection.Input
    p_FromName.SqlDbType = SqlDbType.VarChar
    p_FromName.Size = 50

    Dim p_MessageDate As New SqlClient.SqlParameter
    p_MessageDate.ParameterName = "@MessageDate"
    p_MessageDate.Direction = ParameterDirection.Input
    p_MessageDate.SqlDbType = SqlDbType.DateTime

    Dim p_ToName As New SqlClient.SqlParameter
    p_ToName.ParameterName = "@ToName"
    p_ToName.Direction = ParameterDirection.Input
    p_ToName.SqlDbType = SqlDbType.VarChar
    p_ToName.Size = 50

    Dim p_SessionID As New SqlClient.SqlParameter
    p_SessionID.ParameterName = "@SessionID"
    p_SessionID.Direction = ParameterDirection.Input
    p_SessionID.SqlDbType = SqlDbType.SmallInt

    Dim p_MessageText As New SqlClient.SqlParameter
    p_MessageText.ParameterName = "@MessageText"
    p_MessageText.Direction = ParameterDirection.Input
    p_MessageText.SqlDbType = SqlDbType.VarChar
    p_MessageText.Size = 255

    cmd.Parameters.Add(p_FromName)
    cmd.Parameters.Add(p_MessageDate)
    cmd.Parameters.Add(p_ToName)
    cmd.Parameters.Add(p_SessionID)
    cmd.Parameters.Add(p_MessageText)

    chatDoc.Load(fullFilename)
    SessionID = chatDoc.ChildNodes(2).Attributes(1).Value
    chatNodes = chatDoc.SelectNodes( _
        "//Message[@SessionID='" & SessionID & "']")

    Try
        cn.Open()
        For Each chatNode In chatNodes
            p_FromName.Value = chatNode.ChildNodes(0). _
                ChildNodes(0).Attributes(0).Value
            p_MessageDate.Value = chatNode.Attributes(0).Value _
                & " " & chatNode.Attributes(1).Value
            p_ToName.Value = chatNode.ChildNodes(1). _
                ChildNodes(0).Attributes(0).Value
            p_SessionID.Value = SessionID
            p_MessageText.Value = chatNode.ChildNodes(2).InnerXml
            cmd.ExecuteNonQuery()
        Next
    Finally
        cn.Close()
    End Try
End Sub

A good portion of the code here is focused on setting up the database connection and getting the stored procedure ready to execute. The code walks the structure of XML nodes and attributes to extract the desired information so that it can assign the proper values to the parameters passed to the stored procedure.

Querying via the Research Service

Once the database stores the conversation content, another process can query and extract the contents for further use. The primary consumer of the SQL Server data is a custom Web service that is a research service provider in Office 2003. This custom Web service handles queries from Office 2003 and then queries the SQL Server database for the target search term. It formats responses as item lists in the Research task pane with each additional response marked up with smart tag type information. The real work, then, of the research service is simply to query SQL Server and assemble the response so that it conforms to the research service QueryResponse XML schema.

This is the most important aspect of this part of the overall solution. Research services are Web services with two important methods: Registration and Query. The Registration method sends back important information when a user registers the new research provider. The registration request, sent by the Office 2003 application, conforms to the Registration.Request schema. Responses, on the other hand, need to conform to the Registration.Response schema. Once a research provider is registered, it can field requests that conform to the Query schema and send responses conforming to the Query.Response schema. Office has built-in awareness of these schemas, so nothing needs to be done on the client. All of the work is done in the Web service.

The structure of responses requires the assembly of XML nodes, elements, and attributes, and this can become very tedious. It is much easier using the Research Services Development Extras (RSDE) recently published on MSDN® online. You can find them at the MSDN Office Developer Center at Office 2003 Toolkit: Research Service Development Extras. For more information, read my article introducing research service technology at The Definitive "Hello World" Managed Smart Document Tutorial.

The RSDE contains, among other things, some managed classes that let developers write registration and query responses using an object model rather than using a schema directly. The classes in the object model handle streaming out responses that correspond to the research services schemas for you.

For example, the code in Figure 5 shows the output for returning paragraph text to the task pane along with an action that will allow a user to insert text into the document quickly.

Figure 5 Response Text (Close Figure 5)
<ResponsePacket revision="1" xmlns="urn:Microsoft.Search.Response">
  <Response domain="{96293571-B605-4fd0-9C6A-7202AA689967}">
    <QueryId />
    <Range>
      <StartAt />
      <Results>
        <Content xmlns="urn:Microsoft.Search.Response.Content">
          <P>
            <Char xmlns="urn:Microsoft.Search.Response.Content">
              <Char bold="true">Text to insert: </Char>
              This is text we can insert from the task pane.</Char>
            <Actions defaultAllowed="false">
              <Text>Actions</Text>
              <Insert>
                <Text>Insert</Text>
              </Insert>
              <Copy>
                <Text>Copy</Text>
              </Copy>
            </Actions>
          </P>
        </Content>
      </Results>
    </Range>
    <Status>SUCCESS</Status>
  </Response>
</ResponsePacket>

As long as Office 2003 receives this XML as the response, it doesn't care where it comes from. In the simplest implementation, you could use a StringBuilder object instance to create the string and send it back to the calling program. Or, you could use a couple of statements, shown in Figure 6, to do the same thing using RSDE.

Figure 6 Using RSDE (Close Figure 6)
Public Overloads Overrides Function Query( _
        ByVal request As Query.QueryRequest) As Query.QueryResponse
    Dim response As New Query.QueryResponse
    Dim content As New Query.ContentResponseWriter
    Dim crw As New Query.ContentResponseWriter

    Select Case request.QueryText.ToLower()
      Case "test"
        content.WriteParagraph("<Char bold=""true""> Text to insert:" _
          & "</Char>This is text we can insert from the task pane.",
             True)
    End Select

    response.WriteResponse(content)
    Return response
End Function

Notice that most of the work is done in one statement, namely, the one that executes the WriteParagraph method. Gratefully, the managed classes in RSDE dramatically simplify the amount of code you need to write, and they greatly clarify the meaning of the code you do write. If you are new to research providers creation, these classes will get you up and running without having to search through the schema reference for research services.

Another benefit of the RSDE is that it includes a template for creating the Web service project in Visual Studio® .NET 2003. While you can create your own research provider from scratch by creating a generic Web service in Visual Studio, you will need to put a lot of the underlying pieces together on your own. As before, the RSDE makes this much simpler.

For example, once you start creating a project from the template, the RSDE launches a wizard that collects the information to display when a user registers your service. It also makes it simple to generate the GUID that uniquely identifies the service. You can easily include a EULA and copyright information. Finally, it generates the custom classes with the registration method already coded. For most research services, creating them using the wizard eliminates the need to learn anything about the registration schema or process because it's all taken care of for you.

That said, in the long run, there is no substitute for actually knowing the registration and query schemas. The wrapper classes take care of a great deal of the busywork when creating a Web service that delivers Research task pane functionality, but your service will become more polished and refined as you gain a deeper knowledge of the schemas and combine it with the power of the RSDE classes. You can learn more about the research services schemas by checking out the Research Services SDK on the MSDN Developer Center.

Obviously, the research provider for this solution does more than just display some hardcoded text. In this solution, the Web service needs to take the target search term and find out if any of the harvested messages contains it. A word needs to be said about the SQL Server that contains the database. To achieve the greatest search power, you should install full-text indexing on the database server. This allows you to perform more powerful queries against the data. For example, without it you can use a keyword such as LIKE with wildcard characters. However, this is not really the best way to do your searching. Full-text indexing allows you to use more powerful keywords such as CONTAINS and FREETEXT.

The design of the table that contains the data is shown in Figure 7. As you can see, it has these columns.

Figure 7 Table Design (Close Figure 7)

Column Name | Data Type | Length
MessageID | int | 4
FromName | varchar | 50
MessageDate | datetime | 8
ToName | varchar | 50
SessionID | smallint | 2
MessageText | varchar | 255

This solution requires a full-text index only on the MessageText column. Building a full-text index on this column lets you use the CONTAINS keyword to find any entries that contain the search word, inflectional forms of the word and so much more. The stored procedure for querying the table looks like this:

CREATE PROC sp_SearchData
    @SearchTerm varchar(50)
AS
SELECT FromName, MessageText
FROM   MessageLog
WHERE  CONTAINS(MessageText, 'FORMSOF(INFLECTIONAL,@SearchTerm)')
This search is powerful because it will pick up inflectional forms of the word anywhere in the MessageText column of the table. For example, if the search term is the word "run", the query will return results that include other forms of the word such as "ran", "running," or "runs". Obviously, the chances are that you will be searching for something more specific than this verb, but you can trust SQL Server to ably handle your query.

The research service receives a query request from the Research task pane, which triggers the query to the database. The request is fielded by the code in the Query WebMethod. The sample, shown in Figure 8, queries the database and displays message results in a list with a smart tag action that allows the user to insert the text into the document.

Figure 8 Searching a Database from a Research Service (Close Figure 8)
Public Overloads Overrides Function Query( _
    ByVal request As Query.QueryRequest) As Query.QueryResponse

    Dim response As New Query.QueryResponse
    Dim content As New Query.ContentResponseWriter
    Dim crw As New Query.ContentResponseWriter

      Dim CONNECT_STRING As String = CType( _
         ConfigurationSettings.AppSettings("ConnStr"), String)
      Dim cn As SqlClient.SqlConnection = _
          New SqlClient.SqlConnection(CONNECT_STRING)
      Try
          cn.Open()
          Dim cmd As SqlClient.SqlCommand = _
              New SqlClient.SqlCommand("sp_SearchData", cn)
          cmd.CommandType = CommandType.StoredProcedure

          Dim p_SearchTerm As New SqlClient.SqlParameter( _
              "@SearchTerm", request.QueryText)
          p_SearchTerm.Direction = ParameterDirection.Input
          p_SearchTerm.SqlDbType = SqlDbType.VarChar
          p_SearchTerm.Size = 50
          cmd.Parameters.Add(p_SearchTerm)

          Dim rdr As SqlClient.SqlDataReader = cmd.ExecuteReader()
          While rdr.Read
              content.WriteHeading("Message on " _
                  & rdr.Item("MessageDate"), False, False, crw)
              content.WriteParagraph("<Char bold=""true"">" _
                  & rdr.Item("FromName") & "- </Char>" _
                  & rdr.Item("MessageText"), True)
              content.WriteHorizontalLineSeparator()
          End While
      Catch exc As SqlException
        'Handle the error by posting to the Windows event log, and so on.
      Finally
          cn.Close()
      End Try

      response.WriteResponse(content)
      Return response
  End Function

Using the RSDE wrapper makes it so easy to present the data in the task pane with the right formatting as well as with the appropriate accompanying actions. For example, the WriteParagraph method has two parameters, paragraphText and displayInsertCopyButton. The first parameter, quite sensibly, is the text that you want to display. The second parameter, a Boolean value, allows you to include an action menu that lets the user take action to insert or copy the displayed text. The final display, the task pane shown in Figure 9, lists each of the messages in conversations that included the search term along with an action menu to insert or copy the text.

Figure 9 Research Task Pane
Figure 9 Research Task Pane

The display here is only a simple example of the many varieties of things you can do in the Research task pane. Again, the best way to get going is to use the RSDE, but you should also take the time to study the research service schemas in the Research Services SDK.

Modifications to the Solution

Just when you thought it couldn't get easier to make a research service, things get even simpler. When using the RSDE template in Visual Studio, the wizard presents you with an important choice. Because many research services work with data from multiple sources, the wizard lets you choose the kind of provider you want to connect to (see Figure 10). Simply specify your connection string and an SQL statement or the name of a stored procedure you want to execute when queries come through the service.

Figure 10 Building a Data Layer in Your Research Service
Figure 10 Building a Data Layer in Your Research Service

The next dialog in the wizard lets you specify how you want results presented in the Research task pane and how many results you want displayed at a time. After the wizard completes, you will find that it has created an entire data layer for you in the Web service project. The task pane shown in Figure 11 indicates how a research service created using this technique presents the results. Using the same stored procedure and data as in the custom sample, this new research service worked without writing a single line of code! It worked with no modification or intervention whatsoever. Of course, you will want to modify the code and tailor it to your needs, but the power of the RSDE is astonishing.

Figure 11 Research Results
Figure 11 Research Results

One of the modifications you will want to explore as you develop more research services is the addition of more powerful smart tag actions in the pane. When the research service packs up the results with the schema-based markup, you can assign smart tag type information to results so that registered smart tag action handlers on the client can present the user with actions assigned to that smart tag type. For more information about smart tags and how to create custom smart tag action handlers, see Ben Waldron's article on developing smart tags for Office 2003 in this issue of MSDN Magazine. You can also visit Smart Tags and Smart Documents for more ideas.

You can create a custom DLL with powerful action handlers associated with your own custom smart tag type names. The Research task pane has been designed to sense this type information and let Office know that the marked up content should be presented with the appropriate smart tag actions menu. The code you put in your smart tag actions DLL will fire when a user clicks a given menu item. The insert and copy menu items you see in Figure 8 are built into Office. Yours will be different because they are your own invention. What you do when a user clicks a smart tag action item is up to you. Your action handler can connect to the Web or a database, perform file operations, create presentations, insert hidden text, or pretty much any other thing you can do in a DLL.

Conclusion

It is becoming increasingly common for users to expand their authoring repertoire in Microsoft Office. Research services are a great way to simplify user tasks, reduce manual operations like copying or inserting text, and bring more information to users while they are working with documents. This article uses instant message logs as the data source, but you can choose virtually any other source. Leveraging the research service schemas, you can determine how search results are presented to the user in the task pane. To make things easier, you can use the Research Service Development Extras to save you some of the grunt work that is required to get a research service up and running. From there, you can modify the service to meet the needs of your organization.

John R. Durant manages the MSDN Office Developer Center and is a frequent author and speaker on a wide variety of development topics. He co-authored the latest edition of The XML Programming Bible (Wiley & Sons, 2003).

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Excel: Integrate Far-Flung Data into Your Spreadsheets with the Help of Web Services

Excel

Integrate Far-Flung Data into Your Spreadsheets with the Help of Web Services

Alok Mehta
This article discusses:
  • How you can integrate data from several Web services using Excel and VBA
  • The features of the Office Web Services Toolkit
  • How you can manipulate your data in Excel and use it to create reports
  • A sample application to get you started
This article uses the following technologies:
Office 2003, Web Services, Visual Basic
Code download available at: ExcelWebServices.exe (251KB)
Contents
The Toolkit
An Academic Scenario
ASP.NET Web Services Server Setup
VBA Client Setup
Web Service Security
Wrap-Up

M
icrosoft® Office 2003 contains powerful XML-based functionality, including Web services integration. For example, using the Web services capabilities of Office 2003, developers can create a variety of solutions that help users increase their productivity on eBay. Web services offered by eBay allow you to use Excel to manage dynamic listing data, such as pricing and bids, and update this data automatically as changes occur. Solutions could also enable users to list multiple items on eBay simultaneously or to create a Web site that displays their eBay item listings.

This is all made possible with the Office 2003 Web Services Toolkit, which minimizes the programming required to integrate data returned from Web services with Visual Basic® for Applications (VBA). Office 2003 and Web services together offer a platform for building robust solutions to manage useful applications. You can download this tool from Microsoft Office 2003 Web Services Toolkit 2.01.

Among the applications within Office, Excel 2003 in particular allows developers to dynamically integrate the data provided by Web services. It also allows you to take advantage of the latest capabilities in Office 2003 to customize list views, graphs, and charts, and to catalog bulk items online or offline. The Web service integration within Excel 2003 can also provide ad hoc data analysis in the familiar environment of Excel, such as statistical analysis of the data extracted from the Web service.

This article describes how the Office 2003 Web Services Toolkit provides an API that allows you to select the Web services you want and create proxy classes for your VBA. The tool is accessible from the Web Service References command on the Tools menu in the Office Visual Basic Editor after the toolkit has been installed. By using the Office Visual Basic Editor, programmers can now set references to XML Web services just as they would any intrinsic type library. The VBA proxy classes generated by the tool allow both early binding and IntelliSense®, which makes consuming XML Web services easier and more intuitive, and virtually eliminates syntax errors.

The Toolkit

The Office 2003 Web Services Toolkit allows you to quickly access and use Web services with VBA from Office 2003 applications. It allows you to use a GUI to either search for Web services from any Universal Description Discovery and Integration (UDDI) registry or to specify a particular Web service by providing a URL to the file that contains the Web service description. The target file can be in Web Services Description Language (WSDL) format or the Visual Studio® Discovery (VSDisco) format, which is still supported even though it's obsolete. In case the description file contains information on multiple Web services, the GUI allows the user to select either some or all of the Web services for use within their current document.

The tool then generates the code for classes that access the Web services using the Microsoft SOAP client object. These generated classes will have methods corresponding to the operations supported by the Web services. The user can write simple VBA scripts without having to know that the objects she is using are really proxies to Web services. The toolkit supports Office SOAP 3.0 and requires MSXML 3.0.

The Office 2003 Web Services Toolkit also includes the Web Services References Tool, which allows you to locate Web services. The best part of this toolkit is that it automatically generates code for you. I'll discuss the automatic code generation listings shortly.

An Academic Scenario

To illustrate the power of using ASP.NET Web services in VBA, I have developed a simple example that includes both a sample Web service as well as a spreadsheet that uses it. The purpose of the Web service is to retrieve students' grades for a given professor and semester. This example uses a database with a simple data model (see Figure 1). The general idea is as follows: a professor teaches a course with many students in a given semester. Students are graded on quizzes, homework, a mid-term exam, class participation, and a final exam. Many times, professors want to analyze student performance in a particular semester, looking at a student's average grades and other statistics. By exposing this information through a Web service, professors could consume that data in Excel 2003 and use it to calculate a variety of interesting statistics.

Figure 1 Student Grades Data Model
Figure 1 Student Grades Data Model

The sample is divided into the following three parts: the server-based ASP.NET Web service, called Get_Grades; the client-side VBA in a spreadsheet, called Client.XLS, which was developed in Excel 2003; and the toolkit code that runs on the client side. This code is automatically generated by the toolkit and provides a proxy to the ASP.NET Web service by wrapping calls to it.

ASP.NET Web Services Server Setup

To create the Get_Grades Web service, you first need to create a Web service in ASP.NET and add the code from Figure 2. Start Visual Studio .NET and open a new ASP.NET Web service project. Name the Web service project Get_Grades. If your machine has IIS installed, ASP.NET will automatically create the Web service for it. In addition, ASP.NET creates several files. Among the most important is Service1.asmx.vb, which will contain the public method Get_Grades available via the Web services. Simply cut and paste the source code in Figure 2 into Service1.asmx.vb. I'll describe the Web service source code in the next section. For now, let's work on building the solution.

Figure 2 Sample Web Service (Close Figure 2)
'Alok Mehta
'February 2005
'MSDN VBA WebService Example

Option Explicit On

Imports System
Imports System.Data
Imports System.Data.SqlClient
Imports System.Web.Services

<WebService(Namespace:="http://myCollege.edu/")> _
Public Class Service
    Inherits System.Web.Services.WebService

    'Provide grades for all homework, exams, and so on. Based on
    'professor and course
    <WebMethod()> Public Function Get_Grades( _
        ByVal nProfessorID As Long, ByVal lngCourseID As Long) As String

        'Make sure its a valid request
        If bValidUser(nProfessorID, lngCourseID) Then
            'Dataset to store the results
            Dim GradesDataSet As DataSet = New DataSet()

            'Make connection to the Grades database; the connection
            'string would be in a config file in real application
            Dim GradesConn As SqlConnection = New SqlConnection( _
                "Data Source=localhost;Integrated Security=SSPI;" & _
                "Initial Catalog=Grades")
            Try
                'SQL to get the students grades for a particular
                'professor and course
                Dim strSQL As String
                strSQL = "SELECT tbl_Student.Student_Name, " & _
                    "tbl_Grades.Quiz_Grade, tbl_Grades.HomeWork, " & _
                    "tbl_Grades.MidTerm, tbl_Grades.Participation, " & _
                    "tbl_Grades.FinalExam FROM tbl_Student INNER JOIN " & _
                    "(tbl_Professor INNER JOIN (tbl_Course INNER JOIN " & _
                    "tbl_Grades ON " & _
                  "tbl_Course.Course_ID=tbl_Grades.Course_ID) ON " & _
                  "tbl_Professor.Professor_ID=tbl_Grades.Professor_ID) " &_
                  "ON tbl_Student.Student_ID=tbl_Grades.Student_ID " & _
                  "WHERE tbl_Professor.Professor_ID=@profID AND " & _
                  "tbl_Course.Course_ID=@courseID"

                'Create the command to execute
                Dim Comm as SqlCommand = New SqlCommand(strSQL, GradesConn)
                Comm.Parameters.Add("@profID", nProfessorID)
                Comm.Parameters.Add("@courseID", lngCourseID)

                'Use a data adapter to fill the DataSet, then get as XML
                Dim dataGrades As SqlDataAdapter = New SqlDataAdapter(Comm)
                dataGrades.Fill(GradesDataSet)
                Get_Grades = GradesDataSet.GetXml
            Catch e As Exception
                'Return any error messages
                Get_Grades = e.Message
            Finally
                GradesConn.Close()
            End Try
        Else
            'In case its not a valid user
            Get_Grades = "Invalid user"
        End If
    End Function

    Private Function bValidUser(ByVal nProfessorid, ByVal lngCourseid) _
                                As Boolean
        'This function should be replaced by your security standards
        'such as a Web service token etc. For simplicity we just allow
        'a specific professor ID and course ID
        If nProfessorid = 1 And lngCourseid = 1 Then
            bValidUser = True
        Else
            bValidUser = False
        End If
    End Function

End Class

VBA Client Setup

To set up Client.XLS on the developer's machine, copy it from the code download to any folder you prefer. In Excel, use the Visual Basic Editor to look at the code attached to the XLS file. You should see cmdGetDataFromWebService_Click within the WebServiceClient Sheet object. The code for this is shown in Figure 3. The class module, clsws_Get_Grades, is shown in Figure 4.

Figure 4 Auto-Generated Client Proxy (Close Figure 4)
'*****************************************************************
'This class was created by the Microsoft Office 2003 Web Services Toolkit.
'
'Description:
'This class is a Visual Basic for Applications class representation of the
'Web service as defined by http://localhost/grades/WebService.asmx?wsdl.
'
'To Use:
'Dimension a variable as new clsws_Service, and then write code to
'use the methods provided by the class.
'Example:
' Dim ExampleVar as New clsws_Service
' debug.print ExampleVar.wsm_Get_Grades("Sample Input")
'
'For more information, see Complex Types in Microsoft Office 2003
'Web Services Toolkit Help.
'
'Changes to the code in this class may result in incorrect behavior.
'
'*****************************************************************

'Dimensioning private class variables.
Private sc_Service As SoapClient30
Private Const c_WSDL_URL As String = _
    "http://localhost/grades/WebService.asmx?wsdl"
Private Const c_SERVICE As String = "Service"
Private Const c_PORT As String = "ServiceSoap"
Private Const c_SERVICE_NAMESPACE As String = "http://myCollege.edu/"

Private Sub Class_Initialize()
    '*****************************************************************
    'This subroutine will be called each time the class is instantiated.
    'Creates sc_ComplexTypes as new SoapClient30, and then
    'initializes sc_ComplexTypes.mssoapinit2 with WSDL file found in
    'http://localhost/grades/WebService.asmx?wsdl.
    '*****************************************************************

    Dim str_WSML As String
    str_WSML = ""

    Set sc_Service = New SoapClient30

    sc_Service.MSSoapInit2 c_WSDL_URL, str_WSML, c_SERVICE, c_PORT, _
        c_SERVICE_NAMESPACE
    'Use the proxy server defined in Internet Explorer's LAN settings by
    'setting ProxyServer to <CURRENT_USER>
    sc_Service.ConnectorProperty("ProxyServer") = "<CURRENT_USER>"
    'Autodetect proxy settings if Internet Explorer is set to autodetect
    'by setting EnableAutoProxy to True
    sc_Service.ConnectorProperty("EnableAutoProxy") = True

End Sub

Private Sub Class_Terminate()
    '*****************************************************************
    'This subroutine will be called each time the class is destructed.
    'Sets sc_ComplexTypes to Nothing.
    '*****************************************************************

    'Error Trap
    On Error GoTo Class_TerminateTrap

    Set sc_Service = Nothing

Exit Sub

Class_TerminateTrap:
    ServiceErrorHandler ("Class_Terminate")
End Sub

Private Sub ServiceErrorHandler(str_Function As String)
    '*****************************************************************
    'This subroutine is the class error handler. It can be called from any
    'class subroutine or function when that subroutine or function
    'encounters an error. Then, it will raise the error along with the
    'name of the calling subroutine or function.
    '*****************************************************************

    'SOAP Error
    If sc_Service.FaultCode <> "" Then
        Err.Raise vbObjectError, str_Function, sc_Service.FaultString
    'Non SOAP Error
    Else
        Err.Raise Err.Number, str_Function, Err.Description
    End If

End Sub

Public Function wsm_Get_Grades(ByVal dcml_nProfessorID As Double, _
    ByVal dcml_lngCourseID As Double) As String
    '*****************************************************************
    'Proxy function created from
    'http://localhost/grades/WebService.asmx?wsdl.
    '
    '"wsm_Get_Grades" is defined as XML. See Complex Types: XML Variables
    'in Microsoft Office 2003 Web Services Toolkit Help for details on
    'implementing XML variables.
    '*****************************************************************

    'Error Trap
    On Error GoTo wsm_Get_GradesTrap

    wsm_Get_Grades = sc_Service.Get_Grades(dcml_nProfessorID,  _
        dcml_lngCourseID)

Exit Function
wsm_Get_GradesTrap:
    ServiceErrorHandler "wsm_Get_Grades"
End Function
Figure 3 Client-Side VBA (Close Figure 3)
'MSDN VBA Excel-WebService Client Example Code
Option Explicit

'WebService Dataset will be temporarily saved in the following XML file
Const XMLFileName = "C:\Grades.XML"

'Professor will be given the following two IDs by the IT department.
Const Course_ID = 1
Const Professor_ID = 1

Private Sub cmdGetDataFromWebService_Click()
    On Error GoTo Error_Processing
    Dim objGrades As New clsws_Service  'Instantiate the Web service proxy
    Dim strReturnValue As String        'XML string from the Web service
    Dim bSaveFile As Boolean            'Saved the XML file
    Dim intI As Integer                 'Local Counters
    Dim intJ As Integer
    Dim intOffset As Integer
    intOffset = 8                       'Starting Row
    Dim strColumn As String             'Spreadsheet Columns
    Dim strValue As String              'Values to be copied into cells
    Dim strFieldName As String          'XML fields mapped to the db fields
    Dim oRoot As MSXML2.IXMLDOMNode     'XML files's root
    Dim NewXMLdocument As New MSXML2.DOMDocument    'XML from the service

    'Get data from Web service
    strReturnValue = objGrades.wsm_Get_Grades(Professor_ID, Course_ID)
    bSaveFile = SaveToXML(XMLFileName, strReturnValue)

    If bSaveFile Then
        'Load the data from the file and iterate over the DOM
        Call NewXMLdocument.Load(XMLFileName)
        Set oRoot = NewXMLdocument.documentElement

        For intI = 0 To oRoot.childNodes.Length - 1
            For intJ = 0 To oRoot.childNodes.Item(intI).childNodes.Length 1

                'Get the value and field name for each node
                strValue = oRoot.childNodes.Item(intI) _
                    .childNodes.Item(intJ).Text
                strFieldName = oRoot.childNodes.Item(intI) _
                    .childNodes.Item(intJ).baseName
                 'Map the database fieldanme to the spreadsheet cells
                 Select Case strFieldName
                        Case "Student_Name":  strColumn = "A"
                        Case "Quiz_Grade":    strColumn = "C"
                        Case "HomeWork":      strColumn = "D"
                        Case "MidTerm":       strColumn = "E"
                        Case "Participation": strColumn = "F"
                        Case "FinalExam":     strColumn = "G"
                 End Select

                 'Set the cell's value based on the XML data
                 ActiveSheet.Cells(intI+intOffset,strColumn).Value=strValue

                 'Keep track of what's happening to aid in debugging
                 Debug.Print strFieldName, strValue
             Next
        Next
        MsgBox "Web Service Data Get was successful"
    End If
    Set objGrades = Nothing    'Release the object
    Exit Sub
Error_Processing:
    MsgBox "Error in Web Service Data Get " & Err.Description
End Sub

Private Function SaveToXML(strValue As String, strFileName As String) _
        As Boolean
    On Error GoTo Error_Processing
    'Saves a string value into a text file using FileSystemObject
    Dim fs As Object
    Dim a
    Set fs = CreateObject("Scripting.FileSystemObject")
    Set a = fs.CreateTextFile(strValue, True)
    a.WriteLine (strFileName)
    a.Close
    SaveToXML = True
    Exit Function
Error_Processing:
    MsgBox "Cannot Save XML File " & Err.Description
    SaveToXML = False
End Function

Assuming that you have installed the toolkit, going to Tools | Web References in the editor should show you the screen that you can see in Figure 5.

Figure 5 Office 2003 Web Services Toolkit in Excel
Figure 5 Office 2003 Web Services Toolkit in Excel

Open the Office Visual Basic Editor in any Office application that supports VBA. Set a reference to the Microsoft Soap Type Library (MSSOAP3.DLL) from the References dialog box (located in the Tools menu). Then, on the Tools menu, choose the Web Service References option.

One of the important items in the toolkit is a Web service search, which lets you search against a UDDI Business Registry using either keywords or a business name. (We will look at UDDI later.) The Web Service URL feature lets you perform an XML Web service search (including wildcards) using a specific URL to a WSDL or Visual Studio .NET Discovery (.vsdisco) file. The Search Results page then allows you to select available XML Web services for which you want the tool to create proxy classes. Finally, the test button allows you to test an XML Web method you select from Search Results (if the XML Web service administrator has provided a corresponding .asmx test harness page).

Once you select one or more Web services for which you want the tool to create proxy classes and click the Add button, the toolkit will create one class module (.cls) file for each selected Web service. Each .cls file will begin with the prefix clsws_ followed by the Web service's name. For instance, if I add a Web service to my project with the name Get_Grades, the class name will be clsws_Get_Grades.

Now, let's go back to the code for the Web service itself. Shown in Figure 2, this code exposes a public Web method called Get_Grades that accepts two parameters: nProfessorID and lngCourseID. Its purpose is to return grades of a particular professor and course in the form of an XML string. Figure 6 shows the Get Grades button.

Figure 6 Get Grades Button
Figure 6 Get Grades Button

The example Web service uses the following four .NET namespaces: System, System.Data, System.SqlClient, and System.WebServices. You can get more information about these namespaces in the MSDN Library.

Since this system is designed for professors only, the bValidUser function was created to validate the professor ID that is being passed to the Web service. Likewise, each semester is identified using a semester ID. The code on the client sets and passes these IDs to the Web service to extract the relevant data from the database so that an analysis can be performed in the client Excel file or application. The bValidUser function can be enhanced to include more complex validation and verification logic by implementing signed security tokens such as X.509 and Kerberos tickets as explained in Security in a Web Services World: A Proposed Architecture and Roadmap. You can also enhance bValidateUser to pull data from your own database to map the professor and course IDs. However, for the purposes of this example, I have simply hardcoded the IDs to be 1 so we can focus on power of Web services and Excel.

I made use of ADO.NET DataSets and their connection properties to extract the data from the SQL Server™ database. GradesConn is the SQL Server connection that defines the connection to the Grades database. GradesDataSet is populated from the database, and the DataSet is then converted to an XML string and is ultimately returned using the GradesDataSet.GetXml property.

Figure 3 is the client-side VBA code. The bulk of the code is contained in the cmdGetDataFromWebService_Click event handler. The purpose of this routine is to get the data in the form of an XML string from the Web service and parse it into the specified cells. The data is retrieved by calling objGrades.wsm_Get_Grades(Professor_ID, Course_ID). Recall that Professor_ID and Course_ID are constants and are assumed to be provided by the IT department of the college. Once the XML string is returned from the Web service, it is saved to disk so you can deserialize the data into a DOM now or at a later date.

Now that you have a valid XML file, called c:\Grades.XML, you can iterate through all of its nodes and their children. As I iterate through each of the children, I can extract the values and field names using oRoot.childNodes.Item(intI).childNodes.Item(intJ).Text and oRoot.childNodes.Item(intI).childNodes.Item(intJ).baseName, respectively. The spreadsheet columns are then filled using ActiveSheet.Cells(intI + intOffset, strColumn).Value, where the row in the active sheet is set by intI + intOffset and strColumn is based on strFieldName. Once the code in Figure 3 is executed, you should see results, as shown in Figure 7.

Figure 7 Data Retrieved from Web Services in Excel 2003
Figure 7 Data Retrieved from Web Services in Excel 2003

Now that we have extracted the data from the server, let's explore Client.XLS. Columns A through G are filled by the VBA code I just discussed. Column I presents the automatically calculated, weighted average of the assessment. It represents the final grade that the professor wants to see.

Since Excel provides an excellent utility for creating statistical functions and charts, I decided to use those functions to convert the data extracted from the Web service into useful information. This information is contained in the Information tab of Client.XLS and is shown in Figure 8. These four useful sets of information are marked as A, B, C, and D. Part A contains simple mean, median, and mode. Part B contains frequency (within 20 points) of the final grades. Parts C and D are graphical representations of the frequency and student grades.

Figure 8 Using Excel and Web Service Data (Close Figure 8)

The code in Figure 4 is automatically created by the toolkit. It includes a VBA class representation of the Web service as defined by http://localhost/grades/WebService.asmx?wsdl. This code is created by the Web service component that comes with Excel 2003, and modifying it may result in unexpected behavior—so don't make any changes to it. The most important function in this class is wsm_Get_Grades, which is the proxy function created from http://localhost/grades/WebService.asmx?wsdl. The function wsm_Get_Grades is defined to return XML data.

Web Service Security

A discussion of Web services security is outside the scope of this article, but it is a very important issue. The MSDN Library hosts several articles that discuss security principles in Web services, such as Security in a Web Services World: A Proposed Architecture and Roadmap. Also, the Patterns & Practices team at Microsoft has published several articles on Web service security, such as Web Services Security. You should also consider issuing signed security tokens such as X.509 and Kerberos tickets to your VBA clients for securing your Web service.

Wrap-Up

In this article, I've demonstrated how to use a powerful and familiar tool, Excel, to access XML data from a Web service using VBA. A stateless Web service can wrap complex business logic in a black box that can then be called using Excel 2003 as a client.

Although I only discussed the Web service and client that are on the same machine, in real applications the Web services will often be hosted on a different server. You can certainly publish your Web services on your own enterprise Web servers and have your users use the Excel VBA client to access the Web service API. Additionally, you can register your Web services on UDDI where other businesses can know about your presence and the Web service APIs. UDDI is a comprehensive industry initiative enabling businesses to define their business, discover other businesses, and share information about how they interact in a global registry. UDDI is the building block that enables businesses to quickly, easily, and dynamically find and transact with one another via VBA. The UDDI standard takes advantage of W3C standards such as XML, HTTP, and DNS protocols to provide service description and discovery. UDDI also addresses cross-platform programming features. You can find a lot of useful information on UDDI at uddi.microsoft.com.

Alok Mehta is senior vice president and chief technology officer of AFS Technologies in Weston, Massachusetts. He recently completed his Ph.D. in Computer Science from Worcester Polytechnic Institute.


SharePoint: Add a Recycle Bin to Windows SharePoint Services for Easy Document Recovery

SharePoint

Add a Recycle Bin to Windows SharePoint Services for Easy Document Recovery

Maxim V. Karpov and Eric Schoonover
This article discusses:
  • WSS object model extensibility
  • Building a custom workflow to support recycle bin functionality
  • Writing custom event handler classes
  • Web Part Page customization
This article uses the following technologies:
Office, C#
Code download available at: RecycleBinforWSS.exe (145KB)
Contents
WSS Events
The Solution
Handling Document Library Events
Configuration
Creating the Sinks
Customization of the Web Part Pages
Conclusion

W
indows® SharePoint® Services (WSS), which is part of Microsoft® SharePoint Products and Services, provides a long list of features that help improve both collaboration and workflow while protecting documents and intellectual property. While WSS features continue to improve with each release, one feature that's conspicuously missing is an easy way to back up and restore deleted files from document libraries.

In this article, we'll take advantage of the extensibility of WSS and both its server-side and client-side object models to build a restore feature that works like the Recycle Bin in Windows Explorer. We'll examine WSS document library server-side events and will develop a .NET event handler class for asynchronous server-side processing of those events. In the end, you'll have a working recycle bin for document libraries. Additionally, the lessons learned will help you write your own custom workflow applications based on the WSS document library event model.

WSS Events

WSS provides object models that allow it to be controlled and extended programmatically. At the core of the WSS server-side object model is the Microsoft.SharePoint.dll assembly, installed with the product and accessible to any code running on the server, regardless of whether that code is executing in a WSS Web context. The core of the client-side object model is found in the OWS.JS JavaScript file. This file is referenced from most pages rendered by WSS and contains the logic necessary to handle various events on the client, such as handling the selection of items from a context menu in the browser.

Figure 1 shows a high-level view of how a client interacts with document libraries in WSS. The main view of a document library is exposed from WSS via the AllItems.aspx endpoint that is located in a document library's directory. All incoming HTTP requests for AllItems.aspx handled on the server by IIS are processed through the SharePoint ISAPI filter, STSFLTR.DLL. This filter checks the Windows registry for exclusion and inclusion paths (configured by the SharePoint admin) to determine if the call should be processed by WSS.

Figure 1 WSS Client-Side Architecture
Figure 1 WSS Client-Side Architecture

If the request is to be handled by WSS, the WSS Web Part Page handler, SharePointHandler, gets the requested content from the content database and sends it to the browser. SharePointHandler resides in the Microsoft.SharePoint.ApplicationRuntime namespace in the Microsoft.SharePoint.dll assembly and is used as the HTTP Handler for .aspx pages rendered through SharePoint. For more information, see the article by Ted Pattison and Jason Masterman in the August 2004 issue of MSDN®Magazine, available at Use Windows SharePoint Services as a Platform for Building Collaborative Apps, Part 2.

Dynamic HTML (DHTML) provides client-side event handling. For example, hovering over a document in a document library and clicking the arrow that appears causes a custom context menu to be displayed in the browser, and this menu provides for common document operations such as delete, check in, and check out. When the user selects the delete menu item from the context menu, a JavaScript event fires, causing an HTTP request to be made to the server, the result of which is that the specified file is deleted and a server-side delete event is raised.

The complete list of server-side document library events, as it exists in the current version of WSS, are as follows: cancel check out, check in, check out, copy, delete, insert, move or rename, and update. If you want to handle server-side events, you must develop a custom event handler class, called an event sink, that can then be registered to handle the events for a particular document library. This configuration can be performed using properties of the document library exposed through the Web administration interface, or it can be done programmatically using the server-side object model exposed by SharePoint.

The WSS document library event pattern is similar to the Observer (Producer-Consumer) pattern. However, while multiple WSS document libraries can be registered as event producers for the same event sink, only one event sink per document library can be notified that events were raised, as shown in Figure 2. This many-to-one relationship means that all event handling logic for a single document library must be encapsulated in a single event sink. Alternatively, if a chain or a set of event sinks is required, a custom event sink could be written to receive the document library's events and then delegate them to other handlers.

Figure 2 Producer-Consumer Pattern
Figure 2 Producer-Consumer Pattern

In order to establish the relationship of producer to consumer, you must set the EventSinkAssembly and EventSinkClass properties of the target document library to point to the event sink. Event sinks are .NET classes that implement the IListEventSink interface and its only method, OnEvent. This method has one SPListEvent input parameter, which contains information about the event that occurred, such as the type of the event (available from the Type property, an SPListEventType), the location of the file before the event (available from UrlBefore), and the location of the file after the event (available from UrlAfter). The assembly containing the event handler class should be strongly named and installed into the Global Assembly Cache (GAC).

Before any document libraries can raise events on the server, WSS must be configured to enable events. The default configuration of a WSS virtual server does not allow events to be raised for document libraries, so you have to explicitly enable this feature before your custom event sink objects will be notified of events. You can do that through the administration Web site, by using the SharePoint command-line administration tool STSADM.EXE, or by using the WSS server-side object model. Enabling event handlers programmatically is simple using code like that shown here:

SPGlobalAdmin globalAdmin = new SPGlobalAdmin();

System.Uri uri = new System.Uri("http://localhost:8080");
SPVirtualServer vServer = globalAdmin.OpenVirtualServer(uri);

SPVirtualServerConfig vConfig = vServer.Config;
vConfig.EventHandlersEnabled = true;
vConfig.Properties.Update();

globalAdmin.Close();

Given this description, you might think of an easy solution for implementing recycle bin functionality. You would simply create a document library and delegate it as the recycle bin. When delete events were raised on other libraries, you would move the deleted documents to the recycle bin library rather than deleting the document. From there the user could restore it at a later time, resulting in the document moving back to the original library, or could empty the bin permanently removing all documents that had previously been deleted. The following code snippet is skeleton code for this proposed solution, where the MoveToRecycleBin method would simply copy the document at EventInfo.UrlBefore to the recycle bin library:

public class CustomEventHandler : IListEventSink
{
    public void OnEvent(Microsoft.SharePoint.SPListEvent e)
    {
        switch (EventInfo.Type)
        {
            case SPListEventType.Delete:
                MoveToRecycleBin();
                break;
        }
    }
}

Being able to move the deleted document to a recycle bin library by simply handling the Delete event type would be an ideal solution, but unfortunately there's a problem with this approach: events are processed asynchronously. As a result, a registered event sink will only be notified about the document deletion after that document has already been deleted from the SQL Server™ backend database. As a result, the event sink can't simply copy the document to the recycle bin library because the deleted document no longer exists.

The Solution

The solution we present is a simple one that still makes use of WSS document library events. Instead of attempting to copy the document from the document library when the document is deleted, we create a mirror of the document library that will hold copies of all the files, as shown in Figure 3. This mirror library is a normal document library with its own custom event sink, but more on that in a moment.

Figure 3 Maintaining a Mirror Document Library
Figure 3 Maintaining a Mirror Document Library

When files are inserted or updated in the main document library, a custom event handler object will copy the file into the mirror document library in order to keep the mirror current. When a user deletes a file, the event handler will move the copy of the file from the mirror document library into the recycle bin document library, solving the problems caused by the asynchronous processing of events. This will, of course, incur some database overhead because there will now be two copies of the same document (one in the main document library and one in its mirror), so this implementation may not be a good solution for large databases. This is why we provide the ability to filter the documents which are mirrored, based on the document's type and size. This filter can be set using the event sink's SinkData property, allowing users to back up files based on these filter criteria.

Keep in mind, however, that there's more to this implementation than just the handling of events on the server. We also need to customize the recycle bin document library in order to override the default document context menu displayed on the client. This context menu will sport custom menu items that support our custom operations, such as restore.

Because there is no restore event in WSS, our context menu reuses the OWSSVR.DLL Save method. When the restore option is selected, an event is triggered on the server. The event handler class for the recycle bin document library will be notified of the event and will copy the file to the main document library, thus restoring the file. Of course, with the document now back in the main library, the mirror library also needs a copy. One might expect this to be handled automatically by the event handler that copies from the document library to its mirror, but actions performed by event handlers don't result in other events being raised. So, when you copy the file to the main document library, its insert event handler is not triggered. As a result, in addition to copying from the recycle bin library to the main document library, we also programmatically copy the file to the document library's mirror.

Ultimately, the solution consists of two parts: a pair of event handler classes to enable a workflow for moving files around between document libraries based on events, and a customized user interface for a special document library, which will serve as a recycle bin, overriding the default document library context menu displayed for the files in the library.

Handling Document Library Events

We're handling events for two different categories of document libraries: the main library for which we want to add restore capabilities, and the recycle bin library which will catch deleted documents. As such, we created two different event sink classes for handling events.

In April 2004, the Microsoft SharePoint team released the Document Library Event Handler Toolkit to assist in event handler development. The sample comes with a BaseEventSink class, which contains most of the boilerplate code commonly associated with writing event sinks. In particular, it encapsulates the impersonation code that allows for event handlers to access secure resources under a specific user account. The base class provides the HandleIdentity method, which uses the Windows API in conjunction with the WindowsIdentity class to create a new temporary identity for the event handler. This is used to get around the fact that, by default, event handlers execute under IIS 6.0 application pools, which by default run under the unprivileged account of NT AUTHORITY\NetworkService. Since events are processed asynchronously, the client can't be impersonated in the event handler, resulting in the default account being used.

The Document Library Event Handler Toolkit also contains sample code for event handlers that perform a variety of tasks, including the creation of a mirrored document library, the logging of events to a list, and the sending of events to a custom monitoring application using .NET Remoting. We've reused the BaseEventSink class as a base class for our custom event handlers.

Configuration

In order for this workflow to function correctly, we've used the SinkData text property for event sinks (configured along with EventSinkClass and EventSinkAssembly), which has a limit of 255 characters, to provide configuration data. For each document library configured to support the restoration of files, this data includes the name of the mirror library, the name of the recycle bin library, identity information to allow for impersonation, and filter information for purposes discussed earlier:

<Data>
  <Mirror>Backup</Mirror>
  <Recycle>Recycle Bin</Recycle>
  <Domain>TRAINING</Domain>
  <User>SomeUser</User>
  <Pass>pT19n3$ql</Pass>
  <Type>*</Type>
  <Size>*</Size>
</Data>
The SinkData property can be configured through the Advanced Settings of the document library administration page. Note that in our sample, the credentials are stored as clear text. When writing production code, we recommend you use the .NET cryptography classes to encrypt the password.

Instead of parsing the configuration data, we chose to use a custom EventSinkData class that supports the serialization and deserialization of an XML string into an instance of an object. This lets us easily access configuration data as properties on an object rather than having to manually parse the information from the configuration data. For the XML string to be serialized into the EventSinkData object type, we used this code:

public static EventSinkData GetInstance(string text)
{
    using(MemoryStream mstr = new MemoryStream(
        Encoding.UTF8.GetBytes(text))
    {
        XmlSerializer ser = new XmlSerializer(typeof(EventSinkData));
        return (EventSinkData)ser.Deserialize(mstr);
    }
}
This configuration data is then exposed through a new property we added to BaseEventSink called Configuration, making the configuration information accessible to our custom derived sinks.

Creating the Sinks

The BaseEventSink class declares one abstract HandleEvent method to be overridden by our derived classes, DocLibEventSink and RecycleBinEventSink, allowing us to capture the events and create the proper workflow.

Let's begin with the DocLibEventSink class. As discussed, this sink creates copies of the documents inside of the mirror document library and moves the copy of a deleted file from the mirror library to the recycle bin. These actions need to be performed under a security context that has permissions to do so. That is why when deriving from the BaseEventSink class, the HandlerIdentity method should be overridden. It is important to provide the HandlerIdentity method with credentials that have the proper level of access to all the WSS resources that will be affected by the event handler. This includes the main document library, the mirror document library, and the recycle bin document library. The derived DocLibEventSink class will have the following code for the method:

protected override WindowsIdentity HandlerIdentity
{
    get
    {
        if (m_Identity == null)
        {
            m_Identity = CreateIdentity(Configuration.UserName,
                Configuration.Domain, Configuration.Password);
        }
        return m_Identity;
    }
}

This method calls into the base class's Configuration property, which we added earlier, and returns credentials that are impersonated while processing events. The plain text credentials are turned into a WindowsIdentity using the base class's CreateIdentity method, which in turn uses P/Invoke to access the Win32® LogonUser function. With impersonation in place, you need to override the HandleEvent method and provide logic that will handle each of the specific events, as shown in Figure 4.

Figure 4 Overriding HandleEvent (Close Figure 4)
protected override void HandleEvent()
{
    switch (EventInfo.Type)
    {
        case SPListEventType.Delete:
            DeleteHandler();
            break;
        case SPListEventType.Move:
            if (Include()) MoveHandler();
            break;
        case SPListEventType.Update:
        case SPListEventType.Insert:
        case SPListEventType.Copy:
            if (Include()) UpdateInsertCopyHandler();
            break;
    }
}

Here you can access the EventInfo property of the base class, which contains an instance of SPListEvent representing the event that was raised. This enables the document library to determine what type of event occurred. If files are being updated in the document library, the event handler calls the UpdateInsertCopyHandler method to ensure that a copy of the file is made to mirror the document library. In the case where a user deletes a file from the document library that has a recycle bin associated with it, the DeleteHandler method is executed to copy the file from the mirror library to the recycle bin, as shown here:

private void DeleteHandler()
{
    SPFile delFile = EventWeb.GetFile(String.Concat(
        Configuration.MirrorLib, "/", EventInfo.UrlBefore));
    string newFileLoc = String.Concat(Configuration.RecycleLib,
        delFile.Url.Remove(0, Configuration.MirrorLib.Length));

    EnsureParentFolder(EventWeb, newFileLoc);
    delFile.MoveTo(newFileLoc, true);
}
To access the file that has been stored in the mirror, you can call the EventWeb base class property. This returns an SPWeb object corresponding to the location of the event. The Web's GetFile method can then be used with the EventInfo.UrlBefore property to access the SPFile object for the mirrored document.

To support one recycle bin document library for many document libraries, we use a folder structure inside the mirror and recycle bin that correspond to the names of the document libraries from which the files were deleted. The EnsureParentFolder method of the base class creates and ensures that the folder structure for the mirror and recycle bin libraries are intact.

The RecycleBinEventSink class is designed to provide the workflow for the restore functionality. Since there is no restore event, we reuse client-side Insert and Save events (Save corresponds to Update on the server) in order to trigger the copy of the file from the recycle bin document library into the document library of its origin. By overriding the HandleEvent method in the sink, we specify custom logic for the Insert and Update events by calling into the UpdateInsertHandler method. This retrieves files from the recycle bin document library using the EventWeb property of the base class, and again we use the Web's GetFile method to access the file that will be moved to the main document library and copied to the mirror. As mentioned earlier, actions initiated by an event handler do not cause events to be raised, so an event is not raised when the file is moved to the main document library through the event handler. This means you have to manually ensure that the file gets copied into the mirror library. If you want to change your configuration to use different mirror libraries, you first need to retrieve configuration information from the main document library, as shown in the following:

string strCfg = EventWeb.Lists[doclibName].EventSinkData;
EventSinkData cfg = EventSinkData.GetInstance(strCfg);

To recap, we used the BaseEventSink class from the Document Library Event Toolkit and added features to support XML serialization for configuration data. We then created two classes, DocLibEventSink and RecycleBinEventSink, which derive from the BaseEventSink class and which encapsulate the necessary server-side logic to implement a recycle bin. With the server-side logic in place, we now need to create and customize our document library views to support a restore menu item for the recycle bin.

Customization of the Web Part Pages

The document library's Web page user interface is based on the WSS Web Part Page framework, which extends the ASP.NET control model. The idea behind the Web Part Page framework is to allow the Web designer to customize the contents of the workspace and to allow the end user to personalize the information using Web Parts. Document libraries come with five built-in views, as shown in Figure 5.

Figure 5 Document Library Web Part Pages (Views) (Close Figure 5)

File Name | Description
AllItems.aspx | Default view showing all the items in the document library
DispForm.aspx | Page showing the document properties
EditForm.aspx | Page showing the form for modifying document properties
Upload.aspx | Page holding the upload form for uploading document to the library
WebFldr.aspx | Page holding the Explorer View which is an ActiveX component for manipulating the files

The default view, AllItems.aspx, shows contents and files with a column and row structure. When a new document library is created, the physical ASPX files (available at \60\TEMPLATE\1033\ STS\LISTS\DOCLIB\), are used in conjunction with the SCHEMA.XML template file. A Collaborative Application Markup Language (CAML) definition is used for the fields and scripts, producing five entries inside of the WSS content database. For example, when AllItems.aspx is invoked, the SharePointHandler endpoint retrieves data for the page's ListViewWebPart Web Part and populates its ListViewXml property with values from the database. The output is sent to the Web browser.

There are only two options for modifying design-time information for the views: using a Web browser and the administrative view provided by WSS or using Microsoft FrontPage® 2003. WSS stores all its contents inside its content database, and that information is retrieved at run time by the SharePointHandler. To retrieve design-time values for the Web Part Page, FrontPage uses a combination of FrontPage Server Extensions, WSS Web services, and remote procedure calls (RPC) to retrieve the view and save it back to the database (see Figure 6).

Figure 6 FrontPage Design-Time View
Figure 6 FrontPage Design-Time View

This process only needs to be performed for one recycle bin library, since the library can be saved as a template and reused on other servers. Start by launching FrontPage 2003, creating a new document library, and naming it "Recycle Bin." Next, open the AllItems.aspx page for the library, modify the page in FrontPage 2003, and select the List View WebPart, which is located in the Main Web Part Zone on the page. Disable the toolbar on the document library. From the List View Options, select Style to open the View Styles dialog box. Next, select the Options tab and uncheck the "Show toolbar with options" box and click OK. Finally, save the page and exit FrontPage 2003. This will disable standard document library options such as New Document, Upload Document, and others. Since this document library is going to serve as a recycle bin, it is not necessary to have those options enabled.

Next, switch to code view and paste the JavaScript code shown in Figure 7 between the head tags for the Web Part Page. This script overrides the default menu items by specifying the custom menus that contain Restore and Delete items. OWS.JS contains a client-side JavaScript object model, and through extensibility points it is possible to override menus without modifying the original file. One such point is the following code from OWS.js:

function AddDocLibMenuItems(m, ctx)
{
    if (typeof(Custom_AddDocLibMenuItems) != "undefined")
    {
        if (Custom_AddDocLibMenuItems(m, ctx)) return;
    }
    ... // build regular menu
 }

Figure 7 Overriding Default Menu Items (Close Figure 7)
function Custom_AddDocLibMenuItems(m, ctx)
{
   var RootFolder = GetRootFolder(ctx);
   setupMenuContext(ctx);

   if (currentItemFileUrl == null)
       currentItemFileUrl = itemTable.ServerUrl;
   if (currentItemFSObjType == null)
       currentItemFSObjType = itemTable.FSObjType;
   var currentItemEscapedFileUrl =
       escapeProperly(unescapeProperly(currentItemFileUrl));

   // document library might contain folders; thus we need to enable
   // default delete on the folder
   if (currentItemFSObjType == 1)
   {
       strDisplayText = L_DeleteDocItem_Text;
       strAction = "DeleteDocLibItem('" +
          ctx.HttpPath + "&Cmd=Delete&List=" + ctx.listName +
          "&ID=" + currentItemID + "&owsfileref=" +
          currentItemEscapedFileUrl + "&NextUsing=" + GetSource() + "')";
       strImagePath = ctx.imagesPath + "delitem.gif";
       CAMOpt(m, strDisplayText, strAction, strImagePath);
   }
   else // build custom menus for the files
   {
       // build restore menu item and wire it to trigger Update
       var L_Restore_Text = "Restore File";
       strDisplayText = L_Restore_Text;
       strAction = "SubmitFormPost('" +
          ctx.HttpPath + "&Cmd=Save&List=" + ctx.listName +
          "&ID=" + currentItemID + "&owsfileref=" +
          currentItemEscapedFileUrl + "&NextUsing=" + GetSource() + "')";

       // reused approval image, but custom image can be
       // associated with the menu item
       strImagePath = ctx.imagesPath + "APPROVE.GIF";
       CAMOpt(m, strDisplayText, strAction, strImagePath);

       // build regular delete menu item to trigger a complete delete
       // of the file
       strDisplayText = L_DeleteDocItem_Text;
       strAction = "DeleteDocLibItem('" +
          ctx.HttpPath + "&Cmd=Delete&List=" + ctx.listName +
          "&ID=" + currentItemID + "&owsfileref=" +
          currentItemEscapedFileUrl + "&NextUsing=" + GetSource() + "')";
       strImagePath = ctx.imagesPath + "delitem.gif";
       CAMOpt(m, strDisplayText, strAction, strImagePath);
   }
   return true;
}

The JavaScript code in Figure 7 contains a function called Custom_AddDocLibMenuItems, which integrates with this code from OWS.js and contains logic for adding the Delete menu item for a Folder type and the Restore and Delete items for file types. We mentioned before that there is no built-in restore event, so we're reusing the Save functionality of the OWSSRV.DLL.

The GetSource function results in a redirect to the specified page after the event has been sent to the server. Because there is no event for the Restore, we have to use the Save command to trigger the restoration process on the server. Unfortunately, there is a little bug with this JavaScript code that we could not overcome. Sometimes, when the file size is large, it takes time to move the file from the recycle bin to the document library of its origin. The GetSource function does not wait until the server processes are complete and the document library view is refreshed. The file therefore still appears in the view even though the command has been submitted to OWSSRV.DLL. If you refresh the page manually after the operation has completed, the file will most likely disappear from the document library.

Because the mirror document library contains backup copies of the files, it is a good idea to hide it from the users to prevent tampering. Using FrontPage 2003, open the site that contains the mirror, right-click the mirror document library folder and select Properties. Select the Settings tab and check the Hide from browsers checkbox. This will hide the document library in the site's list of lists (users will still be able to access the document library by using the UNC path in Windows Explorer).

At this point, the infrastructure for the client and the server side is set, so enable the document libraries for which you want to have recycle bin functionality by specifying custom event handlers in the Advanced properties.

Conclusion

Because files deleted from WSS are not recoverable without lots of pain and administrator intervention, recycle bin functionality on the WSS document libraries is a welcome customization. This safety net is important to both portal users and administrators since trying to reload a portal backup for only a few documents is tedious and often results in the loss of more data than it saves.

Maxim V. Karpov is an independent consultant and software architect. He is completely sold on the idea that design patterns provide developers with a universal language for communication and a platform for developing superior application architectures. He can be reached at http://ipattern.com.

Eric Schoonover is a Senior Application Developer for Avanade Inc. Based in Seattle, he travels throughout the country designing portals for the insurance, financial, defense, media, and software industries. Eric can be reached at ericscho@avanade.com.


Smart Tags: Realize the Potential of Office 2003 by Creating Smart Tags in Managed Code

New information has been added to this article since publication.
Refer to the Editor's Update below.

Smart Tags

Realize the Potential of Office 2003 by Creating Smart Tags in Managed Code

Ben Waldron
This article discusses:
  • New smart tag features in Office 2003
  • MOSTL schema for easy smart tag creation
  • Developing smart tags using managed code
  • Smart tag debugging and deployment
This article uses the following technologies:
Office, COM, C#
Code download available at: ManagedSmartTags.exe (128KB)
Contents
Smart Tag History
MOSTL Schema
Creating a Smart Tag Library
Recognizer Class
Action Class
Debugging Tools
Deployment
Security
Conclusion

S
mart tags were first introduced in Microsoft® Office XP as an integrated way to enable users to dynamically present additional information in their documents. For general users of Office, smart tags are a way to boost productivity by having actions linked to keywords. For corporate users, smart tags are used to link users to internal resources based on the type of data they view or input into Office applications. A good example is linking a purchase order number in an Excel spreadsheet to detailed information about the sale from a source on the company intranet. For developers, smart tags represent a new way to link information-driven services into the applications that clients spend most of their time using.

I polled developers in my community and found that most of them recognized the benefits of smart tags but had not yet taken steps to develop them or to use the Microsoft .NET Framework to do so. While many developers are excited about the prospect of building managed smart tags, developing smart tags using .NET has not received the attention it deserves.

In this article I'll demonstrate how to develop smart tags. I'll cover topics such as using the Microsoft Office Smart Tag List (MOSTL) XML schema to develop smart tags quickly as well as using the .NET Framework to create more advanced, managed smart tags for Office 2003 and Office XP. Then I'll talk about how you can deploy these features in your organization.

Smart Tag History

Smart tags came onto the scene in Office XP with full support in Word, Excel, and Microsoft Outlook® (when Word is used as the e-mail editor, which is the default). Microsoft Internet Explorer 5.0 and above support smart tags in a manner that's different from Office applications. Text is not recognized by smart tags automatically in these versions of Internet Explorer; however, custom HTML tags can be added to the HTML source file to enable recognition of text for which you want to have a smart tag action. In this first iteration, smart tags for Office XP were developed using the Microsoft Smart Tag 1.0 Type Library.

The simplicity of smart tags is what makes them so powerful. The smart tag must first recognize data as matching some important criteria and should then present the user with a menu list of actions that pertain to that data. When the user makes a selection, the appropriate action is taken. Two COM classes comprise a smart tag. A recognizer class is responsible for matching data passed by a smart tag-enabled application; an action class tells the application the menu items to display and executes an action when the application signals the action to be taken. The general interaction between the application and these classes is shown in Figure 1.

Figure 1 Smart Tag Interfaces
Figure 1 Smart Tag Interfaces

Based on customer feedback Microsoft made enhancements to smart tags with the release of Office 2003. The number of applications that support smart tags has been expanded to include Microsoft PowerPoint® and Microsoft Access. Besides broader application support, the next iterations of smart tags include some better usability options. In Office XP, smart tags supported only one level of menus when displayed, but now cascading menus are included to group actions within the smart tag menu. The ability to control the period of time that smart tags are active, and allowing users to disable them from a single configuration window are also included in Office 2003. Another feature that developers have asked for is the ability to know what application is calling the smart tag. This enables you to handle recognition and actions one way if the request comes from Word and another if it comes from Excel. For a complete list of new smart tag features in Office 2003, go to What's New with Smart Tags in Office 2003.

Microsoft Smart Tags 2.0 Type Library, which is used to develop smart tags for Office 2003, is completely backward compatible with Office XP. All features that existed in Office XP will work with smart tags developed using the new type library, which is very nice for organizations that run both Office XP and Office 2003.

MOSTL Schema

Simple smart tags that come to mind involve recognizing a word and then linking that word to a Web site. Building this type of smart tag is as easy as constructing an XML file. In Office XP and Office 2003 there is a library named MOFL.dll (C:\Program Files\Common Files\Microsoft Shared\Smart Tag\MOFL.dll) that allows developers to build XML files that conform to the MOSTL schema to build tags with no coding necessary.

The MOSTL schema is used when you want to build smart tags where well-defined words or patterns are recognized and where relatively simple actions are executed once recognition occurs. A new feature in Office 2003 allows developers to use regular expressions to recognize a pattern instead of just a list of terms. For example, you can write a smart tag that can recognize an IP address and develop an action that will go to a Web site listing who owns that IP address. While this may not be practical for a business user, it does illustrate the power of MOSTL combined with regular expressions. This type of pattern recognition can be used in most businesses as unique formatted codes are commonly used for order numbers, tracking numbers, and other identifiers. The MOSTL XML that performs IP address recognition and action is shown in Figure 2.

Figure 2 Recognizing an IP Address (Close Figure 2)
<FL:smarttaglist xmlns:FL="urn:schemas-microsoft-com:smarttags:list">
  <FL:name>WhoIs IP Lookup</FL:name>
  <FL:lcid>1033,0</FL:lcid>
  <FL:description>Lookup up a domain name</FL:description>
  <FL:moreinfourl>http://msdn.microsoft.com/msdnmag</FL:moreinfourl>
  <FL:updateable>false</FL:updateable>
  <FL:autoupdate>false</FL:autoupdate>
  <FL:lastcheckpoint>0</FL:lastcheckpoint>
  <FL:lastupdate>0</FL:lastupdate>
  <FL:updateurl></FL:updateurl>
  <FL:updatefrequency>0</FL:updatefrequency>
  <FL:smarttag type="urn:schemas-microsoft-com:office:smarttags#whois">
    <FL:caption>WhoIs IP Search</FL:caption>
      <FL:re>
        <FL:exp>\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-
          9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b</FL:exp>
      </FL:re>
    <FL:actions>
      <FL:action id="WhoIs">
        <FL:caption>Find WhoIs for IP</FL:caption>
        <FL:url>http://ws.arin.net/cgi-bin/whois.pl?queryinput={TEXT}
        </FL:url>
      </FL:action>
    </FL:actions>
  </FL:smarttag>
</FL:smarttaglist>

XML that is produced for a smart tag must strictly conform to the MOSTL schema and must be a valid XML document. This means that all characters within XML must be properly escaped. The beginning of the MOSTL schema deals with setting up the smart tag so that it is recognized as a valid smart tag, setting the locale, and creating the ability to update the smart tag from a Web site at some frequency. The most important XML tags create the regular expression to be recognized and also the action that is to be taken when a user clicks on an action in the smart tag. In this case, when an IP address is recognized and the user clicks on the "Find WhoIs for IP" action in the smart tag, they are taken to the WhoIs registry for that particular IP address, as shown in Figure 3.

Figure 3 Tag
Figure 3 Tag

Using the MOSTL XML schema is a fast way to create smart tags. Deploying a new smart tag involves copying the XML file to the location where Office is installed. By default, copying the file to C:\Program Files\Common Files\Microsoft Shared\Smart Tag\LISTS or a subdirectory for a specific locale will make it available to a user. This can be done using an installation application, group policy, or through a software update service such as Microsoft System Management Service.

Creating a Smart Tag Library

Oftentimes, it is not quite so easy to link recognized data to the appropriate action so a developer might need to add more programming logic than the MOSTL schema affords. As mentioned earlier, most businesses use unique identifiers for purchase orders, tracking shipments, or products for sale. More often than not, linking these identifiers to meaningful information involves deriving information from multiple sources, and to be more valuable, aggregating the data for the user. A user with a sales order identifier can quickly see the customer, price paid, shipping information, and purchase history without ever having to leave Office. In cases like this, it is necessary to build a library that implements the interfaces required to create a smart tag. Creating a simple smart tag involves implementing defined interfaces and, as with any software project, gets more complicated as more logic is needed to perform advanced functions. Building a managed smart tag allows developers to take advantage of all the features of the .NET Framework that they have been accustomed to.

Before creating a smart tag library, I would like to describe the functionality in my smart tag library example. Being a fan of major league baseball, it would be helpful to recognize terms that relate to teams. Once these terms are recognized, I can link them to various Web sites to show team standing, averages, and other information that a baseball fan might want to see. This smart tag will have to have some internal logic to determine the correct team to link to. For instance, if I match the word "Sox" in the smart tag, then I may have to present the user with choices to the Web site for the Chicago White Sox or the Boston Red Sox.

This major league baseball smart tag will be written using Visual Studio® .NET 2003. The first step to building a managed smart tag is to create a new Class Library project and add a reference to the COM Microsoft Smart Tags 2.0 Type Library, shown in Figure 4.

Figure 4 Microsoft Smart Tags 2.0 Type Library List
Figure 4 Microsoft Smart Tags 2.0 Type Library List

Adding this reference allows smart tags to be created for both Office 2003 and Office XP. Referencing this library automatically wraps the COM library for use by the .NET Framework without requiring the developer to do any more work. The library that will be produced needs to be registered for COM Interoperability, so doing this within Visual Studio .NET will assist with debugging and deployment once the library is produced.

The two aforementioned classes that are required for the smart tag are the recognizer class, responsible for recognizing text or patterns that exist within applications that support smart tags, and the action class, which determines the menus that are shown to the user and the appropriate operation that corresponds with a menu item. I will first explore how to develop a recognizer class and then build the action class.

Recognizer Class

A recognizer class interfaces with the smart tag infrastructure using COM, so it is necessary to add the System.Runtime.InteropServices namespace to include the necessary attributes for COM integration with .NET-based applications. The Microsoft.Office.Interop.SmartTag namespace is also imported so that my Msdn.MlbSmartTag.Recognizer class can easily implement the appropriate interfaces, as shown here:

using System;
using System.Runtime.InteropServices;
using Microsoft.Office.Interop.SmartTag;

namespace Msdn.MlbSmartTag
{
    [ProgId("MlbSmartTag.Recognizer")]
    [GuidAttribute("90D62E8D-79E3-4197-907F-BF3EC133698D")]
    [ComVisible(true)]
    public class Recognizer : ISmartTagRecognizer2, ISmartTagRecognizer
    ...

There are three important attributes that decorate my Recognizer class. The ProgId attribute allows you to specify the programmatic identifier that can be used when the class is registered with COM. More important, though, is GuidAttribute, which forms a unique class ID (CLSID) for the class. When deploying the library, it is possible to use either ProgId or the CLSID to register the interface, but it's recommended that you use the CLSID to avoid a problem in Office XP where the status flag does not update when registering the interface using the ProgId. The ComVisible attribute is added to the class to ensure that it can be seen as a type in COM.

A recognizer class must implement the ISmartTagRecognizer2 and ISmartTagRecognizer interfaces. Visual Studio .NET will include the stubs for implementing the necessary interfaces, which makes it very easy to fill in the necessary logic to complete the recognizer class.

Figure 5 shows a complete list of the properties and methods to implement the interfaces for a valid recognizer. The SmartTagName property is used to uniquely identify the recognizer, as well as serving as the action key when text is recognized and sent to the corresponding action class. The SmartTagName is formed by using a namespace URI, and then adding the smart tag name separated by a pound sign. In this example the SmartTagName is urn:schemas-microsoft-com:msdnmag#mlb. It is important that this name does not conflict with any other SmartTag name that is deployed within the same environment. This name is also very important when building an action class since the smart tag name serves at the glue that binds the recognizer to the action. These names must be the same; otherwise the two classes will not be properly linked together.

Figure 5 Recognizer Properties and Methods (Close Figure 5)

Smart Tag Recognizer Members | Description
Recognize2 method | Version 2.0 method that recognizes character strings as smart-tag actionable. Includes ability to detect application name and additional parameters to assist recognition.
Recognize method | Version 1.0 method that recognizes character strings as smart-tag actionable.
SmartTagInitialize method | Initializes objects when recognizer is loaded.
DisplayPropertyPage method | Launches the property dialog for the recognizer.
PropertyPage method | Provides a way to customize a smart tag recognizer to enable or disable features via the user interface.
Desc property | A description of the smart tag recognizer.
Name property | A name for the recognizer. The name appears in the smart tags tab (Tools menu, AutoCorrect Options dialog box) in Office applications.
ProgID property | The unique identifier (programmatic ID) for the recognizer's class.
SmartTagCount property | The number of smart tag types that this recognizer recognizes.
SmartTagDownloadURL property | The URL that is navigated to if the More Smart Tags button (Tools menu, AutoCorrect Options dialog box, smart tags tab) is clicked.
SmartTagName property | The unique identifiers for smart tag types that the recognizer supports.

It is important to remember that all COM-accessible .NET classes must provide a default constructor; that is, a constructor that requires no parameters. COM object creation functions do not pass parameters to the objects that they create so you need to ensure that the default constructor is present.

The SmartTagInitialize method is used to perform any initialization necessary when the smart tag recognizer is loaded. In the example, a list of team names are initialized into an array and that will serve as the terms that should be recognized. It should be mentioned that there are many other ways to recognize terms other than a simple list of tokens. Regular expressions are typically used to match patterns such as unique identifiers, while programming logic can be used to retrieve data or determine other match criteria. A good example of this is the Person Name Smart Tag that is included with Office. This smart tag retrieves a list of names from Outlook Contacts for recognition and allows the user to quickly send an e-mail, schedule an appointment, or see if that user is currently online. By performing this sort of setup and information retrieval in SmartTagInitialize, performance of the smart tag recognizer can be improved, since it can access the cached data rather than having to query for it on every recognition request.

The Recognize2 method (see Figure 6) is where the bulk of the responsibility exists for this class. The calling app passes to this method the text to be recognized, the locale, and type of data where the text was found in the app. The data type is set based on the text's surrounding context. The type would be IF_TYPE_CELL if it is found within a cell in Excel, and IF_TYPE_PARA if it is within a paragraph in Word. Some new parameters have also been added that did not exist in the original Recognize method. The calling app name allows a recognizer to perform alternate recognition based on the application, and a tokenized list of text items has been added to assist in developing match algorithms.

Figure 6 Recognizer Class Implementation (Close Figure 6)
public class Recognizer : ISmartTagRecognizer2, ISmartTagRecognizer
{
  private string[] terms;
  private int termCount;

  /// default constructor - no parameters for COM Interop
  public Recognizer(){}

  public void Recognize2(
    string Text, IF_TYPE DataType, int LocaleID,
    ISmartTagRecognizerSite2 RecognizerSite2, string ApplicationName,
    ISmartTagTokenList TokenList)
  {
    for (int i=1; i<=TokenList.Count; i++)
    {
      // get token
      ISmartTagToken token = TokenList.get_Item(i);

      // match token for each term
      foreach (string term in terms)
      {
        if (token.Text.ToLower() == term.ToLower())
        {
          ISmartTagProperties propertyBag  =
            RecognizerSite2.GetNewPropertyBag();

          // commit token as recognized term
          RecognizerSite2.CommitSmartTag2(
            "urn:schemas-microsoft-com:msdnmag#mlb",
            token.Start, token.Length, propertyBag);
        }
      }
    }
  }

  public void SmartTagInitialize(string ApplicationName)
  {
    terms = new string[] {
        "Cubs", "Cardinals", "Astros",
        "Yankees", "Pirates", "Angels", "Sox"
    };
    termCount = terms.Length;
  }

  public string get_SmartTagName(int SmartTagID)
  {
    return "urn:schemas-microsoft-com:msdnmag#mlb";
  }

  public string ProgId { get{ return "MlbSmartTag.Recognizer"; } }
}

The Recognize2 method loops through all of the tokenized text passed to it by the calling application to try to find a match. If a match is found, a smart tag property bag is created to let you add more properties for the text that is matched. The RecognizerSite2 is responsible for interfacing with the calling app. The CommitSmartTag2 method passes the property bag and the location where the recognition occurred within the app. A call to this method causes the app to call the corresponding Action class along with the property bag to display the menu that lists possible actions.

Action Class

Building my Msdn.MlbSmartTag.Action class is similar to building my Recognizer class. Figure 7 shows the most important properties and methods needed to build the Action class. The same namespaces are necessary to make the class accessible to COM and to use the smart tag interfaces. It is important to note that the ProgId and GuidAttribute should not be the same as the values in my Recognizer class (they are, after all, different COM classes), so it is necessary to create a new unique identifier that will be the CLSID for the Action class.

Figure 7 Action Class (Close Figure 7)
public class Action : ISmartTagAction2, ISmartTagAction
{
  public bool get_ShowSmartTagIndicator(
    int VerbID, string ApplicationName, int LocaleID)
  {
    return true;  // show the underline indicator
  }

  public int get_VerbCount(string SmartTagName)
  {
    if (SmartTagName == "urn:schemas-microsoft-com:msdnmag#mlb")
      return 4;  // there are four possible actions
    else
      return 0;
  }

  public bool get_IsCaptionDynamic(
    int VerbID, string ApplicationName, int LocaleID)
  {
    return true;  // the caption is changed based on team name
  }

  public string get_VerbCaptionFromID2(
    int VerbID, string ApplicationName, int LocaleID,
    ISmartTagProperties Properties, string Text,
    string Xml, object Target)
{
    string captionText = string.Empty;
    switch (VerbID)
    {
      case 1:
        if (Text == "Sox")
          captionText = "MLB///Go to Red Sox website";
        else
          captionText =  String.Format("MLB///Go to {0} website", Text);
        break;
      case 2:
        if (Text == "Sox")
          captionText = "MLB///Go to White Sox website";
        break;
      case 3:
        captionText = "MLB///Show Standings";
        break;
      case 4:
        captionText = "MLB///Show Schedule";
        break;
    }
    return captionText;
  }

  public void InvokeVerb2(
    int VerbID, string ApplicationName, object Target,
    ISmartTagProperties Properties, string Text,
    string Xml, int LocaleID)
{
    ProcessStartInfo startInfo = new ProcessStartInfo("iexplore.exe");
    switch (VerbID)
    {
      case 1:
        if (Text == "Sox")
          startInfo.Arguments = "http://www.redsox.com";
        else
          startInfo.Arguments = String.Format("http://www.{0}.com", Text);
        break;
      case 2:
        startInfo.Arguments = "http://www.whitesox.com";
        break;
      case 3:
        startInfo.Arguments = "http://msn.foxsports.com/mlb/standings";
        break;
      case 4:
        startInfo.Arguments = "http://msn.foxsports.com/mlb/schedule";
        break;
    }
    if (startInfo.Arguments != string.Empty)
      Process.Start(startInfo);
  }

  public string get_SmartTagName(int SmartTagID)
  {
    return "urn:schemas-microsoft-com:msdnmag#mlb";
  }
}

The Action class implements two interfaces from the Smart Tag 2.0 Type library: ISmartTagAction2 and ISmartTagAction. As with a recognizer class, all the properties and methods are necessary and Visual Studio .NET will generate stubs when adding the interfaces to the class declaration, as shown here:

using System;
using System.Runtime.InteropServices;
using Microsoft.Office.Interop.SmartTag;

namespace Msdn.MlbSmartTag
{
  [ProgId("MlbSmartTag.Action")]
  [GuidAttribute("63CB4201-3387-47c3-B0B2-0CCF56BB892D")]
  [ComVisible(true)]
  public class Action : ISmartTagAction2, ISmartTagAction
  ...

Figure 8 lists the properties and methods used by an Action class. Some properties are the same as those in the recognizer class but there are a few that deserve more attention. As I mentioned in the recognizer discussion, the SmartTagName property value must be the same in an Action class as it is in the corresponding recognizer so that the calling application can forward the request to the appropriate action handler.

Figure 8 Smart Tag Action Class Properties and Methods (Close Figure 8)

Smart Tag Action Member | Description
InvokeVerb2 method | Version 2.0 method that runs when an item is clicked on the Smart Tag Actions menu. This method includes the locale as a parameter.
InvokeVerb method | Version 1.0 method that executes when an item is clicked on the Smart Tag Actions menu.
VerbCaptionFromID2 property | Version 2.0 property to display captions for the available actions in the Smart Tag Actions menu. Additional parameters include the property bag and the recognized text.
VerbCaptionFromID property | The captions for the available actions in the Smart Tag Actions menu.
Desc property | A description of the smart tag action.
Name property | A title for the action.
IsCaptionDynamic property | Determines whether caption is static or dynamically created.
ProgID property | The unique identifier (programmatic ID) of the action's class.
SmartTagCaption property | The caption that will be displayed at the top of the Smart Tag Actions menu.
SmartTagCount property | The number of smart tag types the corresponding recognizer recognizes.
SmartTagName property | The unique identifiers for smart tag types that the recognizer supports. This is the same as the name in the corresponding recognizer.
VerbCount property | For a given smart tag type, the property that determines how many actions are supported.
VerbID property | The property that returns a unique ID (used internally within the application) within the smart tag. This mechanism is supplied so that smart tag DLLs can mix and match smart tag actions for the various smart tag types they support.
VerbNameFromID property | The property that returns a name (used internally within the application) to represent the smart tag action.

The ShowSmartTagIndicator property is used to determine whether the underline indicator should be shown in order to signal to the user that the smart tag has been recognized and has action(s) associated with it. Typically, this property is always set to true, but based on the calling application or locale, it might be set to not show the indicator.

The VerbCount property is set to the maximum number of actions that the smart tag will display. In the example, a simple check is made to make sure that the correct smart tag is calling this property and then the value 4 is returned because the Smart Tag Action menu has four possible actions.

The VerbCaptionFormID2 property does a substantial amount of work for an action class. Based on the value set in the VerbCount property, this property is called repeatedly, each time with a different VerbID. It is responsible for creating the text that will de displayed in the Smart Tag Action menu. Cascading menus are a new feature in Office 2003. Using the "///" delimiter in the text that is returned by the property will apportion the text into submenus.

The first time the VerbCaptionFromID2 property is called, the value of VerbID will be 1 and the appropriate case statement is called. It is important to note that if you are using the VerbIndex as the VerbID (as in the example), the beginning index for VerbIndex is 1 rather than 0. This is something that can sometimes cause issues for those accustomed to doing development primarily with the .NET Framework. Custom logic is embedded in this property to handle the case where the word "Sox" is recognized. This is because we do not know if the user wanted to match the Boston Red Sox or the Chicago White Sox.

The IsCaptionDynamic property comes into play when the menu that is displayed to the user changes because of some value. If the menus for the action do not change, then this property should be set to false. For this example, we display a different menu based on what has been recognized so this property is set to true. These dynamic menus are one of the reasons why this smart tag is being built as a library rather than using the MOSTL schema.

The capstone to a smart tag is when a user clicks on a menu item to invoke an action. This is handled by the InvokeVerb2 method and is where developers can provide the horsepower for doing interesting things. The VerbID, application name, property bag, and recognized text are passed to this method. New to the InvokeVerb2 method is the locale so that developers can more easily build multilingual applications.

The example InvokeVerb2 is relatively straightforward and serves as a good example. Based on the menu item that the user selected, a browser opens the appropriate Web page. A user who types in "Cubs" or "Chicago Cubs" can navigate quickly to their stats. (If they are a Cubs fan like me then they will no doubt be disappointed—a common theme for us Cubs fans.)

There are countless other actions that can be taken when a user clicks a smart tag action. Using the .NET Framework to create the smart tag increases the possibilities by allowing you to do virtually anything that you can already do in .NET. This includes calling Web services, querying a database, and finding other interesting ways to add value to the data in Office documents.

One other feature of Office 2003 that is complementary to smart tags is the Office Research pane. The Research pane gives developers a place to display information to users without having to leave Office. Smart tags offer a nice link to notify users that they can view more information in the Research pane. For further information, see John Durant's article on storing and searching instant messages using the Research pane in Research Services in this issue of MSDN Magazine.

Debugging Tools

There have been some tools released recently that help developers debug and test smart tags without having to test directly within Office applications. The first utility, called the Smart Tag Explorer, enables developers to view all of the smart tag recognizers and actions that are configured on the local system. This is especially useful for ensuring that registration has completed successfully and that the smart tags are enabled on the system. It also shows which applications are configured to run the smart tags along with other settings, such as where to check for updates and the color of the smart tag indicator.

Another priceless utility is the Smart Tag Recognize Tester. This utility can be run from the Smart Tag Explorer and is used to test the recognition functionality. This utility is a time saver when the recognition logic is complicated and you want to quickly test a variety of data to be recognized. Both of these utilities are managed applications, so it is very easy to attach to them with a debugger in order to quickly step through the code to debug or test the smart tag. It's easy to see how the interface for the Smart Tag Recognize Tester, shown in Figure 9, can be helpful for debugging and testing by indicating the text and offsets of the recognition. These tools are available for download at Introduction to the Smart Tag Explorer and Smart Tag Recognize Tester.

Figure 9 Smart Tag Recognize Tester (Close Figure 9)

Deployment

Now that you have developed the functionality of the smart tag, it is just as important to get it in the hands of the users who need it. Deploying a smart tag that has been developed using the MOSTL schema involves copying the XML file to all the machines that need the smart tag. MOSTL also lets you set the frequency and location where updates can be downloaded from a Web site.

Deploying smart tags libraries is not as simple because the library must be registered and Windows registry keys must be created to allow the application to know that the smart tag exists. Creating a managed smart tag library provides the ability to modify the registry using the ComRegisterFunctionAttribute when the library is registered using the Register Assembly (regasm.exe) app.

The are only two registry keys that are needed to allow the smart tag to run: one for the recognizer (HKEY_CURRENT_USER\Software\Microsoft\Office\ Common\ Smart Tag\Recognizers\{CLSID}) and one for the action (HKEY_CURRENT_USER\Software\Microsoft\Office\ Common\Smart Tag\Actions\{CLSID}). The unique identifier created for each class is placed into the appropriate sub key. The code in Figure 10 demonstrates how to ensure that these correct keys are placed in the registry for the recognizer class when the library is registered. It also shows the UnregisterFunction for when the library is unregistered.

Figure 10 Adding and Deleting Keys for Smart Tags (Close Figure 10)
// registry keys
static readonly string clsId = "90D62E8D-79E3-4197-907F-BF3EC133698D";
static readonly string recognizersKey = "SOFTWARE\\Microsoft\\Office" +
  "\\Common\\Smart Tag\\Recognizers\\{" + clsId + "}";

// Install registry keys to make recognizer available to applications
[ComRegisterFunction]
public static void RegisterFunction(Type type)
{
  // create registry keys for recognizer
  using(RegistryKey regHkcu =
    Registry.CurrentUser.CreateSubKey(recognizersKey))
  {
    regHkcu.SetValue(null, string.Empty);
  }
}

// Uninstall registry keys
[ComUnregisterFunction]
public static void UnregisterFunction(Type type)
{
  Registry.CurrentUser.DeleteSubKey(recognizersKey);
}

Having the registry keys applied when the library is registered simplifies deployment. Copying the library to the clients and registering it can be accomplished in a variety of ways. The first is to build an installation for the smart tag library and allow users to install it themselves. Other ways include deploying the library via group policy or other software update applications that exist in the environment to which you are deploying.

[ Editor's Update - 2/8/2005: While registering managed smart tags for COM interop is a great way to get them up and running quickly, developers are strongly advised to isolate their managed smart tags from other managed Office add-ins. .NET provides isolation through the means of application domains, and loading each smart tag into its own application domain will improve the overall stability of the host application. Office 2003 provides for this separation through the Microsoft Visual Studio Tools for the Microsoft Office System loader. It's easy to use and represents the correct way to deploy managed smart tags. Moreover, the Visual Studio Tools for Office loader leverages the .NET security model, providing tighter security for managed add-ins. For more information on designing, developing, and deploying Office XP and Office 2003 add-ins implemented with the .NET Framework, see Information about designing Office XP add-ins and Office 2003 add-ins by using the .NET Framework and Support for Managed-Code Smart Tags. Additionally, the documentation at Isolating Office Extensions with the COM Shim Wizard discusses another way of implementing application domain isolation through the use of a C++ shim.]

Security

When creating a smart tag developed with the .NET Framework in Office XP, the library was subject to Office security checks based on a digital signature. Unfortunately, the .NET runtime engine (Mscoree.dll) is not digitally signed for Office. This meant that it was possible for users to see a macro security dialog box when using smart tags. The workaround for Office XP was to develop an unmanaged shim to call the .NET component.

In Office 2003, it is no longer necessary to develop an unmanaged shim because Office 2003 allows smart tags and other managed add-ins to be subject to the .NET security policy instead of to the Office security checks. The Visual Studio Tools for Office loader, which is included with Office 2003, loads the managed smart tag and is subject to Office security checks instead of the managed code—another good reason to start building managed smart tags.

Conclusion

There are many great uses for smart tags, and the tags are easily built using the MOSTL schema. Using the .NET Framework to build smart tags makes it easier to link Office users to external data quickly and intuitively. Likewise, building managed smart tags makes it simpler to deploy smart tags to users who need them.

Ben Waldron is a senior consultant with Microsoft Consulting Services providing architecture guidance and developing solutions for customers in the Federal government. He is based in Washington, D.C. and can be contacted at bwaldron@microsoft.com.


SQL Server: Display Your Data Your Way with Custom Renderers for Reporting Services

SQL Server

Display Your Data Your Way with Custom Renderers for Reporting Services

James Yip
This article was based on Beta 2 of Microsoft SQL Server 2005 and all information contained herein is subject to change.
This article discusses:
  • Introduction to SQL Server 2005 and Reporting Services
  • Architecture of Reporting Services renderer extensions
  • Creating a customer report renderer
  • Rendering a Reporting Services report into HTML
This article uses the following technologies:
SQL Server, Visual Basic
Code download available at: CustomRenderers.exe (123KB)
Contents
Reporting in Reporting Services
Getting Started
Rendering the Report
Rendering a Report Object
Let's Try It
Conclusion

S
QL Server™ 2005 Reporting Services from Microsoft is a great tool that offers a centralized approach to storing and rendering reports. It also allows users to view and download reports without installing additional software. But what's most convenient for users is that reports can be saved in any number of different formats using custom report renderers. In this article, I will demonstrate how to develop one such report renderer that outputs HTML reports, but the skills you'll learn can easily be used to create a renderer for Microsoft® Word documents or any other format of your choosing.

SQL Server 2005 includes many new features and enhancements. These include support for stored procedures written in managed code, user defined functions, and managed user-defined data type (UDT) support, which provides flexibility and reduces the need for data type conversion when reading and saving data to and from the database. As you will see, these particular features will come in handy when you build your custom report renderer.

Reporting in Reporting Services
Figure 1 Deployment Scenario
Figure 1 Deployment Scenario

Before I create a custom renderer for use with Reporting Services, let's briefly review the basics. Reporting Services is a server-based reporting tool that produces transactional and business intelligence reports. Figure 1 shows a typical deployment scenario.

Reporting Services is modular, so you can easily extend its functionality. Existing extensions to the model include those for data delivery, data processing, and report rendering. Data delivery extensions can be used with the subscription service in Reporting Services. They are used to send rendered reports to a particular recipient. Data processing extensions enable Reporting Services to process data from a variety of data sources. Report rendering extensions process the Report Definition Language (RDL) and create reports in a predefined format (such as PDF, HTML, and Excel) specific to the particular rendering extension being used.

Figure 2 illustrates the architecture of Reporting Services. Using these extensions, you can turn your report servers into a complete reporting solution. Reporting Services will be able to process data from DB2 and SAP, to name two, and output the report in your preferred format.

Previously, to create a report using Reporting Services you used the Report Designer, which works with Microsoft Visual Studio® .NET 2003. In SQL Server 2005, you use the Business Intelligence Development Studio, which is installed when you select Reporting Services or Analysis Services during the installation of SQL Server 2005. RDL, an XML document with a specific schema used to store the definition of the report, can be created with the Report Designer or with other third-party tools.

Figure 2 Reporting Services Architecture
Figure 2 Reporting Services Architecture

Getting Started

Before you begin, you need to get your development environment ready. First, create a Visual Studio 2005 Class Library Project. (You cannot use Visual Studio .NET 2003 for creating rendering extensions for SQL Server 2005 but you can use it for creating rendering extensions for SQL Server 2000. An assembly that targets the .NET Framework 2.0 is the only supported assembly for all extensions of SQL Server 2005. In this article, I will be using Visual Basic®.) Next, add a reference to two of the Reporting Services assemblies, which are located at \Program Files\Microsoft SQL Server\ MSSQL.3\Reporting Services\ReportServer\bin (see Figure 3).

Figure 3 The Reporting Services Libraries (Close Figure 3)

A report renderer must implement the IRenderingExtension interface so that Reporting Services can recognize it as a renderer. The IExtension interface is also required to be implemented by all Reporting Services extensions. Implementing these interfaces requires you to implement five core members: the SetConfiguration method, the LocalizedName property, the GetRenderingResource method, the Render method, and the RenderStream method.

The SetConfiguration method inherits from IExtension interface and is used for processing any additional configuration data stored in the report server configuration file. LocalizedName is a read-only property derived from the IExtension interface. The string returned will be the user-friendly name of the Reporting Services Extension that is displayed to the user by Report Manager. The GetRenderingResource method comes from the IRenderingExtension interface and is used for rendering any embedded resources of a report, such as images and icons. The Render method is required for the IRenderingExtension interface and contains the logic used for report rendering. Finally, the RenderStream method is required for the IRenderingExtension interface. This method is used to support the report rendering method—Render—to return additional resources which are not created during the report rendering process, for example images in an HTML report.

The skeleton of a custom renderer should look like Figure 4. Before you write the code to render the report, you need to configure the report server to make sure it recognizes the existence of the new renderer and can be used in Report Manager. The first thing to do is to add code in the LocalizeName property, which returns a descriptive name that will be shown in Report Manager. You can return a different name of the renderer based on the CurrentCulture property of class System.Globalization.CultureInfo. Figure 5 shows an implementation of this property and the return name of the renderer based on the user's current culture.

Figure 5 Implementing LocalizedName (Close Figure 5)
Public ReadOnly Property LocalizedName() As String _
        Implements IRenderingExtension.LocalizedName
    Get
        Dim p_strName As String = "MSDN Magazine Custom Renderer"
        Dim p_CultureInfo As CultureInfo = CultureInfo.CurrentCulture

        ' Determine the text to be returned (displayed) by
        ' the Name property of CultureInfo class
        If p_CultureInfo.Name = "zh-HK" Then
            p_strName += " (Traditional Chinese)"
        ElseIf p_CultureInfo.Name = "zh-CN" Then
            p_strName += " (Simplified Chinese)"
        End If
        Return p_strName
    End Get
End Property
Figure 4 Skeleton Code for Custom Renderer (Close Figure 4)
Imports System.Collections.Specialized
Imports Microsoft.ReportingServices.Interfaces
Imports Microsoft.ReportingServices.ReportRendering

Public Class Renderer
  Implements IRenderingExtension

  Public Function RenderStream( _
      ByVal streamName As String, ByVal report As Report, _
      ByVal reportServerParameters As NameValueCollection, _
      ByVal deviceInfo As NameValueCollection, _
      ByVal clientCapabilities As NameValueCollection, _
      ByVal headerFooterExpressions As EvaluateHeaderFooterExpressions, _
      ByVal createAndRegisterStream As.CreateAndRegisterStream) _
          As Boolean Implements IRenderingExtension.RenderStream
      ... ' Code to render report
      Return False
  End Function

  Public Function Render( _
      ByVal report As Report, _
      ByVal reportServerParameters As NameValueCollection, _
      ByVal deviceInfo As NameValueCollection, _
      ByVal clientCapabilities As NameValueCollection, _
      ByVal headerFooterExpressions As EvaluateHeaderFooterExpressions, _
      ByVal createAndRegisterStream As CreateAndRegisterStream) _
          As Boolean Implements IRenderingExtension.Render
      ... ' Code to render the report
      Return False
  End Function

  Public Sub GetRenderingResource( _
      ByVal createAndRegisterStreamCallback _
      As CreateAndRegisterStream, _
      ByVal deviceInfo As NameValueCollection) _
          Implements IRenderingExtension.GetRenderingResource
    ... ' Render any embedded resource in the report
  End Sub

  Public ReadOnly Property LocalizedName() As String _
    Implements IRenderingExtension.LocalizedName
      Get
          ... ' Returns the name of this extension
      End Get
  End Property

  Public Sub SetConfiguration(ByVal configuration As String) _
    Implements IRenderingExtension.SetConfiguration
      ... ' Obtain configuration from RS config file
  End Sub
End Class

Once you have everything ready, compile the project and configure the report server to use this renderer. All Reporting Services extensions need to be placed in the \Program Files\Microsoft SQL Server\ MSSQL.3\Reporting Services\ReportServer\bin folder, so copy all your compiled assemblies there. Alternatively, you could set the output directory of your renderer project to this directory. If you do, make sure that you do not remove any references to the Reporting Services library, as doing so will also remove those library files from the build directory (the build directory in this case is the location of the Report Server used to store those libraries). This will remove library files from the report server causing it to fail. There will be a large number of files created after compiling the project, but you only need to copy files related to your project.

After placing the files into the Bin directory of your report server, you need to update two report server configuration files that allow Reporting Services to load and use this new renderer. In \Program Files\Microsoft SQL Server\MSSQL.3\Reporting Services\ReportServer, you'll find the configuration file, RSReportServer.config, to which you need to add an entry as a child node of \Configuration\Extensions\Render, like so:

<Extension Name="CUSTOM_RENDERER"
  Type="MSDNMagazine.CustomRSRenderer.Renderer,
  MSDNMagazine.CustomRSRenderer" />

The name attribute, which must be unique among all Reporting Services extensions, will be used when you want to tell the report server to render a report using this renderer. The type attribute tells Reporting Services which class to load and which assembly contains the class. You can also add another attribute to this element called visible. This attribute is set to true by default. Setting it to false allows the renderer to be referenced programmatically but keeps it hidden from the Report Manager.

Reporting Services uses Code Access Security to control the permissions granted to different components. All Reporting Services extensions need to run with full trust, so you need to add an entry in the report server's security configuration file to grant full trust to your custom assembly. To do so, open rssrvpolicy.config, which is located in the same folder as RSReportServer.config. Here is an example of the entry that you need to add as child node of \configuration\mscorlib\security\policy\PolicyLevel\CodeGroup:

<CodeGroup class="UnionCodeGroup" version="1"
    PermissionSetName="FullTrust" Name="Custom Render"
    Description="This code group grants Custom Renderer code full trust.">
    <IMembershipCondition class="UrlMembershipCondition" version="1"
        Url="C:\Program Files\Microsoft SQL Server\MSSQL.3\Reporting
        Services\ReportServer\bin\MSDNMagazine.CustomRSRenderer.dll" />
</CodeGroup>

Once everything is ready, you simply need to restart your Report Server service. You should then be able to see the new output format displayed in Report Manager.

Rendering the Report

When you create a report renderer, you need a component to parse the content of the RDL file. Luckily, Reporting Services has a built-in component that will parse the RDL and construct a hierarchy of data-bound objects with a top-level object, Microsoft.ReportingServices.ReportRendering.Report, that represents the report. When you're rendering the report, you can use this top-level object in order to retrieve all the report configuration information as well as its content.

In order to construct the report output, you need to add code to the Render and RenderStream methods. The Render method renders the body or content of the report, while the RenderStream method renders all the associated objects, such as charts and other embedded objects. For simplicity, I'll focus on the Render method.

A number of parameters can be passed to the Render and RenderStream methods which can affect the way the renderer renders the report. The reportServerParameters NameValueCollection is passed to the report server, typically for filtering. The deviceInfo NameValueCollection contains report properties, while the clientCapabilities NameValueCollection informs the renderer about client technologies such as scripting support. Figure 6 shows a sample implementation of the Render method. This will list all the values stored in these parameters as well as a sample output.

Figure 6 Render Method (Close Figure 6)
Public Function Render( _
    ByVal report As Report, _
    ByVal reportServerParameters As NameValueCollection, _
    ByVal deviceInfo As NameValueCollection, _
    ByVal clientCapabilities As NameValueCollection, _
    ByVal headerFooterExpressions As EvaluateHeaderFooterExpressions, _
    ByVal createAndRegisterStream As CreateAndRegisterStream) _
        As Boolean Implements IRenderingExtension.Render

    Dim s As System.IO.Stream = createAndRegisterStream( _
        "parameters", "htm", System.Text.Encoding.UTF8, "text/html", _
        true, StreamOper.CreateAndRegister)

    Dim sw As New System.IO.StreamWriter(s)

    sw.Write("<html><body>")
    sw.Write("<b><u>reportServerParameters</u></b><br>");
    For Each key As String in reportServerParameters.AllKeys
        sw.Write(key+"="+reportServerParameters(key)+"<br>")
    Next
    sw.Write("<br>");
    sw.Write("<b><u>deviceInfo</u></b><br>")
    For Each key As String in deviceInfo.AllKeys
        sw.Write(key+"="+deviceInfo(key)+"<br>")
    Next
    sw.Write("<br>")
    sw.Write("<b><u>clientCapabilities</u></b><br>")
    For Each key As String in clientCapabilities.AllKeys
        sw.Write(key+"="+clientCapabilities(key)+"<br>")
    Next
    sw.Write("</body></html>")
    sw.Close()

    return True
End Function

Besides these parameters, two very important delegates are also passed. The headerFooterExpressions delegate invokes a Reporting Services internal function to determine if a page needs to include a header or footer when the page header or footer depends on the content of the page. The createAndRegisterStream delegate invokes a Reporting Services internal function to create a stream to which the renderer writes the rendered report. The stream created acts as the output stream and is ultimately sent to the client.

Now that you know what kind of information is sent to the renderer, you can start writing the rendering code. Before I talk about rendering a specific kind of object in the report, let's look at rendering a report using the Render method. First, invoke the createAndRegisterStream delegate. This will create a stream to which your renderer will write the rendered report to pass to the client. Second, use the information stored in the report object to construct the report. Of course, the report object passed to the Render/RenderStream method represents the report that the Reporting Services user selected. You may also need to make use of other information passed to the Render method to control the look of your report. For example, you probably don't want to render any JavaScript in your HTML report if the clientCapabilities object tells you the client does not support JavaScript.

The following is an example of how to use the createAndRegisterStream delegate to create an HTML stream:

System.IO.Stream s = createAndRegisterStream("sample", "htm",
    System.Text.Encoding.UTF8, "text/html", true,
    StreamOper.CreateAndRegister);

At this point you can use a variety of techniques to add content to the stream. For simplicity, I will be using a StringBuilder to construct the HTML output; you should use HtmlTextWriter in order to construct the HTML in your production report renderer.

Now that you have created an empty report, let's look at how to render the content of the report. I will focus on rendering Matrix and Textbox objects to the report output. There are lots of objects available in the Report Designer, and all of them must be handled accordingly since their behaviors differ quite a bit.

Rendering a Report Object

I've selected Matrix as an example to render because it's one of the most widely used objects in Reporting Services. Most of my team's projects use matrices to present business information.

The report object is passed into the two Render methods. The Report object's Body property contains the information relating to the Body, such as formatting. It contains a property, ReportItemCollection, which holds references to all rendered instances of the report elements in the Body section of your report in the Report Designer. To get the objects contained in the Body section of the report, each of which is a ReportItem, you need to loop through the ReportItemCollection. You need to cast to a ReportItem-derived class in order to access specific properties related to each element. This information can then be used in your code to render the report element to the output.

All report element types such as Matrix and Textbox are derived from the ReportItem type, and each container object exposes a property to allow access to the contained child objects. As with the Body's ReportItemCollection, these must be cast to a ReportItem-derived type to access properties specific to those types. The code in Figure 7 enumerates all of the elements contained in the Body by using the ReportItemCollection property.

Figure 7 Enumerate Elements (Close Figure 7)
Public Function Render( _
    ByVal rpt As Report, _
    ByVal reportServerParameters As NameValueCollection, _
    ByVal deviceInfo As NameValueCollection, _
    ByVal clientCapabilities As NameValueCollection, _
    ByVal headerFooterExpressions As EvaluateHeaderFooterExpressions, _
    ByVal createAndRegisterStream As CreateAndRegisterStream) _
        As Boolean Implements IRenderingExtension.Render

    For Each rptItem as ReportItem in report.Body.ReportItemCollection
        ... ' Implement your report element rendering code here
    Next

End Function

I suggest you create an individual object to handle each report element (one class for Matrix, one class for Textbox, and so on) for ease of maintenance and development. When rendering the report, you need to pay attention to the structure of the matrix. For example, first you render the Corner, then the Row Member Collections together with the Column Member Collections, and lastly, you render the content of the cell.

One of the challenges I faced when rendering the matrix is that I first needed to determine the levels in the row member collection and column member collection. To compensate, I loop through all levels of the row and column member collection to find out the RowSpan and ColSpan of the corner (that is, the number of columns in the row member collection and the number of rows in the column member collection).

When drawing the content of the matrix to the report output, you need to make use of the Cell property of the matrix object. In the case of matrix, the cell will be a textbox object by default. Any other kind of object, like Chart or Table, can also be returned by this property. Some checking should be in place to ensure that you use the right internal renderer to render the returned object. For simplicity, I assume all cell properties will return the textbox object. A textbox rendering class is created to handle the rendering job of the textbox. You can also use your own custom code to handle the textbox in the matrix, but you really should make it a separate renderer so that it can be reused later on.

One interesting thing I found in the Style property in all kinds of ReportItem objects is that the formatting information is stored in a key-value pair collection (similar to a Hashtable). Note, however, that the key used by Reporting Services is for internal use (or more specifically, for the use of the report renderer). They are not necessarily aligned with the property names used in HTML. The rendering object model property names match the RDL property names. Various rendering targets all have their own property names, many of which don't match each other, so it makes sense that the RDL property names don't match the HTML property names. In the sample code that comes with this article, I created a class to convert the Reporting Services styles to those used in HTML.

Figure 8 shows a sample Textbox Renderer together with style handling functions for converting styles stored in the Style property of objects representing elements.

Figure 8 TextBoxRender (Close Figure 8)
Friend Class TextBoxRender

    Friend Sub New()
    End Sub

    Friend Function Render(ByVal textBox As _
        Microsoft.ReportingServices.ReportRendering.TextBox) As String
        Dim pHtml As New System.Text.StringBuilder
        Dim sh As New MSDNMagazine.CustomRSRenderer.StyleHandler( _
            textBox.Style)

        pHtml.AppendFormat("<DIV STYLE=""backgroundcolor:{0}"">", _
            sh.GetStyleColorRGBCode("BackgroundColor"))
        pHtml.AppendFormat("<FONT FONT=""{0}"" SIZE=""{1}""" & _
            "COLOR=""{2}"" STYLE=""font-size:{1};"">", _
            sh.GetStyleString("FontFamily"), _
            sh.GetStyleFontSize.ToPoints().ToString() + "pt", _
            sh.GetStyleColorRGBCode("Color"))

        If sh.GetStyleFontStyle = FontStyle.Italic Then
            pHtml.Append("<I>")
        End If
        If sh.GetStyleFontWeight = FontStyle.Bold Then
            pHtml.Append("<B>")
        End If
        If sh.GetStyleTextAlignment = ContentAlignment.MiddleCenter Then
            pHtml.Append("<CENTER>")
        ElseIf sh.GetStyleTextAlignment = ContentAlignment.MiddleLeft _
            Then
            pHtml.Append("<DIV ALIGN=""LEFT"">")
        ElseIf sh.GetStyleTextAlignment = ContentAlignment.MiddleRight _
            Then
            pHtml.Append("<DIV ALIGN=""RIGHT"">")
        End If

        pHtml.Append(textBox.Value)

        If sh.GetStyleTextAlignment = ContentAlignment.MiddleCenter Then
            pHtml.Append("</CENTER>")
        Else
            pHtml.Append("</DIV>")
        End If
        If sh.GetStyleFontWeight = System.Drawing.FontStyle.Bold Then
            pHtml.Append("</B>")
        End If
        If sh.GetStyleFontStyle = System.Drawing.FontStyle.Italic Then
            pHtml.Append("</I>")
        End If

        pHtml.Append("</FONT></DIV>")

        Return pHtml.ToString()
    End Function

End Class

Let's Try It

Once everything is ready, compile the code and then copy the generated assemblies to the bin directory of the ReportServer directory. One simple way to test if the renderer is working correctly is to use Report Manger. To do so, first select the report that you want to use with the renderer, select the name of the renderer you created, and then click Export. A new window will be launched which will display the rendered and downloaded report.

Using our sample renderer, you should then see the screen shown in Figure 9—a user-friendly HTML error message telling you an error occurred in your extension code. Unfortunately, user-friendly error messages are usually not very useful to developers. However, there is some valuable information in the HTML source of the user-friendly error screen. If you open and view the HTML source of the error page, inside the head tag, the stack trace of the error is stored as a meta tag which can aid in debugging the program. This is especially useful when the renderer is deployed to the machines in the production environment.

Figure 9 User-Friendly Error Message
Figure 9 User-Friendly Error Message

Another place to look when you reach an error is the log file located at \Program Files\Microsoft SQL Server\MSSQL.3\Reporting Services\LogFiles. In order to use the debugging features provided by Visual Studio, you have to configure and test the renderer with Reporting Services running on your development machine. After adding the configuration to the RSReportServer.config for the renderer, try to render the report using the preview report function in Visual Studio. Doing so means the debugger can be attached to your custom renderer.

If everything has gone well, you should be able to see the report rendered as shown in Figure 10. Figure 11 shows the same report rendered by the HTML renderer that comes with Reporting Services. As you can see the rendered report is very similar to the one rendered by Reporting Services.

Figure 11 The Rendered Report in HTML (Close Figure 11)

Figure 10 The Rendered Report (Close Figure 10)

You can use this custom HTML renderer to render an Excel file. The trick is that Excel is able to open an HTML file and display it in a workbook style. To start, change the parameters that are passed to the createAndRegisterStream delegate, as shown in the following code:

createAndRegisterStream(report.Name, "xls", System.Text.Encoding.UTF8, _
    "application/vnd.ms-excel", true, StreamOper.CreateAndRegister);
Doing so makes the report server think that this is an Excel file, which tells the browser that the rendered report is an Excel file even though the content is HTML. The browser will then use Excel to open the rendered report, making it appear to the user that the report is an Excel file rather than an HTML file. This approach does have some limitations when compared to a native Excel report renderer, such as the lack of image and multiple worksheet support. To render a native Excel, Word, or PowerPoint® file, you can use the Office Automation in your report renderer, or a third-party component for construction of the type of document required.

Conclusion

After mastering the basics for creating a renderer for Reporting Services, you can create renderers that render reports into the format of your choice, such as Word. Once more, although this article focuses on Reporting Services in SQL Server 2005, all the techniques I've described are applicable to Reporting Services in SQL Server 2000. Have fun creating your own report renderers!

James Yip, MCT, MCSE, MCDBA, MCSD, MSF Practitioner, is a consultant with Edvance Limited, a Hong Kong-based technologies consulting company. James is also a trainer teaching MOC classes in both Windows Server Systems Administration and .NET development for a local Microsoft Certified Partner - Learning Solution.


Testing: Get Your Customers Involved in the Testing Process with Functional Tests in Excel

Testing

Get Your Customers Involved in the Testing Process with Functional Tests in Excel

Will Stott
This article discusses:
  • Why you need to do functional testing as well as structural testing (NUnit)
  • How to improve the communication value of specification documents by having customers test the code using FIT
  • WinFITRunnerLite, a program that allows Excel files to be executed as functional tests of your software
  • A sample development episode showing how to test software using FIT and WinFITRunnerLite
This article uses the following technologies:
Visual Studio, C#, Testing, Excel
Code download available at: ExcelUnitTests.exe (800KB)
Contents
Customer Tests
What is FIT and How Does it Work?
Using FIT to Perform Functional Testing
Writing Tests in Excel
WinFITRunnerLite
A Sample Development Episode
Conclusion

M
any specification documents don't provide the full story about the products that they define. Often no one notices this until a significant amount of development time has gone into the project. For specifications to be truly valuable, they need to give an accurate picture of all the requirements of a project.

In this article I'll demonstrate how the communication value of specification documents can be improved by permitting users to test the code under construction using the Framework for Integrated Test (FIT), an open-source tool. I'll explain how you can build a Windows® Forms application in C# (WinFITRunnerLite) that converts functional tests, as written by your customers using Excel, into a form that allows you to run them with FIT against the code you're developing. I'll also present a practical development example to explain how you can set up and perform this sort of customer testing using FIT and WinFITRunnerLite.

Customer Tests

My previous article for MSDN®Magazine (Improve the Design and Flexibility of Your Project with Extreme Programming Techniques) explained how to perform automated unit testing using NUnit. This sort of structural (glass-box) testing is for developers. It validates the operation of code at the level of classes and their methods. However, structural testing alone is not sufficient to ensure that you can release software into production confident that it will deliver good value to the business.

Consider a method that calculates percentages that are accurate to two decimal places. A programmer test proves its behavior, so you send the software to your client certain that it will correctly calculate the sales tax for his invoices. However, the next day he's on the telephone complaining because the total sales tax on his invoices doesn't agree with the sum of sales taxes for each individual item. This is a situation where you could have run a simple functional test to prove that the tax was calculated correctly. Let's now look at how you might perform such a function test in two different types of projects.

In an Extreme Programming (XP) project, you sit down with the customer to discuss what the software must do; in this case, calculating the sales tax on an invoice. Together you work out the details of the customer's needs, his "story." The customer frames his requirements for each story in terms of functional tests, known in XP as acceptance or customer tests. Developers start work implementing stories with an understanding of the tests that are significant to the customer. The story itself is only deemed to be complete when it passes both its unit and customer tests. In agile development projects that use techniques such as XP, these customer tests are valued throughout the development cycle for their importance in helping customers communicate their requirements to developers for setting acceptance criteria for the finished software.

Functional tests are called black-box tests because they are written without any knowledge of how the software works. They simply define what the software must do in terms of measurable responses to certain external input in a given environment. This approach to testing is the basis of the customer test, as well as the foundation of many other types of tests performed on a software product during its development.

While it is your job as a developer to decide how the software will work, it is the customer's responsibility to know what the software must do, so he must have some influence on the writing of functional tests. In traditional projects, the involvement is indirect, proceeding as it does through the business analyst and tester. However, I believe that the agile approach of involving customers directly in the writing of functional tests leads to the development of more useful software that does what it's supposed to do.

What is FIT and How Does it Work?

The Framework for Integrated Test allows customers to write their tests in the form of HTML tables that can then be executed directly against the code under test. For example, a customer can use Microsoft® Word to create a specification document containing various tests (entered into tables), as well as input values and expected results, as shown in Figure 1.

Figure 1 FIT Input - FIT Test Table in Word
Figure 1 FIT Input - FIT Test Table in Word

This HTML document can then be used as an input file for FIT, which parses the HTML and matches each table to a corresponding test method in the software test code, as discovered by reflection. FIT invokes the test method, passing parameters read from the HTML table and runs the test. The results are captured into a HTML output document of the same form as the input file, but with any errors added into the cells of each table. The customer can see the results of the various tests by opening the output file in their browser. Figure 2 shows that such a test failed because the customer incorrectly calculated the test result. I'll explain the FIT testing process and how it relates to Excel later in the article, but for now I want to concentrate on the form of the tests, both in the HTML document and in the software test fixture.

Figure 2 FIT Output - HTML File Displayed in Browser
Figure 2 FIT Output - HTML File Displayed in Browser

A test fixture is a bit of code that supports a test. Such code is usually held in test libraries (DLLs) separate from your domain code libraries, so they don't get released into the production environment. You keep everything that makes up the software under test in the domain code libraries. In your test environment, FIT is told where it can find these test libraries and discovers their classes and methods using reflection. It then runs a particular test by instantiating the appropriate test object and invoking the necessary method to execute the test fixture, passing and receiving the parameters as specified in the HTML test specification document. In turn, the code forming the test fixture creates the objects from the domain code libraries and invokes their methods, passing and receiving data, and therefore exercising the software under test.

To write a functional test, the customer creates a table using whatever tool is appropriate (Word or an HTML editor). The customer enters the name of the test fixture and a specification of the test, specifying the responses expected given certain inputs. Although the format of the test fixture name must conform to a particular convention, the name itself along with those used in the rest of the specification is expressed in language that the customer understands (see Figure 1). Similarly, the format of the table is not difficult for the customer to comprehend as there are only a few possible variations, which are related to the three main base types of the FIT fixture used in such tables: ColumnFixture, RowFixture, and ActionFixture.

A feature of customer tests is that they require considerably fewer different test fixtures than you might discover in the same project for structural testing (such as with NUnit) as the same test fixture also tends to be reused by several functional tests. This means that once developers have supplied some initial FIT test fixtures, the customer is able to write functional tests without having to wait for a specific fixture to be developed for each table.

Writing a test fixture for FIT is much like writing a test fixture to support NUnit testing. A fixture class is created in the test library derived from a Fixture base class appropriate for the type of tables in which it will be used. Methods for this class are then created to correspond to the requirements of the table's functional tests and implemented using the domain code libraries. You can, of course, create additional base fixture types beyond those supplied by FIT because all the source code is included in the download. This gives your customers the flexibility to create different types of tables for their functional tests. However, the supplied base fixture types are often all that is required to provide adequate functional testing capability for a project.

Using FIT to Perform Functional Testing

To use FIT, you first need to download it (from fit.c2.com) and install it in your test environment. Ask your customer to write the functional tests using Excel and then write the necessary test fixtures using an appropriate development environment (such as Visual Studio®) and language (such as C#). Lastly, build WinFITRunnerLite and then run the tests that are written in Excel directly from your desktop.

I used the version of FIT called fit-b030825cs.zip for this article. In FIT's source\fit directory, you'll find the base classes corresponding to the three main types of tables: ColumnFixture.cs, RowFixture.cs, and ActionFixture.cs. These three base classes are the only part of the FIT source code that you really need to know about; they are necessary for creating the test fixtures that are required to support your customer's tests.

Each of the three table types is useful for testing one of the three layers of architecture commonly found in today's software: the business layer, data layer, and user interface layer. This might appear to break the concept of black-box testing, but in practice the customer's view of the system doesn't have to correspond with the actual implementation. Using these three types of tables, the system can be modeled in terms of the parts that customers can see (the user interface layer), the business rules that they defined (the business layer), and the important data they expect to exist in the system (the data layer). Creating such a model of the software's structure can really help the customer understand the tests they are performing. Let's take a closer look at the tables.

Figure 3 Column Fixture Table
Figure 3 Column Fixture Table
Column Fixture Tables This table type is intended to test the program's business logic. For example, if you had derived your FIT fixture from ColumnFixture and called it TaxCalcFixture, then the table in Figure 3 might emulate performing tax calculations for various employees with different tax codes and earnings in order to test the program's tax calculation logic.

There is no requirement for the names of the test fixture methods and their parameters to correspond to those found in your domain code libraries. You should express names in the language of the problem domain so that the customer can understand the test without needing to know the underlying software structure.

In the table, the first row contains a reference to the FIT fixture developed for your project to support the tests in this table. The second row contains headers for the subsequent rows which map each column to the name of a particular method or input field. The remaining rows contain data for each iteration of the test, listed in the order of processing. For tables that have more than one method in the header row, methods are invoked in sequence from left to right. The input data is given in the field name columns, such as TaxCode, and the expected output data is given in the method name columns, such as CalcTax.

Row Fixture Tables This type of table is used to validate collections of data held by the program. For example, if you derived your FIT fixture from RowFixture and called it StaffDataFixture, Figure 4 might represent the collection of staff data known to the program (as set by a previous test).

Figure 4 Row Fixture Table
Figure 4 Row Fixture Table

Row-type tables look similar to column-type tables except that if any rows in the table are not found in the program's DataSet, then a "missing" error is reported. Likewise, a "surplus" error is reported should the program's DataSet contain more records than are listed in the table. The header row (row two) contains the names of methods that reveal the various properties of each data item found in the collection, while any fields in the header are used for sorting the collection (left-most columns have primacy).

Action Fixture Tables This table is designed to allow emulation of the user interface so you can test how the program responds to certain external inputs. For example, if you had derived your FIT fixture from ActionFixture and called it NewStaffDlgFixture, then Figure 5 might emulate a particular user interaction with one of your program's dialog boxes, the one called NewStaffDlg.

Figure 5 Action Fixture Table
Figure 5 Action Fixture Table

The general layout of Action tables is shown in Figure 6. The first column defines a command (start, enter, check, or press) and the contents of the remaining columns depend upon its type.

Figure 6 Action Table Commands (Close Figure 6)

Command | Arguments |  | Description
start | myClass |  | myClass defines the fixture in use
enter | inputA | value1 | myClass.inputA (value1)
press | doB |  | myClass.doB()
check | methodC | result2 | assert (myClass.methodC() == result2)

There is no reason why your domain code library should contain a class called NewStaffDlgFixture that implements this dialog box. It's enough that the customers and developers can agree that this FIT fixture emulates the behavior of whatever dialog box in the program is known to the customer as NewStaffDlg.

Writing Tests in Excel

Excel provides extensive functionality for manipulating data. Customers can quickly generate values for tests with features like "fill series" and "import external data," so it is sensible to use Excel for writing functional tests. It is particularly appropriate to employ Excel when customers are testing complex business logic as the calculations can be modeled in the worksheet and then used to provide the expected results of a test.

Although you can save your Excel file in HTML format, the resulting file is not suitable for use as an input file for FIT, so some form of conversion is going to be required. Fortunately, WinFITRunnerLite will perform this conversion if you create your tables just like the one shown in Figure 7.

Figure 7 ColumnFixture Table in Excel
Figure 7 ColumnFixture Table in Excel

WinFITRunnerLite reads all the worksheets in the workbook file and scans each one for tables with the following features:

  • $TEST$ in the table's top-left cell (B3) identifies it as being for a FIT test. This allows you to put other types of information in your worksheet besides tests.
  • The first blank cell in column one (B11) defines the number of rows in the table, whereas the first blank cell in row two (E4) defines the number of columns.
  • The remaining cells in the top row identify the name of the test and the name of test fixture that's needed to run it. The name of the test ("2004 Tax Calcs" in this example) is used for identification purposes only.
  • The second (header) row associates the columns with the fields and methods of the test fixture.
  • All subsequent rows contain input and return values for each iteration of the test according to the various test fixture methods as specified by the header (second) row.

Worksheets are read from left to right and from top to bottom in search of appropriate tables that correspond to functional tests using the ColumnFixtures, RowFixtures, or ActionFixtures types. These tests are executed in the order in which they are found. Further examples of tests in an Excel worksheet are given in the sample development episode section at the end of this article.

WinFITRunnerLite

WinFITRunnerLite provides for customer tests what NUnitGUI does for programmer (unit) tests. It collects all the customer tests for a particular project from one or more Excel workbooks and allows the customer to run them with a single click. The customer can easily see which tests passed or failed, as well as the actual results of each test. Because the tool is designed for use by customers, it looks like any other Windows-based application and is operated entirely from the desktop.

I was developing the program using test-driven development (see my previously mentioned article) and therefore created a Visual Studio solution containing three projects: WinFITRunnerLite, FITRunLite, and FITRunLiteNUnit. WinFITRunnerLite is a C# Windows-based application containing the program's main entry point, its Main form, the About dialog box, and not much else. FITRunLite is a C# class library containing the domain code that does all the work. FITRunLiteNUnit is a C# class library, a test-code library containing all the fixtures and other code needed to unit test the domain library (FITRunLite).

WinFITRunnerLite contains two key files: WinFITRunnerLite.cs and MainForm.cs. The program's controller class is implemented in WinFITRunnerLite.cs and contains the app's entry point, Main, and holds a single instance of its document class. Because there can be only one instance of the program, it was developed using the Singleton pattern, such that all classes accessing it must go through the static Instance property which does much the same job as the MFC AfxGetApp function. Among the other methods that you'll find in the WinFITRunnerLite class are ErrorMsg, for displaying errors to the user, GetDocument, for accessing the document object, and methods to save and load the document from a file. WinFITRunnerLite derives from the Program class in the FITRunLite domain library.

MainForm.cs implements the application's main window and contains its menu as well as its TreeView and TabControl Page, as shown in Figure 8. This was developed using the Visual Studio graphical editor in a way that will be familiar to anyone with experience building Window Forms applications. The only component that you might not find in your graphical editor's toolbox is the DHTML edit control used to display HTML files in the TabControl Page. You can add this component by right-clicking your toolbox, selecting the Add/Remove Items menu, and then locating the "DHTML Edit control for IE5" in the COM components.

Figure 8 WinFITRunnerLite - Main Window After Running a Test
Figure 8 WinFITRunnerLite - Main Window After Running a Test

The TreeView and TabControl Page present two views of the data held in the program's document. These two views are managed by the variables m_leftView and m_rightView, which are initialized in MainForm's constructor, with the form's TreeView and DHTML edit control being passed to the constructors of the LeftView and RightView classes, respectively. These classes are implemented in the domain library.

The application's views are updated in the MainForm_Update method whenever the document changes. This method invokes the Update method of the LeftView and RightView objects as well as driving an update of the form's menus and title bar. Depending on the state of the document object (and item selected in the LeftView), menus will be enabled or disabled. The title bar will also indicate the name of the file and whether it has been altered since last saved. The remaining methods in MainForm are concerned with handling menu activation events as well as handling a couple of events generated by the views, such as the TreeView selection. Most of these various events are deferred to objects in the domain library for processing.

WinFITRunnerLite contains code that will not be unit-tested automatically with NUnit, so this GUI layer is made as thin as possible. The design of MainForm was not driven by tests because most of the work involved declarative programming using the graphical editor. I've found it is more productive to let customers influence such parts of the design than to drive it with worries about creating a set of cohesive and loosely coupled objects. Test-driven development purists may resist this approach, but I've found it to be more pragmatic.

FITRunLite is the domain library and contains the classes shown in Figure 9.

Figure 9 FITRunLite Classes (Close Figure 9)

Class | Description
Document and its associated classes (Project, TestDoc, TestSuite, TestCase, TestCol, TestRow, and TestCell) | Represents the application's data, which is persisted as XML in its "fit" file.
LeftView and RightView | Concerned with the behavior of the application's views of its data. They are updated whenever it changes.
DlgProject and DlgTestDoc | Simple data entry type forms, each initialized by passing an appropriate data object to their constructor. This object is updated with information supplied by the user when their Update method is called.
DlgTestRun | Implements the Windows Form that is opened whenever the tests are run. It is responsible for creating the TestRun object and periodically invoking its DoTimerEvent method to drive the execution of tests.
TestRun | Responsible for iterating through the collection of test suites held in the application's document, preparing each one for execution by FIT, and then starting the process for runFile.exe that will run the test cases (tables) it contains.
SpreadsheetApp | An interface for the Excel11Adapter class, which uses the adapter pattern to make Excel object library v11.0 fit the Spreadsheet class.
Spreadsheet | Modeled after the Facade pattern. This is how the program reads the data in an Excel worksheet into a collection of Table objects (one for each test table found). Table objects are then used to update the application's document.
Program | An abstract class that forms the base class for WinFITRunnerLite. It holds the program's instance variable and makes it available to classes in the domain library using the static m_instance.

The application's document has one project containing a collection of test documents (Excel files). Each test document has a collection of test suites (Excel worksheets), which in turn has a collection of test cases (one for each table found in the worksheet). A test case has a collection of columns (table headers) and a collection of rows each containing the data associated with an iteration of the test. A test row has a collection of test cells containing input and output data for the test instance.

The classes corresponding to these data objects are self-evident from their names and use the .NET XML serialization classes to save and load themselves from a file, which I defined as being of the type .fit. You can easily serialize your documents to an XML file using the .NET Framework by adding some attributes to each class contained in the document, as shown here:

[XmlRoot("TestCell")]
public class TestCell
{
   private string m_Expected;

   [XmlElement("Expected")]
   public string Expected
   {
     get { return m_Expected; }
     set { m_Expected = value; }
   }
}
You then implement the document's save and load methods (see the code in Figure 10).

Figure 10 Document.cs (Close Figure 10)
public bool SaveAs(string filename)
{
   bool rc = false;

   FileStream fs = null;
   try
   {
      XmlSerializer serializer = new XmlSerializer(typeof(Document));
      fs = new FileStream(filename, FileMode.Create);
      serializer.Serialize(fs, this);
      m_Dirty = false;
      SetFileName(filename);
      rc = true;
   }
   catch(Exception e)
   {
      Program.m_instance.ErrorMsg("Document.SaveAs()", e.Message);
   }
   finally
   {
      if ( fs != null)
         fs.Close();
   }
   return rc;
}

Any changes to the document's data causes a refresh of the application's views by invoking the NotifyDoc method of its base class, DocumentItem. However, not all the document data needs this notification mechanism, because data read from an Excel workbook file will not change unless the file itself changes. It is therefore sufficient to provide notification for Project and TestDoc objects and refresh all data items below TestDoc in the data hierarchy whenever the associated file is changed.

TestDoc uses the Program.GetHashForFile static method to calculate a hash value for the Excel file it references. When this hash value changes indicating that the file has changed, the Excel file is read and the tables it contains are converted into TestCase objects. The Spreadsheet class provides the GetTestSuiteNames method to obtain a list of the worksheets found in the file, and the GetTables method creates an array of Table objects from the test cases found in a given worksheet. All the complex classes in the Excel object library are hidden behind this simple facade. A Table object contains the address of the test case in the worksheet and an array of TableRows, each holding an array of strings corresponding to the contents of the cells found in a particular row of the test case. When you have a collection of Table objects, using them to reconstitute the document's TestDoc object is quite simple.

The Spreadsheet class does not itself access the Excel object library, but uses four small classes implementing interfaces defined in the abstract classes SpreadsheetApp, SpreadsheetFile, SpreadsheetPage, and SpreadsheetArea. Each of these interfaces contains three or four methods so supporting a different object library involves changing only a small, isolated part of the codebase. I decided to use Excel object library version 11.0 when building WinFITRunnerLite, but you might have a different version of the object library on your machine and therefore may need to create different types of adapter classes.

To create Excel11Adapter, I added a reference to the Excel object library as found in the COM tab page of the Add Reference dialog. To select FITRunLite references for your Visual Studio project, right-click Add Reference in the Solution View. I created a class derived from SpreadsheetApp and added "using Excel;" to the top of its file. Then I implemented the constructor so that it would create a single instance of the Excel application object. Finally, I wrote the methods and property defined by the interface: IsInstalled, OpenFile, and Name. Figure 11 shows a simplified version of my OpenFile implementation that just adapts the given Excel class for use with the Spreadsheet class. If you need to create your own adapters, you'll be glad to discover that the unit tests needed to validate them are supplied in the code download.

Figure 11 Excel11Adapter.cs (Close Figure 11)
public override SpreadsheetFile OpenFile (string Filename)
{
   Excel.Workbook openbook = null;

   object OpenMode = true;
   openbook = m_excel.Workbooks.Open(Filename,
    Type.Missing, OpenMode, Type.Missing, Type.Missing,
    Type.Missing, Type.Missing, Type.Missing, Type.Missing,
    Type.Missing, Type.Missing, Type.Missing, Type.Missing,
    Type.Missing, Type.Missing);

   return new Excel11Workbook(openbook);
}

All the test cases contained in the application's document are executed whenever a user opens the DlgTestRun dialog box. This creates an instance of TestRun and initiates a timer to invoke the dialog's TimerEventProcessor method every 500 ms. The timer event invokes the TestRun.DoTimerEvent method which determines whether a test suite is executing; if no suite is currently running, it finds the next test suite, prepares it for execution, and then starts the process for FIT's runFile.exe to run the test. Preparing the test suite for execution involves writing its data to a temporary file and then using the project's XSLT file to translate this XML file into an HTML file suitable for FIT.

When TestRun.DoTimerEvent can find no more test suites to run, it changes the Project's LastTestRun data. This generates an update of a result file displayed on the right corresponding to any test suite selected on the left. DoTimerEvent completes by returning null, which causes the dialog box to close.

The domain library is small and contains simple code. To understand it, look at the unit tests in FITRunLiteNUnit and experiment a little by adding your own tests or altering the domain library.

A Sample Development Episode

Now that you've installed FIT and built WinFITRunnerLite, let's see how these tools might be used by a customer to create functional tests with Excel that can be executed directly against a program that is under development. This sample development episode will also describe how the developer can support such testing.

The program under test is a payroll application for calculating the pay-packets for a team of workers paid weekly. You join the project at the start of its development, so the codebase is simple and quite small. Your Visual Studio solution already has three projects: PayApp (the Window Forms application), Payroll (the domain code library), and PayrollNUnit (unit tests for the library code). All these project files, as well as their source code, are available for download and follow the general approach for test-driven development explained in my previous article.

The episode starts with the customer, Bob, writing tests for parts of a story he's already discussed at great length with you. When implemented, the plan will save the business $150 a week by automating part of the company's payroll system. In the first part of the story, Staff Info, the program needs to hold personal details about team members, such as staff number, name, and tax codes. The second part, Tax Calc, is concerned with the program calculating the tax deductible for a given member of the staff, which is determined by their gross weekly pay.

Bob had previously explained the rules for calculating tax and he now wants to check that they are correctly implemented in the program. He starts Excel and writes out a test, as shown in Figure 7. It's not a particularly comprehensive test, but it's enough to reassure him that you've done some work on the project. At this stage, Bob doesn't know the name of the test fixture, PayrollFIT.Tax04Fixture, so he leaves the cell blank. He saves the workbook as "Payroll.xls" in a shared folder called PayrollFIT on the team's test PC. Bob then goes to this test PC, starts WinFITRunnerLite, and goes to File | New to open a dialog box, allowing him to define a new project called Payroll. He enters the details that you gave him about the program, shown in Figure 12, into this dialog box and then presses OK.

Figure 12 WinFITRunnerLite Project Details (Close Figure 12)

Field | Contents | Comment
Name | Payroll | Appears as the root item in the TreeView
Test Tool exe | c:\PayrollFIT\Bin\runFile.exe | Standard FIT application
FIT test fixtures dir | c:\PayrollFIT\Bin | Location of test fixtures you've developed
XML to HTML | C:\WinFITRunnerLite\FITLite.xsl | WinFITRunnerLite installation directory

Next, Bob clicks the WinFITRunnerLite menu bar and selects Project | Add File to open the Test Doc dialog box, so that he can add a reference to Payroll.xls, the Excel file containing his tests. He specifies the directories to be used for the FIT input and output files generated when the tests are run as being the subdirectory Temp of his PayrollTests folder. When he closes the dialog box the tree view is updated to show the Excel file and the worksheets it contains. Bob then saves this information in the WinFITRunnerLite file Payroll.fit and runs the tests (Project | Run Tests). Not surprisingly, his test fails because the test fixture hasn't been written yet.

You compare Bob's test in Payroll.xls to those in Figure 3 through Figure 5 and decide that it most closely resembles a test of business logic and needs a test fixture derived from the ColumnFixture class (as supplied by FIT). However, as you are supporting the project's initial customer test, you must first create a library (.dll) for this and subsequent FIT fixtures. Go to the Visual Studio menu and select File | New | Project to open a dialog box that allows you to select Class Library from a Visual C#® projects folder. You then enter PayrollFIT in its name field, and click Add to Solution, before finally closing the dialog box in order to create the project as well as its preliminary class, Class1.

In order to derive your class from ColumnFixture, you need to add a reference to the library, FIT.dll, as supplied with FIT. You can do this from the Visual Studio solution view by selecting the References folder for your PayrollFIT project, opening the Add Reference dialog box, and then using its Browse button to locate the correct file. You may also need to follow the same procedure for NUnitCore.dll, located in FIT's tools\nunit folder. Once you've added "using fit;" to the top of the file containing Class1, your code will have access to ColumnFixture and the rest of the classes that are contained in the FIT libraries.

You rename Class1 to Tax04Fixture and enter the code shown in Figure 13 in order to implement the method and fields defined by Bob in his customer test: CalcTax, TaxCode, and GrossPay. Assuming you've already built the TaxDetails04 class in your domain code library and unit-tested it to ensure it implements the tax calculation logic according to your understanding of Bob's instructions, all that remains for you to do is add to the project a reference to payroll.dll and build it (Build | PayrollFIT).

Figure 13 Tax04Fixture.cs (Close Figure 13)
using System;
using fit;
using Payroll;

namespace PayrollFIT
{
   public class Tax04Fixture : ColumnFixture
   {
      public int TaxCode;
      public int GrossPay;

      public int CalcTax()
      {
         int rc = 0;

         TaxDetails04 tax = new TaxDetails04(TaxCode);
         rc = tax.WklyIncomeTax(GrossPay);

         return rc;
      }
   }
}

Deploying your FIT fixture simply requires you to copy the contents of the directory PayrollFIT\bin\debug from your development PC to the shared c:\PayrollFIT\Bin directory on the team's test PC. Make sure you include all the libraries referenced by PayrollFIT.dll as well as the runFile.exe file supplied with FIT. Once this has been done, Bob can add the test fixture reference to his test, (see Figure 7) and rerun the tests. This time they pass, so he's satisfied that the development team is making progress.

Next Bob is concerned with verifying that the system is able to hold details about his staff—the StaffInfo. After talking to Peter, the team's testing specialist, he decides to write two tests that will be run consecutively. The first test will enter staff information into the program and the second test will confirm that the program holds only these records. This first test simulates the NewStaffDlg dialog box that the team's user interaction specialist showed him earlier that morning, so he creates a new worksheet in Payroll.xls called Data Tests and creates a test, using an Action Fixture Table as his template. His second test is based on the Row Fixture Table described by Peter and contains a set of rows, one for each row in the first test that contains "Press OK"—a signal that a new staff record should be created using the data in the preceding two rows.

Bob sends you the Payroll.xls and asks you to build the necessary test fixtures. You open your PayrollFIT project in Visual Studio and add a new class called NewStaffDlgFixure just as you did for his previous test, except this time you derive it from FIT's ActionFixture class. You flesh out the class with the code shown in Figure 14 in order to implement the fixture. The methods for Forename and Taxband save their input parameters in private data members, so the OK method can create a new Staff object using this information and then store it in the collection of Staff objects held by the program.

Figure 14 NewStaffDlgFixture.cs (Close Figure 14)
using System;
using fit;
using Payroll;

namespace PayrollFIT
{
   public class StaffListFixture : ActionFixture
   {
      private string m_Name;
      private int       m_TaxCode;
      private Staff  m_Staff;

      public void Forename(string name)
      {
         m_Name = name;
      }
      public void Taxband(int taxcode)
      {
         m_TaxCode = taxcode;
      }
      public void OK()
      {
         m_Staff = new Staff(m_Name, new TaxDetails04(m_TaxCode));
         if (m_Staff != null)
          App.Instance.team.Add(m_Staff);
      }
   }
}

The second of Bob's tests needs a class called StaffDataFixture, which you create as before, but this time derive it from FIT's RowFixture class. This second test is a bit more complex to write because you need to implement the getTargetClass and query methods, defined as abstract in the base class. The first of these methods allows FIT to discover what class it should use to adapt the data as it is known to the system into the data as it is known to the test. The second method, query, is invoked by FIT to obtain an array of these adapter objects for use in the test. It is your job to create this array from the collection of Staff objects held by the program, but first you must create the adapter class which you call StaffAdapter:

public class StaffAdapter
{
   private Staff m_Staff;

   public StaffAdapter(Staff staff) {m_Staff = staff; }
   public string FirstName() { return m_Staff.Name; }
   public int StaffNo()    { return m_Staff.StaffID; }
   public string CurrentTaxcode(){return m_Staff.TaxInfo.ToString();}
}

Once you've implemented StaffAdapter, you can code the getTargetClass as well as the query (StaffDataFixture.cs). When you're finished testing these new fixtures on your development PC, PayrollFIT is deployed to the test environment, and Bob can now run his tests.

There's a lot more testing to do before Bob can consider the story complete, but at least he can see that writing tests is really helping him get the product his department needs.

Conclusion

Customers have much to gain by writing and running their own functional tests for the software they want created. It helps them communicate their specs more clearly to developers, which is particularly important for a project where requirements are changing or were not well understood at the outset. Testing also gives customers a sense of control. Instead of asking the developers about progress, the customer can simply run the tests and see exactly what functionality is present and what is not working as it should. Tools like Excel, WinFITRunnerLite, and FIT put this sort of power into a customer's hands.

Will Stott is a freelance consultant living in Zurich, Switzerland. He is also an associate of Exoftware, a European company specializing in helping organizations become more agile in their software development practices. Reach him at wstott@exoftware.com.


Editor's Note: C++, Plus...

Editor's Note
C++, Plus...

How long have you been reading MSDN Magazine? Did you start picking it up five years ago? Ten years ago when we were Microsoft Systems Journal? Even longer? If you've been reading our publication for a decade or more, you've seen a lot of changes in what's considered "cutting edge" development. Looking back at the pages of MSDN Magazine and MSJ, you can see the history of programming for Windows reflected over a 20-year span.

In the first segment of the magazine's life (1986-1990), our coverage was heavily skewed towards the dominant programming language of the time, C. For many of you long-time Windows aficionados out there, your first "Hello World" program was written in C with the Windows API, most likely a simple GetMessage loop straight out of Charles Petzold's Programming Windows book.

In the early 1990s (1991-1995), C++ really captured the imagination of the Windows programming community at large. C++ adoption was propelled further by the 1992 advent of MFC, a set of handy classes that helped objectify programming for Windows by encapsulating lots of functionality. When was the last time you saw a TranslateMessage call? It's rumored that they still have three in captivity at the Microsoft Museum in Redmond, but they're only on display for private events.

C++ was pretty well established by the late 1990s (1996-2000). By July 1996, Paul DiLascia's C/C++ Q&A column was renamed simply C++ Q&A. However, as C++ and MFC cemented its place in the forefront of Windows programming, Visual Basic was gaining huge popularity, especially after version 4.0 provided the ability to target Windows 95 with 32-bit programs. At this point, developers tended to split into opposing camps.

Visual Basic was considered tantamount to "cheating" by many hardcore programmers: it was easy to get code written and running, and you didn't have to take meticulous care with your malloc statements. Nevertheless, C++ and Visual Basic were both integral parts of the GUI-based world of programming.

The past five years (2000-2005) have ushered in the .NET Framework and a new language, C#. A lot of people jumped on the C# bandwagon—it was new, it was cool, it was sharp, it was efficient to use, and it was ideally suited to working with .NET. But where did that leave C++? C++ has been treated like a teenage crush for many people, who have moved on to the seemingly more sophisticated stylings of C#. But now that Microsoft has refreshed C++ for Visual Studio 2005, many people are finding that it's like attending your high school reunion and meeting that formerly dowdy classmate who's had the stunning makeover. And yes, he or she doesn't hold a grudge.

Since along with Visual Studio 2005 comes an improved flavor of C++, there's no better time than now to introduce what many readers have been asking for: expanded C++ coverage. We're doing this in two ways. First, we're pleased to announce a new column, Pure C++, by noted C++ expert Stanley B. Lippman. (If you were wondering, the B stands for C++.) Stan will discuss the C++ language itself, touching on topics like the CLI and generics and their integration with C++.

Aaaaah, you say, but you already have a C++ column! That's correct. Paul DiLascia, who has written more pieces for us than anyone else since 1986, will continue his popular work, with one twist. His column will now be called C++ At Work, representing a focus that will sometimes leave behind the Q&A format. While the column will still answer reader questions, Paul will go beyond the language specification, describing how to solve real-world C++ problems as only he can.

And, as if this weren't enough, we've got even more for you! This issue marks the start of another new column, Inside MSDN. Over the years, the MSDN online team has created all sorts of innovative infrastructure and site functionality. They're willing to share some of their accumulated knowledge with you, and what better place than in MSDN Magazine? This month, Tim Ewald discusses MSDN2—the next generation of MSDN. One of the key features of MSDN2 is a simplified URL scheme. Tim tells us how it was accomplished, and it's pretty darn cool.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Susan Chory, Joe Duffy, John Durant, Dan Fernandez, Mike Fitzmaurice, Kit George, Sean Gephardt, Chris Hays, Larry Jordan, Ronald Laeremans, Martyn Lovell, Jim Newkirk, John Nisi, Erik Olson, Matt Powell, Bill Ramos, Raphael Renous, Jay Roxe, Christina Storm, Roberto Taboada, and Rick Winter.

Active Directory, ActiveX, DirectX, FrontPage, IntelliSense, Microsoft, MSDN, MSN, Outlook, PowerPoint, SharePoint, Visio, Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Server is a trademark of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Marnie Hutcheson

Contents
Clean Up Your Registry
Synchronize Your Outlook Tasks
Convert Apps into Flexible Formats
Add 3-D to Your Presentations
Solving Simultaneous Equations
The Bookshelf

Clean Up Your Registry

Registry Medic 3.0, released by Iomatic, allows you to clean up your registry, improves PC performance, protects your machine from Trojans, and gets rid of those annoying start-up program artifacts that can be so hard to remove.

Registry Medic 3.0 uses an "intelligent" engine to scan your registry for invalid entries. By clicking a button you can repair or remove registry entries that refer to a file or folder that has been moved or deleted. You can back up and restore your registry on a one-time basis or on a regular schedule.

It's risky to interfere with the registry on this scale, so I backed up everything before I attempted to run the free trial version of this program (the trial will remove five defunct registry entries). It found 536 invalid registry entries—quite impressive. I am happy to report that I tried it on several PCs that were really running slowly, and most of them showed a definite performance improvement.

A couple of things did cause a bit of concern when I ran the program for the first time. First, Registry Medic 3.0 prompts you to back up your registry before it does any clean-up operations, but it doesn't give you a confirmation that it saved the registry, or tell you where it put it, or what it was called. So I was glad that I had used RegEdit to back up my registry before I started.

Also, several of the machines reported errors closing objects and DLL files when I rebooted the systems after having Registry Medic 3.0 remove the invalid registry entries and defunct software. These errors at shutdown were a one-time event and so far none of the PCs that were cleaned up have had any major problems. I have had only one app-lication stop running since the clean-up was performed.

Registry Medic 3.0 runs under Windows® 95, Windows 98, Windows Me, Windows NT® 4.0, Windows 2000, Windows XP, and Windows Server™ 2003.

Price: $29.95. A free trial version is available for download.

www.iomatic.com

Synchronize Your Outlook Tasks

SynchPst for Outlook 1.4.7 by Wisco synchronizes your Outlook® e-mail, tasks, journal entries, contacts, notes, distribution lists, and calendar appointments on two or more networks connected without a Microsoft Exchange Server.

You can synchronize Outlook on your laptop as easily as you sync your Pocket PC, by setting up the folders you want to keep updated. You can also share Outlook folders with others. So, for example, you can update your calendar on your laptop when traveling and transfer those changes automatically to the Outlook calendar on your desktop and your co-workers's computers on the network. This is very useful for those who don't have Microsoft Exchange Server in their network.

In addition to the basic version, the professional version lets you save and restore your settings, set the time intervals for synchronization, set filters on your e-mails, tasks, and appointment folders, and exclude private entries from synchronization.

SynchPst supports Outlook 2000, Windows XP, Windows Server 2003, Windows 95, Windows 98, Windows Me, and Windows NT 4.0.

Price:Basic is $39.95; Professional is $69.95. A free trial version is available for download.

www.synchpst.com

Convert Apps into Flexible Formats

Thinstall 2.75, from Jiti,converts existing applications into a format that can be run on any computer without file system or registry modifications, and without making any changes to your source code. It works with binary EXE, DLL and data files, integrates into your current process, and is compatible with almost all available libraries, tools, and installers.

Thinstall applications are deployed and run as a single EXE file, eliminating dependencies on outside files and registry keys. No more DLL Hell. You just put the EXE on a network drive and users can run it directly without installation. You simply delete the file to uninstall the application.

Thinstall 2.75 works in three stages. Link Stage binds all application files with the "Virtual Operating System" into a single EXE file. In the Load Stage the virtual operating system decompresses the application into memory and redirects all of its Win32® API calls to pass through the operating system. Finally, in the Run Stage, the program is executed normally with Thinstall performing virtualized file system and registry operations, thus enabling the application to run without installation.

Applications running under the virtual operating system access files from the real system and virtualized environment simultaneously without knowing the difference between the two. Applications think they have been installed, but no changes have been made to the computer.

Thinstall loads almost all 32-bit Windows-based applications and COM files without requiring any source changes. Applications based on the Microsoft® .NET Framework and Visual Basic® that are compressed with Thinstall can run without requiring that the Framework or Visual Basic are installed on the machine.

In addition Thinstall 2.75 adds anti-virus protection and obfuscation technology which generates unique secure EXEs for each customer. You can protect your documents and applications against viruses and eliminate "false positives" that occur due to mistakes from anti-virus software.

Price: $1,475 for a single license. Free 30-day trial available.

www.thinstall.com

Add 3-D to Your Presentations

Tired of the same old PowerPoint® presentation templates? OfficeFX 1.3 from Instant Effects will really wake them up for you with impressive 3-D backgrounds and foregrounds that move in a variety of interesting (even amazing) ways through your presentation, without disturbing your content. OfficeFX 1.3 features a number of advances in video textures and uses DirectX® 9.0 shaders and geometry deformation to increase its capabilities, so you can display images and videos across your slides.

This sophisticated capability requires a powerful PC with a very recent and capable 3-D graphics card.

You start by creating your PowerPoint presentation as usual, then click the OfficeFX button on the PowerPoint toolbar and a new window opens with the OfficeFX theme interface. After selecting a theme, you can alter it by modifying materials, shading, and lighting effects. Finally, you can choose from a variety of animations, interactions, and transitions that supplement your presentation without affecting your text formatting and placement. FXPre-view lets you view the presentation as you're making your changes to it.

Price: $149, for a single license, including 14 FXThemes

www.instanteffects.com

Solving Simultaneous Equations

TK Solver 5.0 Premium, by Universal Technical Systems Inc., is a rules-based programming environment that lets you set up and solve one or more equations simultaneously. This new release allows you to harness TK Solver 5.0 in your Microsoft Excel spreadsheets and use it to solve problems without writing any code or equations. You can also package the Excel solutions that you create and then distribute them freely to others.

Instead of a project, TK Solver 5.0 uses a model as the organizational container. A model can contain rules (equations), variables, functions, lists, units (for conversions), tables of values, comments, and plots. TK Solver 5.0 comes with many library and sample models to help you get started, or you can create your own.

You can export your models to Excel from TK Solver 5.0, or you can import from them directly into Excel using the TK Solver tool bar functions in Excel. The TK Solver 5.0 equations and model are plugged into the worksheet for you. The output values, plots and dynamic model elements all update automatically with their new values as soon as the user moves the cursor out of an input cell. Users can interact with your solution in Excel by changing inputs and viewing the updated outputs.

TK Solver 5.0 Premium runs on Windows 2000 and Windows XP. It comes with a full set of animated screenshot-based tutorials and tons of library and sample models.

TK Solver is powered by a DLL called rulemaster that integrates with Visual Basic, VBScript, and all of the Visual Studio® .NET-supported languages such as C#.

Price: $599.00 per desktop.

www.uts.com

The Bookshelf
Office 2003 XML

Office 2003 XML by Evan Lenz, Mary McRae, and Simon St. Laurent (O' Reilly, 2004) examines how the various products in the Office suite both produce and consume XML. The book covers the diverse set of XML tools in Office, exploring how XML is supported in Word, Excel, and Access. Developers learn how they can connect Office to other systems, and power users learn to create and analyze XML documents using familiar Office tools.

The book covers both the user interface side, creating interfaces so that users can comfortably (and even unknowingly) work with XML, and the back end, exposing Office information to other processes. It also looks at Microsoft InfoPath and how it fits with the rest of Office. Finally, the appendices introduce various XML technologies that may be useful in working with Office, including XSLT, W3C XML Schema, RELAX NG, and SOAP.

Price: $39.95, 576 pages.

www.oreilly.com/catalog/officexml

All prices were confirmed at press time and are subject to change.

Send your New Stuff to newstuff@microsoft.com.

Marnie Hutcheson is president of Internet Development Associates (Ideva), a firm in Ocala, Florida, that specializes in Internet and intranet Web application design and development. She has published technical papers and books on various computing topics. You can reach her at marnie@ideva.com.


Web Q&amp;A: ActiveX Privileges, Making Icon Files, Sticky Sessions, and More

Web Q&A
ActiveX Privileges, Making Icon Files, Sticky Sessions, and More
Edited by Nancy Michell

Code download available at: WebQA0502.exe (2,302KB)

Q I have an application that downloads ActiveX® controls in an HTML page. On certain machines the download fails, but if I give the user local admin rights, the download succeeds. What are the specific privileges that need to be granted to a user for them to be able to download ActiveX controls via HTML pages?

A Don't hack your user privileges to allow them to install ActiveX controls, as this would allow users to install any arbitrary code and trash the machine. You should deploy the ActiveX controls to the users in a Microsoft® Installer (MSI) package through Active Directory® so that you don't have to loosen user restrictions. You can also use Systems Management Server (SMS) or your tool of choice.

Q Do you know of any good tools for converting .png (or .psd, .jpg, .gif) into .ico files?

A There are a number of programs that do this. You can try Axialis IconWorkshop. You might also want to look at Irfanview. It's fast and does all the conversions you mentioned, along with other useful stuff. You can also use icoConverter developed by Alexandre Souza. It lets you select an image (.bmp, .png, or .jpg) that will be converted to .ico file. You can download it from the link at the top of this article.

Q I'm using load balancing across my Web services farm. Both my UI servers and servers hosting the Web services are load balanced. Also note that I have affinity set to "none". Occasionally when calling a Web service from ASP.NET, I get "Underlying Connection Was Closed" error messages. When load balancing is not involved in the test environment, I don't get the error. The error is described at FIX: Intermittent "Underlying Connection Was Closed" Error Messages When You Call a Web Service from ASP.NET. What's up?

A This type of error is common in load-balancing situations. To avoid it you need to use sticky sessions; in other words you need to set the Affinity to "single host" (note that Cisco and other load-balancing packages may refer to it by a different name).

If you are using secure sockets layer (SSL), it doesn't make sense to negotiate an SSL connection from the client to server A and then have the load balancer move your next round-trip to server B. Since no SSL connection was established with server B, you get "The underlying connection was closed: Could not establish secure channel for SSL/TLS" error.

Sticky sessions are a requirement if you are using any HTTP connection-level (as opposed to request-level) protocols like SSL or Windows NT® LAN Manager (NTLM). Like SSL, if you are using NTLM authentication, when the load balancer moves the client to server B for the second round-trip, the client will get some kind of HTTP 400-level Bad Request message (like 401 or 403) or some access denied error, because the proper NTLM negotiation didn't take place between the client and server B (the client negotiated with server A).

SSL and NTLM operate at the connection level, not at the request level. Load balancers operate at the request level unless configured for client affinity.

Depending on your particular circumstances, it is sometimes possible to use UnsafeAuthenticatedConnectionSharing to prevent this NTLM error. Here is further reading on the subject: ASP and Web Session Management, How To Handle Web Client Session State in an Application Center 2000 Cluster, Avoiding Traps When Using the Application and Session Objects, and How to use ASP.NET session state SQL Server Mode in a failover cluster.

Q How can I capture all Web requests that come into my site using a single .aspx page that will examine the path requested and then generate the response from a database? I added the following code to my Global.asax, but it's working on only some requests:

protected void Application_BeginRequest(Object sender, EventArgs e)
{
    HttpContext hContext = HttpContext.Current;
    string requestURL = hContext.Request.Path;
    hContext.RewritePath("/default.aspx?path=" +
        hContext.Server.UrlEncode(requestURL));
}

A Your module will only execute ASP.NET requests because .html and .asp extensions will be served by IIS without being passed to the ASP.NET runtime. If you want to capture all requests in that single .aspx page, you have to do the following:

  • Go to Properties for your virtual directory in the IIS administrative tool and click the Configuration option.
  • Map the .asp, .html, and other extensions you desire to the aspnet_isapi.dll (copy the path used for .aspx extension). That will tell IIS to pass requests for these extensions to ASP.NET. You should make sure to deselect "Check if file exists" if the .asp and .html files are virtual and do not actually exist on the server.

Q In my dynamic DataGrid, (see Figure 1) why am I able to see all of the columns from the DataSet, even though I have set the AutoGenerateColumns property to false and have one bound column added to the grid?

Figure 1 Dynamic DataGrid (Close Figure 1)
Dim dsServer As DataSet = New DataSet
sda.Fill(dsServer)

Dim dgServer As DataGrid = New DataGrid
dgServer.DataSource = dsServer
dgServer.DataBind()
dgServer.AutoGenerateColumns = False

Dim bcolumn As BoundColumn = New BoundColumn
bcolumn.DataField = "GMTOffset"
dgServer.Columns.Add(bcolumn)

Page.Controls.Add(dgServer)

A You have already bound the grid before you set any of the settings. Calling DataBind should be the last step before adding it to the page, as shown in Figure 2.

Figure 2 Binding the Grid (Close Figure 2)
Dim dsServer As DataSet = New DataSet
sda.Fill(dsServer)

Dim dgServer As DataGrid = New DataGrid
dgServer.DataSource = dsServer
dgServer.AutoGenerateColumns = False

Dim bcolumn As BoundColumn = New BoundColumn
bcolumn.DataField = "GMTOffset"
dgServer.Columns.Add(bcolumn)

dgServer.DataBind()
Page.Controls.Add(dgServer)

Q I have some questions about the Local Group Policy (Windows® XP SP2). What is the difference between this registry key

User Configuration\Windows Settings\Internet Explorer Maintenance\Security
and the following registry key:
User Configuration\Administrative Templates\Windows Components\
Internet Explorer\Internet Control Panel\Security Page\...
Do I need to set both, one, or the other?

Both of these keys deal with the security zones in Microsoft Internet Explorer. The second key looks like the GUI piece but also sets the keys, while the first key directly sets the key. Which one of these takes precedence?

A The Admin Templates settings are true policies (no registry tattooing) and provide for very granular control. This is new with Windows XP Service Pack 2 and is a great improvement in Internet Explorer manageability.

Internet Explorer Maintenance (IEM) provides functionality similar to the Internet Explorer Administration Kit (IEAK). Those settings tattoo the registry and do not provide for granular control. Internet Explorer Maintenance does not provide the ability to configure just one URL Action (like Active Scripting to Prompt for the Internet zone) without configuring all the other URL actions for all the zones.

Another important point is that IEM controls registry settings outside of policy and sets the corresponding registry key in preferences. Savvy users can then manipulate the URLAction settings in the registry after the IEM portion of the Group Policy Object (GPO) applies. Windows XP SP2 policy settings (those configured in Admin Templates) provide protection from non-administrative user manipulation because the policy subkeys have ACLs which prevent non-admins from changing these values.

The following is the order of precedence of policy (Admin Templates) and preference (IEM) processing of URLAction values:

  • HKEY_Local_Machine Policy (Admin Templates)
  • HKEY_Current_User Policy (Admin Templates)
  • HKEY_Current_User Preference (IEM)
  • HKEY_Local_Machine Preference (IEM)

The first match is the enforced setting within Internet Explorer, meaning that if you set it in the policy section and the IEM section, policy wins. This includes modifications made in the UI (which simply control URLAction settings in the preferences subkeys).

Q A free newspaper Web site I use confirms password updates with an e-mail in which the password is revealed in plain text. This does not seem secure to me. Is it?

A The right question to ask is, "is this secure enough?" This kind of question is a good opportunity to really get a handle on how far you need to go to secure different kinds of data. Remember, security isn't a binary, all-or-nothing bit that you set on an application. Security comes from a process whereby you identify threats, then identify the value of the data being threatened, and design a strategy to mitigate those threats in the context of the value being protected. How worried is anyone that someone else might read that free newspaper using your ID?

Do you keep silverware in your house? You must eat with something. Is your house secure? Someone certainly could get into your house and steal your silverware if he really wanted to. The question then is whether your silverware is so valuable and the threat of me stealing it is so great that you're willing to shutter all your windows, bar your doors, buy a safe for your silverware, and hire guards to keep me out.

So, what's the bottom line? It makes little sense to implement a security system that costs more than replacing the data being threatened. If you paid $30 for your silverware at Target, you wouldn't exactly be worried about it. Having a password costs something—the password database has to be maintained, the code has to be debugged, and so on. But passwords mitigate a whole lot of threats. Not every threat, by any means, but lots of them.

You've identified a threat not mitigated by the existing strategy—someone could be packet-sniffing your e-mail. There could be a rogue admin at your ISP. Sending the password in the clear doesn't mitigate against these threats.

However, you need to ask yourself the following questions: what is the value of the data being protected—the ability to read your newspaper? What is the likelihood of attack? Are there really rogue admins reading your e-mail to steal your newspaper password? What is the cost of mitigating the attack? Are there any other factors? For instance, if recovering from an attack is cheap and easy, then preventing the attack is less important. Of course, you make things worse for yourself if the password you use to access this newspaper is the same password you use to check your e-mail or to access your financial records.

In this case, the value of the data being protected is low, the likelihood of attack is low, the cost of recovering from a successful attack is low, and therefore the cost of the security system should also be low.

Q I have a client certificate authorization requirement on a virtual directory of my IIS 6.0 Web site. After IIS authenticates the certificate, I verify that the usage policy of the client certificate is enabled for "smartcard logon." If the client certificate does not have this usage policy, then the request will be redirected.

But after Internet Explorer successfully authenticates a client certificate against a specific virtual directory, it will use the same client certificate in the future, and never pop up the certificate picker dialog. It seems there is nothing that I can do to bring the certificate selection box back up again the next time the user hits this virtual directory, but is there?

A Your only possible solution in Internet Explorer is to clear the SSL state by choosing Tools | Internet Options | Content | Clear SSL State from the tools menu.

The other possibility lies not in Internet Explorer, but in CryptoAPI. In Windows XP Service Pack 2 there is a design change in the way that the Crypto API caches the certificates on the client for a given process (see Windows Prompts You for Your Password Multiple Times When You Use Outlook If Strong Private Key Protection Is Set to High).

This is the registry key that controls the new behavior:

HKLM\Software\Policies\Microsoft\Cryptography
To change it, add a new DWORD named "PrivKeyCacheMaxItems." Setting the value to " = 0" will prevent any keys from being cached. By default 20 keys will be cached per process.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Betsy Aoki, Gaurav Bansal, Rob Campbell, Richard Ciapala, Dan Crevier, Rob Franco, Jay Herbison, Simon Hoare, Kenneth Lassesen, Drew Leaumont, Eric Lippert, Eilon Lipton, Lawrence Liu, David Lovell, Duncan Mackenzie, Rob Maushardt, Alan Melia, Jerry Orman, Nikola Penkov, Minh Pham, Jiri Richter, Mike Shepperd, Nuno Silva, Eugene Siu, Morgan Skinner, Andy Sterland, Sean Sutherland, Stephen Toub, Joel Yoker, and Fan Zhang.


Inside MSDN: Designing URLs for MSDN2

Inside MSDN
Designing URLs for MSDN2
Tim Ewald

Contents
The Brittle URL Problem
Identifying Content
Content Aliases
Revisions and Versions
Locales
Alternates
Using Content in More Than One Context
So, How Does It Work?

This is the first installment of a new column about MSDN® projects: what we're doing, how we're doing it, and what we're learning along the way. It will be written by MSDN staff with the goal of sharing the team's experiences in solving the real-world business problems MSDN faces. This first installment introduces the URL model we implemented in the alpha release of the MSDN/TechNet Publishing System (MTPS), the new online infrastructure we launched in September 2004 to support MSDN2 (msdn2.microsoft.com). And as you're reading, keep in mind that because the technology discussed here is the alpha release of MSDN2 and the MTPS system behind it; it describes the functionality and implementation as of this writing. Things may change before the final version ships. If so, this column may revisit this topic in the future. Now, onto URLs!

It's easy to take URLs for granted. When you create ASP.NET pages, they get URLs intrinsically, based on their location on disk. For a small site, especially on an intranet, that's a very reasonable approach. But for a large site or one that's accessible on the Internet, how your URLs work is a key design aspect of your system. How will you support versions and translations of content? Can the same content appear in more than one place in your site? Or can the same content appear in multiple sites? What will happen to your URLs if you restructure the way you store the data your site exposes? What will happen if you change your implementation technology? Most importantly, how can you keep your customer's links to your content stable over time?

The Brittle URL Problem

One of the leading causes of dissatisfaction among the users of the MSDN Library online is that every so often, a lot of URLs break. This is such a problem that the 404 handling page is one of the most common entry points for visitors arriving at MSDN. On the surface, this seems like a simple problem to solve. After all, how hard can it be to keep links to a collection of documents—even a really big collection— stable over time? The answer, it turns out, is that it can be very hard, as you'll come to understand once you learn how the existing MSDN Library is built.

Here's how it goes. MSDN is responsible for publishing technical documentation for a wide range of products and technologies. The user education (UE) teams working on those products produce documentation in a format intended for installation directly on a user's machine. These documentation sets (or docsets) are a combination of HTML and XML packaged together in a single binary (either a .chm or .hxs file). Those files are consumed by the HTML Help Viewer, which renders topics locally.

The current MSDN Library is built by deconstructing those packaged help files, spreading the content out on disk, massaging it a bit, and building a site on top. The location of the topics on disk is derived from their location within the docset file that they came from. That location on disk is also reflected in the MSDN Library URL. Sometimes, a UE team decides to restructure the layout of the files in a docset. When that docset is given to MSDN and the Library is rebuilt, the URLs for topics change. Part of the Library build process updates internal links for consistency. External links from other sites or products to MSDN, however, can break.

Brittle URLs is one of the problems we set out to solve 18 months ago when we started to build a new online infrastructure that's now known as the MSDN/TechNet Publishing System. There were other goals, too, like publishing XML instead of HTML and making sure we could expose content in more than one context, for example in both the Library and a Developer Center. We developed a model that worked for MSDN and also met the needs of our content contributors. The first site to use the new system is the MSDN2 Library, which contains prerelease documentation for Visual Studio® 2005.

Identifying Content

The crux of the brittle URL problem is that topics in most UE docsets do not have unique identifiers that remain stable over time. Topics are only identified by path within a docset and, as I mentioned, those paths change from release to release. Solving this problem would require every topic to be assigned a unique ID at the point of origin. We chose to use a GUID which, once assigned to a topic, remains stable. If a topic's GUID ever changes, then it's really a new topic. The old topic remains, with the old GUID. (Some topics already have locally unique identifiers that remain stable over a period of time. In that case, the local ID is mapped to a GUID when the content is published.)

GUIDs are a natural choice for identifying content but they aren't a great thing to include in URLs if you ever expect people to transcribe them. MSDN URLs often appear in magazines and books. Readers often type them in. If our URLs contained GUIDs, the chance of a typo would be high and the chance of finding the error, low. To that end, we opted to assign each GUID a corresponding short ID. The short ID is an eight-character alphanumeric string. It's meaningless, but it's easier to type. Here is an example of a URL that uses a short ID: http://msdn2.microsoft.com/library/b8a5e1s5.aspx.

Short IDs are the preferred format for our new URLs, but there are others. It is legal to use the GUID associated with a topic as well. The two values are treated as synonyms. The mapping between the values is unique and bidirectional and it never changes. Also, the .aspx extension is optional. In other words, this URL is equivalent to the previous one: http://msdn2.microsoft.com/library/b8a5e1s5.

Some people wonder why we bother with the .aspx extension, as it ties us to a particular technology. Others wonder why we aren't using ASP.NET when we tell them they should (we are using ASP.NET, but without the .aspx it might appear as though we're not). To make both groups happy, we chose to use the .aspx extension, but you can leave it off if you feel like it. If you do leave it off, the system will add it for you via redirection.

Content Aliases

One limitation of short IDs is that while they are easy to transcribe, they're impossible to infer. This is okay for a lot of MSDN content because there's nothing you could base your inference on anyway. There are, however, some topics where this is possible, like APIs or error codes. To that end, some topics are also identified by an alias. An alias is a unique text string that is synonymous to a topic's GUID or short ID.

The alpha release of MSDN2 contains the Visual Studio 2005 documentation. These documents are split into two separate categories: conceptual and managed reference. The managed reference topics have aliases derived from their namespace, type, and member names. For instance, you can access a list of all the types in the System.Xml namespace, an overview for the System.Xml.XmlReader type, and a list of the XmlReader type's members using the following three URLs: http://msdn2.microsoft.com/library/system.xml, http://msdn2.microsoft.com/library/system.xml.xmlreader, http://msdn2.microsoft.com/library/system.xml.xmlreader_ members.

Ultimately, you'll be able to access individual properties, fields, methods, events, and so forth, but the full implementation has not yet been completed.

This is my favorite MSDN2 feature, and based on feedback in the blogsphere, a lot of other people like it too. It provides the same quick access to API information that you get from the index in the offline help (which is not to say that I wouldn't like to see a real index online too). However, I'm compelled to mention that at the time of this writing, MSDN2 is based on an alpha version of MTPS, and some of the details on content aliasing may change. Since one of the big goals for this project is non-brittle URLs, you should link using short IDs until all the remaining aliasing details are sorted out.

Revisions and Versions

Once topics have identifiers that stay fixed over time, MSDN can provide URLs that don't break. But what happens when those topics are updated? Do they replace the previous copy or do both copies exist side by side. For MTPS-based sites like MSDN2, the answer is that it depends. The system differentiates between revisions and versions. A revision is a minor change to an existing topic, for instance, fixing a typo in a code sample. When a topic is revised, the new revision replaces the existing revision. A version is a major change to an existing topic, for instance, updating it to cover the .NET Framework 2.0 instead of 1.1. When a topic is versioned, the new version is published next to the current version and both remain available.

This leads to an interesting question: if two versions of a topic are published side by side, which one does a URL retrieve when the URL has no version specifier? The answer is the latest version that's available for the given product or technology. If you want a specific version, you can ask for it explicitly by including a version identifier in the URL. Here's an example: http://msdn2.microsoft.com/library/b8a5e1s5 versus >http://msdn2.microsoft.com/library/b8a5e1s5(vs.8).

The first URL is version independent. It will always provide access to the latest version of the topic (the overview for XmlReader). The second URL is version specific. This second URL refers to the Visual Studio 2005 version of the topic. When the next version of Visual Studio (the version after Visual Studio 2005, code-named "Orcas") ships, the first URL will point to the Orcas version of the topic but the second URL would keep pointing to the Visual Studio 2005 version of the topic.

While this isn't currently reflected in the MSDN2 alpha release, ultimately all links across topics in a given docset will include a version identifier so that once you've started browsing between topics in, say, the Visual Studio 2005 docs, you won't silently be moved to topics in the Orcas docs.

By supporting both version-independent and version-dependent URL formats, other sites and tools linking to MSDN2 topics have a lot of control over what version of a topic they're referring to.

Locales

Just as there can be multiple versions of a topic, there may be multiple locales as well. As with versions, multiple locales are published side by side. You can request a particular locale by adding an appropriate identifier to a URL. The following is an example: http://msdn2.microsoft.com/library/b8a5e1s5(fr-FR) and http://msdn2.microsoft.com/library/b8a5e1s5(vs.8, fr-FR). The first link explicitly requests the French/France locale. The second link requests the Visual Studio 2005 content for that same locale.

So what happens if you don't request a particular locale? It depends on the site you go to. The initial release of MSDN2 defaults to U.S. English. Other sites will default to other locales. You can ask explicitly for any locale within any site and it should be displayed, but the navigational elements for the site won't change.

Alternates

In the previous two sections, I described how the MTPS/MSDN2 URL scheme supports multiple side-by-side copies of the same topic, each representing a different version and locale. You can think of a single topic as a two-dimensional sparse array of content, with one dimension representing version and the other locale, as shown in Figure 1.

Figure 1 Content in Two Dimensions
Figure 1 Content in Two Dimensions

You can index into the array by including version and/or locale identifiers in parenthesis at the end of the URL. The order of these parameters doesn't matter and they can come before or after the .aspx extension if it's present.

You may be wondering, though, what values you're allowed to use. The answer is that it depends, and new values can be added over time. As Figure 1 shows, there is no requirement that every version/locale combination exists. Further, when a new version is added, it's unlikely that all translations to all locales will be done at the same time.

The easiest way to find out what version/locale combinations are legal for a given topic at a particular time is to request a list, called alternates. In theory, you can end a URL with an empty set of parentheses, like this: http://msdn2.microsoft.com/library/b8a5e1s5().

Unfortunately, this isn't implemented at the time of this writing. Instead, you have to pass an invalid parameter; any single letter (which is never a valid version or locale identifier) will do. Here's an example: http://msdn2.microsoft.com/library/b8a5e1s5(x).

If you resolve this URL, you'll get a page containing the following information (with essentially no formatting, but hey, it's only an alpha release):

b8a5e1s5    en    VS.8    b8a5e1s5.aspx(VS.8,en)
The table lists all the possible legal combinations of versions and locales for a given topic, along with a hyperlink to each one.

Using Content in More Than One Context

Everything I've talked about so far has focused on the last token in a URL because that's what identifies a piece of content. The rest of the URL path identifies the context within which the content is being viewed. The two are entirely separate and any content can appear anywhere within a context, even in multiple places.

The reason the team designed the MTPS URL scheme this way is to be able to organize and present information in multiple ways, letting users choose the approach that works best for them. It's done that way to some extent today with the current MSDN Library and the various Developer Centers, but the implementation is not as flexible as the team wants it to be. In MTPS, there is no intrinsic link between content and context, making it easy to do a wide array of interesting things. MSDN2 is just the start.

So, How Does It Work?

The alpha version of MTPS and MSDN2 are built on ASP.NET 2.0 and SQL Server™ 2005. The system differentiates between physical content—pages on disk—and virtual data stored in our new XML-based publishing system. Virtual content uses the URL model described earlier. Physical content may or may not. Over time, more and more content will be made virtual, allowing us to use it in multiple contexts and to update it without touching the image of a site or the virtual root on each Web server in our cluster.

Figure 2 shows the complete architecture for the HTTP module, the virtual path provider and the content cache that MTPS uses to expose virtual content within a site like MSDN2. The pieces shown in blue are part of MTPS. Everything else in the figure is part of ASP.NET.

Figure 2 HTTP Module Architecture
Figure 2 HTTP Module Architecture

When a request for a page arrives, the HTTP module intercepts it and processes the URL, using logic like that encapsulated in the OnPreResolveRequestCache event handler shown in Figure 3.

Figure 3 OnPreResolveRequestCache (Close Figure 3)
public void OnPreResolveRequestCache(
    object o, EventArgs ea)
{
  try
  {
    // check to see if file is virtual (not physical)
    // and if it is a page
    string ext = this.GetExtension(httpApp.Request.Path);
    if (false == File.Exists(httpApp.Request.MapPath(
         httpApp.Request.Path)) && (ext == @".aspx" || ext == @".gif"))
    { // it's virtual

      // parse off query string
      string[] path = httpApp.Request.Path.ToLower().Split('?');
      string urlPath = path[0];

      // figure out what piece of content set this
      // request targets, the main document or an image
      this.requestInfo = this.GetDocumentInfo(urlPath);
      httpApp.Context.Items[requestInfoContextKey] = this.requestInfo;

      // parse out version and locale, applying defaults as needed
      string locale, version;
      this.GetVersionAndLocale(HttpContext.Current.Request,
        out locale, out version);
      string identifier = this.GetIdentifier(
        this.requestInfo.Type, urlPath);

      // encapsulate key information from URL
      ReadOnlyContentKey requestKey =
        new ReadOnlyContentKey(identifier, locale, version);

      // resolve requested key against content cache
      // and store result in context
      this.contentSet = DbContentLoader.CreateContentSet(
        true, this.connectionString, requestKey);

      httpApp.Context.Items[contentSetContextKey] = this.contentSet;

      // if key didn't identify anything, 404, otherwise
      // remap request to normalized virtual path
      if (this.contentSet == null)
      {
        httpApp.Context.AddError(new HttpException(404,
          "File not Found", new ApplicationException(
          "invalid content identifier")));
      }
      else
      {
        string newPath = this.GetVirtualPath(
          this.contentSet.Key, this.requestInfo);
        httpApp.Context.RewritePath(newPath);
      }
    }
  }
  catch (Exception e)
  {
    // log error with EIF and propagate,
    EifEvent.Error(Source.MpsDelivery, e,
      @"ContentModule::OnPreResolveRequestCache");
    throw;
  }
}

The first thing OnPreResolveRequestCache does is check to see if the requested URL targets a page or an image that does not exist on disk. If it does not, then the event handler does nothing and the request is processed using normal ASP.NET techniques. If the handler determines that the URL targets a virtual page or image, it breaks the URL apart and pulls out several pieces of information. First, it determines what the URL targets, the text content associated with a topic or an image associated with a topic. It encapsulates that target information in a DocumentInfo object, which it stores in HTTP context for later use.

Then it looks for version and locale information, applying defaults as necessary, and pulls out the content identifier—a GUID, short ID, or alias. Finally, it combines the version, locale, and content identifier into a ReadOnlyContentKey.

After the URL parsing is complete, the OnPreResolveRequestCache event handler uses the DbContentLoader class, which encapsulates access to the content cache, to create a ContentSet object from the key it built from the URL. It stores the value in HTTP context for later use. If the value is null, there is no topic in the content cache with the requested content identifier. In this case, the handler raises an exception signaling a 404. If the value is non-null, there is a topic in the content cache with the requested content identifier. In this case, the URL path is rewritten to point to a normalized path and processing continues. This level of indirection is what allows every content ID to be used with any URL path, making it appear that all topics exist everywhere.

The normalized path points to a file that does not exist on disk. Rather, the page data is stored in the content cache. The system uses a VirtualPathProvider (VPP) to bridge the gap between the two. In essence, a VPP intercepts all of the ASP.NET plumbing's requests for file streams and gives you a chance to load them from wherever you like. Every ASP.NET app uses a default VPP that simply maps to the file system. An MTPS-based site registers a custom VirtualPathProvider, which sits in front of the default VPP, forming a chain. The custom VPP uses the DocumentInfo and ContentSet objects that the HTTP module's OnPreResolveRequestCache event handler stored in HTTP context to load a topic from the content cache and return it as an .aspx file stream.

It's important to note that the MTPS VPP gets called for every file request, even for pages that exist physically on disk. In that case, it simply delegates the request down the chain to the default path provider that's built into ASP.NET. Whichever path the request takes—whether the custom VPP loads content from the cache or delegates to the built-in VPP that reads files from the file system—the rest of the ASP.NET plumbing doesn't care; it treats all returned file streams the same way. It's completely unaware that the custom VPP materializes some pages on demand.

This implementation is likely to change in future versions of the MTPS plumbing. The VPP code is pretty complicated. The MSDN team went with that model because it made it possible to transform content being retrieved from the cache before the ASP.NET page compiler processed it. That let us convert XHTML markup elements in the cached topic to server-side control tags, in essence injecting site-specific behavior into a page before any other processing occurs. Another approach is to map all virtual paths to a single page that knows how to load content and convert markup elements to controls during the page rendering process. We had a version of that working, but moved away from it to explore the VPP option. It may prove simpler to go back.

This initial Inside MSDN column describes the alpha release of MSDN2 and the MTPS system behind it. I've described the existing functionality and implementation as of the time of this writing. Hopefully, the ideas and design I've discussed here will help you when you are developing similar functionality in your own Web sites and content libraries.

Send your questions and comments to insdmsdn@microsoft.com.

Tim Ewald is a developer at Mindreef, where he writes tools to help developers build Web services systems successfully. He was recently a Program Manager Lead at MSDN, where he designed MTPS, the XML-based publishing system at the center of the next generation of MSDN online infrastructure.


Basic Instincts: Using the ReaderWriterLock Class

Basic Instincts
Using the ReaderWriterLock Class
Ted Pattison

Contents
The Limitations of Monitors
Synchronization Using ReaderWriterLock
A Tale of Two Queues
Upgrading a Shared Lock to an Exclusive Lock

In several installments over the past year I have written about multithreading programming techniques in Visual Basic® .NET. In the September 2004 issue of MSDN®Magazine, I discussed the need for thread synchronization and how to write thread-safe code using monitors. While monitors are easy to use, there can be undesirable trade-offs. My discussion this month looks at an alternative technique for writing thread-safe code using the ReaderWriterLock class. As you will see, this approach can result in faster response times and increased levels of throughput.

The Limitations of Monitors

A monitor is a synchronization mechanism that is accessible through the Monitor class in the System.Threading namespace. Let me quickly review monitors and explain how they work. Examine the code in Figure 1, which demonstrates a Point class with two Integer fields. Read and write access to these two fields for any given Point object has been synchronized using calls to Monitor.Enter and Monitor.Exit. Given this code, the Microsoft® .NET Framework provides thread synchronization by blocking threads from entering the monitor once another thread has entered it. The .NET Framework also provides the means to unblock the next thread in line as soon as the thread inside a monitor calls Monitor.Exit and releases its lock.

Figure 1 Thread-Safe Class (Close Figure 1)
Imports System.Threading

'*** a thread-safe class
Class Point

  '*** private fields
  Private x As Integer
  Private y As Integer

  '*** constructor does not require synchronization
  Sub New(ByVal x As Integer, ByVal y As Integer)
    Me.x = x
    Me.y = y
  End Sub

  '*** read point position with synchronization
  Sub GetPointPosition(ByRef x As Integer, ByRef y As Integer)
    Monitor.Enter(Me)
    Try
      x = Me.x
      y = Me.y
    Finally
      Monitor.Exit(Me)
    End Try
  End Sub

  '*** update point position with synchronization
  Sub SetPointPosition(ByVal x As Integer, ByVal y As Integer)
    Monitor.Enter(Me)
    Try
      Me.x = x
      Me.y = y
    Finally
      Monitor.Exit(Me)
    End Try
  End Sub

End Class

The purpose of using a monitor in this design is to ensure that a writer thread is able to change both the X position and the Y position of a Point object as an atomic unit of work before any other thread is able to read its underlying data. The resulting synchronization prevents inconsistent reads where another reader thread would be able to see the logical Point at an invalid position.

Figure 2 shows a simplified version of the Point class rewritten using the SyncLock construct. When you use the SyncLock construct in this manner, the Visual Basic .NET compiler expands your code to call Monitor.Enter. The code expansion also supplies a Try statement with a Finally block containing a call to Monitor.Exit. The result is that after compilation the code shown in Figure 2 is almost identical to the code shown in Figure 1. You should see that the Visual Basic .NET programming language provides the SyncLock construct to make this common pattern for writing synchronization code a bit easier to type.

Figure 2 Thread-Safe Class Using the SyncLock Construct (Close Figure 2)
Imports System.Threading

'*** class rewritten to be thread safe
Class Point

  '*** private fields
  Private x As Integer
  Private y As Integer

  '*** constructor does not require synchronization
  Sub New(ByVal x As Integer, ByVal y As Integer)
    Me.x = x
    Me.y = y
  End Sub

  '*** read point position with synchronization
  Sub GetPointPosition(ByRef x As Integer, ByRef y As Integer)
    SyncLock Me
      x = Me.x
      y = Me.y
    End SyncLock
  End Sub

  '*** update point position with synchronization
  Sub SetPointPosition(ByVal x As Integer, ByVal y As Integer)
    SyncLock Me
      Me.x = x
      Me.y = y
    End SyncLock
  End Sub

End Class

So given the effectiveness and simplicity of using monitors and the SyncLock keyword, what are their limitations? The first is that the convenience of the SyncLock keyword is offset by a loss of control. One example of this limitation is that the SyncLock construct does not permit you to pass a timeout value. That means you should avoid using the SyncLock construct in more complicated synchronization schemes where your code could be vulnerable to deadlock scenarios.

If you are required to protect your code against deadlocks when using a monitor, you should avoid the SyncLock construct and replace calls to Monitor.Enter with calls to Monitor.TryEnter. The TryEnter method allows you to specify a timeout interval in milliseconds that can effectively prevent the calling thread from blocking indefinitely. The TryEnter method returns a value of true if the calling thread is able to enter the monitor before the timeout interval is exceeded. The TryEnter method returns a value of false if the timeout interval is exceeded and the calling thread was not able to enter in order to monitor. This should lead you to structuring your code like the code in Figure 3.

Figure 3 Protecting Against Deadlocks (Close Figure 3)
Dim LockAcquired As Boolean
LockAcquired = Monitor.TryEnter(Me, 2000)
If LockAcquired Then
  Try
    '*** perform work while inside the monitor
    Me.x = x
    Me.y = y
  Finally
    '*** release lock
    Monitor.Exit(Me)
  End Try
Else
  '*** supply contingency code for when timeout occurs
End If

A second noteworthy problem with using monitors is that they only provide synchronization based on exclusive locking. In other words, a monitor cannot permit access to more than one thread at a time. This is true even when a set of threads are only attempting to perform read operations. This limitation can lead to inefficiencies in a scenario in which there are many read operations for each write operation.

Consider a robust and scalable DBMS such as SQL Server™. Think about how much the performance of high-volume applications would degrade if SQL Server supported only exclusive locking but not shared locking. It would have a severely negative impact on concurrency because only one request would be able to read a specific record or a specific table at a time, resulting in slower response times and a decrease in throughput because reader threads would be blocking other reader threads unnecessarily.

The key point is that exclusive locking is inefficient in cases where multiple threads are performing only read operations. Inconsistent reads can't happen until one or more threads update data. Fortunately, the internal plumbing for a DBMS such as SQL Server supplies a more sophisticated concurrency mechanism which mixes shared locking with exclusive locking to speed up response times and increase overall system throughput. SQL Server uses shared locking whenever it can and then resorts to exclusive locking only when it's required.

Synchronization Using ReaderWriterLock

The exclusive locking scheme used by monitors doesn't consider whether threads are readers or writers. Unfortunately, this isn't always the best choice, especially in scenarios in which there are many more read operations than write operations. Sticking with monitors in such a scenario can unnecessarily degrade your response times and overall application throughput. Therefore, you should consider using the ReaderWriterLock class in order to assist with your synchronization.

The ReaderWriterLock class allows you to design a synchronization scheme which employs shared locks together with exclusive locks. This makes it possible to provide access to multiple reader threads at the same time, which effectively reduces the level of blocking. The ReaderWriterLock class also provides exclusive locking for write operations so you can eliminate inconsistent reads.

Figure 4 shows a rewritten version of the Point class, which uses a ReaderWriterLock object instead of a monitor to achieve thread safety. The obvious improvement with this new design is that a reader thread will not block other reader threads when calling the GetPointPosition method.

Figure 4 A More Efficient Thread-Safe Class (Close Figure 4)
Class Point

  '*** private fields
  Private x As Integer
  Private y As Integer
  Private lock As New ReaderWriterLock

  '*** constructor does not require synchronization
  Sub New(ByVal x As Integer, ByVal y As Integer)
    Me.x = x
    Me.y = y
  End Sub

  '*** read point position with shared lock
  Sub GetPointPosition(ByRef x As Integer, ByRef y As Integer)
    lock.AcquireReaderLock(Timeout.Infinite)
    Try
      x = Me.x
      y = Me.y
    Finally
      lock.ReleaseReaderLock()
    End Try
  End Sub

  '*** update point position with exclusive lock
  Sub SetPointPosition(ByVal x As Integer, ByVal y As Integer)
    lock.AcquireWriterLock(Timeout.Infinite)
    Try
      Me.x = x
      Me.y = y
    Finally
      lock.ReleaseWriterLock()
    End Try
  End Sub

End Class

In this design a ReaderWriterLock object is created each time a new Point object is created. This internal ReaderWriterLock object is referenced using a private field named lock. The lock field makes it possible for all instance methods associated with a Point object to access the same ReaderWriterLock object.

The GetPointPosition method calls AcquireReaderLock on the ReaderWriterLock object to obtain a shared lock before doing any of its work. This means that two or more threads can call the GetPointPosition method at the same time without blocking one another. The SetPointPosition method, on the other hand, calls AcquireWriterLock to obtain an exclusive lock. Once a thread has called AcquireWriterLock and acquired an exclusive lock, all other reader threads and writer threads will be blocked until this thread can complete its work and call ReleaseWriterLock.

When you call either AcquireReaderLock or AcquireWriterLock, you can optionally pass a timeout value to protect your code against deadlock situations. A timeout value indicates the amount of time a thread is willing to wait in a blocked state before it can obtain the lock it needs to do its work. The version of the Point class shown in Figure 4 passed Timeout.Infinite, a constant with a value of -1, indicating that the thread should wait for an infinite amount of time. However, you can change the blocking behavior by passing a timeout value in milliseconds, as shown in the following code:

Sub SetPointPosition(ByVal x As Integer, ByVal y As Integer)
  lock.AcquireWriterLock(2000)
  Try
    Me.x = x
    Me.y = y
  Finally
    lock.ReleaseWriterLock()
  End Try
End Sub

With this new version of the SetPointPosition method, the calling thread will only wait for two seconds when attempting to obtain an exclusive lock. If the calling thread's wait time exceeds the timeout interval, the AcquireWriterLock method throws an ApplicationException with the error message "This operation returned because the timeout period expired." It is then your responsibility to catch and handle this exception as gracefully as possible.

Now think about what would happen if a single thread calls AcquireWriterLock more than once before calling ReleaseWriterLock? Your intuition might tell you that this thread will acquire an exclusive lock with the first call to AcquireWriterLock and then block on the second call. Fortunately, this is not the case.

The ReaderWriterLock class is smart enough to associate exclusive locks with threads and track an internal lock count. Therefore, multiple calls to AcquireWriterLock will not result in a deadlock. However, this issue still requires your attention because you must ensure that two calls to the AcquireWriterLock method from a single thread are offset by two calls to the ReleaseWriterLock method. If you call AcquireWriterLock twice and only call ReleaseWriterLock once, you haven't released the lock yet.

The behavior that results from performing multiple calls to AcquireReaderLock is similar to that of making multiple calls to AcquireWriterLock. A ReaderWriterLock object simply increments the count of shared locks for any given thread. Once again, you should ensure that two calls to the AcquireReaderLock method are offset by two calls to the ReleaseReaderLock method.

A Tale of Two Queues

There is a classic synchronization problem that exists anytime shared locks are used together with exclusive locks. Consider a situation in which a writer thread is blocked waiting for a reader thread to release its shared lock. What if a second reader thread acquires a shared lock just before the first reader thread released its shared lock. In effect, the second reader thread showed up after the writer thread but was able to jump ahead of it in line.

Next, imagine that a third reader thread acquired a shared lock just before the second reader thread released its shared lock. The writer thread could theoretically block forever as reader threads continue to jump ahead of it in line. This scenario is known as a live lock situation.

Fortunately, the ReaderWriterLock class has been designed to deal with live locks. Each ReaderWriterLock object contains one request queue for reader threads and a separate request queue for writer threads. When a writer thread calls AcquireWriterLock to obtain an exclusive lock, its request is queued up in the writer request queue. When there are one or more writer requests pending and a reader thread calls AcquireReaderLock to obtain a shared lock, it is not instantly granted a shared lock. Instead, the reader thread is blocked and its request to acquire a shared lock is appended to the reader request queue. This effectively forces reader threads to wait until the writer thread has completed its work.

When the writer thread calls ReleaseWriterLock, all the reader threads that have requests pending in the reader request queue are granted shared locks and can complete their work. When all those reader threads release their shared locks, the next writer thread in the queue is granted an exclusive lock so it can complete its work. You should observe that a ReaderWriterLock object plays the role of a traffic cop, alternating back and forth between one writer and a collection of readers.

Upgrading a Shared Lock to an Exclusive Lock

Note that it is not possible for any single thread to hold both a shared lock and an exclusive lock at the same time. This is largely due to the way that the ReaderWriterLock class implements its synchronization scheme using a reader request queue and a writer request queue. Things would get confusing if a single thread had requests in both queues at the same time. However, it is not uncommon that a thread will need to obtain a shared lock at first and then later escalate to an exclusive lock to perform write operations.

Let's look at a common real-world scenario in which a thread must read data to verify the number of units in inventory and then write data to record a purchase. For example, a thread might first acquire a shared lock so that it can read inventory to verify that there are enough units in stock to make the purchase. If the thread determines that the inventory is sufficient to make a purchase, it escalates its shared lock to an exclusive lock so it can decrement the number of units to reflect the purchase.

The key point about using the ReaderWriterLock class is that a thread should never call AcquireReaderLock and then follow that with a call to AcquireWriterLock. If you do this, your call to AcquireWriterLock will block indefinitely. Instead, after calling AcquireReaderLock you should call UpgradeToWriterLock to escalate from a shared lock to an exclusive lock, as shown here:

'*** acquire shared lock
lock.AcquireReaderLock(Timeout.Infinite)

'*** escalate shared lock to exclusive lock
lock.UpgradeToWriterLock(Timeout.Infinite)

While this example demonstrates the way the .NET Framework documentation recommends you write your code, the locking behavior that you will experience is not what you might expect. I was certainly surprised the first time I wrote and tested this type of code. An example will help to illustrate the problems that can occur when escalating shared locks to exclusive locks with the UpgradeToWriterLock method.

Imagine that thread A calls AcquireReaderLock just before thread B calls AcquireWriterLock. In this scenario, thread B cannot obtain an exclusive lock and do its work until thread A has released its shared lock. At this point, what happens if thread A calls UpgradeToWriterLock in an attempt to obtain an exclusive lock for itself? The important question to answer is whether it is thread A or thread B that gets the exclusive lock first?

Your intuition might tell you that thread A gets the exclusive lock first because it already had acquired a shared lock. However, this is incorrect. When thread A calls UpgradeToWriterLock, its shared lock is instantly released and its request to obtain an exclusive lock is placed in the writer request queue behind any pending request. This means that thread B acquires an exclusive lock and can perform its update before thread A. It also means that the underlying data that thread A saw when it had its shared lock can be changed before it is able to obtain its exclusive lock. In other words, the data you are trying to lock down can be changed by other threads between the time that a thread calls UpgradeToWriterLock and the time it actually acquires the exclusive lock.

The key point is that a call to UpgradeToWriterLock doesn't lock down your data. It has the same effect of calling ReleaseReaderLock and then following it with a call to AcquireWriterLock. If you want to read data, verify a value, and then write data, you should do all this work after you have called AcquireWriterLock. You should observe that the lock escalation behavior of the ReaderWriterLock class is not as sophisticated as that which is provided by most DBMSs, such as SQL Server.

This column has examined how to overcome some of the limitations of using monitors by employing the ReaderWriterLock class. Unlike monitors, the ReaderWriterLock class mixes shared locking with exclusive locking to improve response times and to increase system throughput. Be sure to keep in mind that synchronization techniques using the ReaderWriterLock class work best in scenarios where there are many reads for each write and where all write operations are short in duration.

Send your questions and comments for Ted to instinct@microsoft.com.

Ted Pattison is a cofounder of Barracuda .NET, an education company that assists companies building collaborative applications using Microsoft technologies. Ted is the author of several books including Building Applications and Components with Visual Basic .NET (Addison-Wesley, 2003).


Data Points: ADO.NET and System.Transactions

Data Points
ADO.NET and System.Transactions
John Papa

Contents
Getting Started
Local Implicit Transactions
Transaction Options
Distributed Implicit Transactions
Smart Transactions with SQL Server 2005
Alternatives
Conclusion

The Microsoft® .NET Framework versions 1.0 and 1.1 represented major changes in software development. However, one important thing that did not change much was support for distributed transactions. All of that is about to change because the .NET Framework 2.0, which includes a new namespace called System.Transactions, will offer significantly improved support for distributed transactions. This new namespace enhances transactional support for managed code and makes it possible to handle transactions without using interception or reflection and without deriving from a ServicedComponent. System.Transactions is also designed to integrate well with the upcoming release of SQL Server™ 2005 and offers automatic promotion of local lightweight transactions to fully distributed transactions.

In addition, the namespace is flexible enough to handle both implicit (automatic) and explicit (manual) transactions. The implicit transactions require less code, and I find that they cover most of the situations you may encounter. You can even modify some of the settings of the implicit transactions such as the transaction timeout period. Manual transactions are useful when you want to make modifications to a transaction's behavior.

To give you a feel for how this all works, I'll take a look at a code sample that implements an implicit local transaction using the TransactionScope object and its default settings. Then I'll walk through a code sample that modifies the default transactional settings of the implicit local transaction to show you how to modify the isolation level, the timeout period, and a transaction's context. You'll see that the real power is revealed when you design your code for implicit distributed transactions. I will demonstrate how and when the System.Transactions namespace delegates the promotion of a lightweight transaction to a fully distributed transaction and I will explain when more specific control over a transaction is required. Finally, I will walk through an example that shows how to develop explicit transactions and manually choose if and when to enlist in a transaction context. (Don't forget that this column is based on a beta version of the product and details may change.)

Getting Started

The transactional examples I will present will access SQL Server 2000 and SQL Server 2005 databases using ADO.NET and some features of the System.Transactions namespace. Keep in mind that while System.Transactions interacts well with ADO.NET, it is not part of ADO.NET nor the System.Data namespace. This separation allows it to manage transactions across different resource managers including SQL Server, Microsoft Message Queue Server (MSMQ), and Oracle.

The classes I require are in Systems.Transactions.dll, so I need to add a reference to that file from my .NET project in order to take advantage of the managed transactions functionality I'm discussing. You should assume that all of my examples also include the following using statements:

using System.Data;
using System.Data.SqlClient;
using System.Transactions;

Finally, to avoid repeating the connection strings and SQL statements, I have created a few private class fields to store these values (in a real app, these would be properties and would more likely retrieve these values from a configuration file). These fields are shared by all of the methods that I will demonstrate. The first connection string accesses a SQL Server 2005 Beta 2 Northwind database, while the second connects to a SQL Server 2000 Northwind database. There is only a single UPDATE statement that will be executed against both databases, modifying a single row's value in the Customers table. I use it merely to demonstrate the transactions:

private string sCn1 =
  "Server=MyYukonServer;Database=Northwind;Integrated Security=True";
private string sCn2 =
  "Server=MySQL2000Server;Database=Northwind;Integrated Security=True";
private string sSQL =
  "UPDATE Customers SET City = 'New York' WHERE CustomerID = 'ALFKI'";

Local Implicit Transactions

The first transaction scenario I will walk through is the simplest—a local implicit transaction. This example, shown in Figure 1, shows a connection being made to a single database and a SQL command being executed against it, all wrapped inside of a transaction. There is no distributed transaction required here since only a single database is involved.

Figure 1 Implicit, Local Transaction (Close Figure 1)
void btnImplicitLocal_Click(object sender, EventArgs e)
{
    // Create the TransactionScope
    using (TransactionScope oTranScope = new TransactionScope())
    {
        using (SqlConnection oCn1 = new SqlConnection(this.sCn1))
        {
            SqlCommand oCmd = new SqlCommand(this.sSQL, oCn1);
            oCn1.Open();
            oCmd.ExecuteNonQuery();
            oCn1.Close();
        }

        // Tell the transaction scope that the transaction is in a
        // consistent state and can be committed
        oTranScope.Consistent = true; // oTranScope.Complete() in beta 2

        // The following bracket completes, commits, and disposes
        // the transaction
    }
}

The code in Figure 1 begins by creating an instance of the TransactionScope object from the System.Transactions namespace. It creates it within a using statement so that the TransactionScope is guaranteed to be disposed of, even if an exception is thrown in the middle of the code block. The using statement is handy here because it makes sure that the Dispose method is called on the TransactionScope whenever the block is exited. This is important because just before the TransactionScope is disposed, the transaction is completed. When the transaction is complete, the TransactionScope evaluates the Boolean value of the Consistent property (set to false by default). If the value of Consistent is true, the transaction is committed. If the value is false, then the transaction is rolled back. So by employing the using statement, the transaction is guaranteed to make sure the transaction completes, the Consistent flag is checked, and the transaction is committed or rolled back. Note that in the upcoming .NET Framework 2.0 Beta 2, the Consistent flag has been replaced by a new method, Complete, that is to be called on the TransactionScope object when the actions have been performed successfully.

The using keyword does not have to be used, but it is highly recommended. Note that Visual C++® 2005 and Visual Basic® 2005 both support a similar construct, so you should follow the same pattern when using those languages. The important thing here is to make sure that the TransactionScope is disposed, and the using statement just makes it simpler to do that. In addition, the using statement makes it really easy to see everything that is contained within the TransactionScope since it is commonly indented and between the curly braces.

When the SqlConnection.Open method is invoked in Figure 1, the connection is opened and automatically enlists in the current transaction scope. In fact, any connection opened inside of the using statement's curly braces is automatically included in the transaction scope and will participate in the same transaction.

The TransactionScope is smart enough not to create a fully distributed transaction with all of its overhead unless it is necessary. If the resource involved in the transaction is volatile, everything will stay in memory and the Microsoft distributed transaction coordinator (DTC) will not be involved. If the resource is a connection to SQL Server 2005, the DTC will not be involved, and SQL Server 2005 will handle the transaction. This is equivalent to opening an internal SQL transaction. If SQL Server 2000 is used, it will resolve to using a full DTC transaction.

In this example, the TransactionScope will first create a type of transaction called a LightweightTransaction, which is a form of a local, non-distributed transaction. A LightweightTransaction does not use the DTC service and therefore will not show up as a transaction in Component Services transaction statistics (Component Services | Computers | My Computer | Distributed Transaction Coordinator | Transaction Statistics).

To examine this important feature in action, set a breakpoint on the using statement where the TransactionScope is instantiated, then execute the next line by stepping forward once. Then, enter the following command

? System.Transactions.Transaction.Current
in the Immediate window, which will cause the following output to be generated:
{System.Transactions.Ltm.LightweightTransaction}
    [System.Transactions.Ltm.LightweightTransaction]:
{System.Transactions.Ltm.LightweightTransaction}
    Consistent: false
    Identifier: {a611007b-25fe-4ef5-b92c-5382bbc365be}
    IsolationLevel: System.Transactions.IsolationLevel.Serializable
    Status: System.Transactions.TransactionStatus.Active
    TransactionTraceId: {System.Transactions.TransactionTraceIdentifier}

Notice that from this output you can see that this is a LightweightTransaction object whose Consistent flag is set to false (the default value). The default IsolationLevel is set to Serializable and its Identifier is displayed. The Identifier uniquely identifies the transaction from other transactions. The Status is set to Active because you are currently at a breakpoint in the middle of an active TransactionScope code block. If you examine the properties of the current transaction as it exists immediately after it steps through the line of code that sets the Consistent property, you'll see that the Consistent property's value would change to true.

It's easy to implement transactions using the TransactionScope. If an exception is thrown, the TransactionScope is disposed, the transaction is completed, the Consistent flag is still set to false, and it is rolled back automatically. Just make sure that you do not set the Consistent property to true until you're completely finished with all the actions that should be included in the transaction (in my example, this is just before the closing curly brace of the TransactionScope's using statement).

Transaction Options

You can set the isolation level and the transaction's timeout period on the TransactionScope object by creating a TransactionOptions object. The TransactionOptions class has an IsolationLevel property, which can be used to deviate from the default isolation level of serializable and employ another isolation level such as ReadCommitted. (You should keep in mind that the isolation level is merely a suggestion to the database. Most database engines will try to use the suggested level if possible.) The TransactionOptions class also has a TimeOut property which can be used to deviate from the default of one minute.

Figure 1 used the default TransactionScope object with its default constructor. This sets the isolation level to serializable, the transaction's timeout period to one minute, and its TransactionScopeOptions setting to Required. However, if you want to alter these settings, there are 14 additional overloaded constructors you can use. While the production release of the TransactionScopeOptions enumeration is not yet final, I have listed some of the more familiar values in Figure 2. These settings allow you to control how nested transaction scopes will respond to each other. Figure 3 shows an example of how to take the implicit local transaction from Figure 1 and modify its default settings using these techniques.

Figure 3 Implicit Local Transaction with Optional Settings (Close Figure 3)
void btnImplicitLocalWithOptions_Click(object sender, EventArgs e)
{
  // Create the TransactionOptions object
  TransactionOptions oTranOpt = new TransactionOptions();

  // Set the Isolation Level
  oTranOpt.IsolationLevel =
    System.Transactions.IsolationLevel.ReadCommitted;

  // Set the timeout to be 2 minutes
  // Uses the (hours, minutes, seconds) constructor
  TimeSpan oTime = new TimeSpan(0, 2, 0);
  oTranOpt.Timeout = oTime;

  // Create the TransactionScope with the RequiresNew transaction
  // setting and the TransactionOptions object I just created
  using (TransactionScope oTranScope =
    new TransactionScope(TransactionScopeOption.RequiresNew, oTranOpt))
  {
    using (SqlConnection oCn1 = new SqlConnection(this.sCn1))
    {
      SqlCommand oCmd = new SqlCommand(this.sSQL, oCn1);
      oCn1.Open();
      oCmd.ExecuteNonQuery();
      oCn1.Close();
    }

    // Tell the transaction scope that the transaction is in a
    // consistent state and can be committed
    oTranScope.Consistent = true;

    // The following bracket completes and disposes the transaction
  }
}
Figure 2 Some TransactionScopeOptions (Close Figure 2)

TransactionScopeOptions | Description
Required | If within a currently active transaction scope, this transaction scope will join it. Otherwise it will create its own transaction scope.
RequiresNew | This transaction will create its own transaction scope.
Supports | If within a currently active transaction scope, this transaction scope will join it. Otherwise no transaction scope will be created.
NotSupported | No transaction scope will be created.

Distributed Implicit Transactions

Thus far I have shown how the TransactionScope object can be used to automatically include a single database connection in a transaction scope. Building on this ability by adding a second database connection to the transaction scope is as simple as including the database connection code in the same TransactionScope object's using statement. Figure 4 shows two SQL UPDATE statements being executed against two different databases, both inside of a single transaction scope. The first connection is made to a SQL Server 2005 version of the Northwind database. The second connection is made to a SQL Server 2000 version of Northwind.

Figure 4 Distributed Implicit Transaction (Close Figure 4)
void btnImplicitDistributed_Click(object sender, EventArgs e)
{
    // Create the TransactionScope
    using (TransactionScope oTranScope = new TransactionScope())
    {
        // Open a connection to SQL Server 2005
        using (SqlConnection oCn1 = new SqlConnection(this.sCn1))
        {
            SqlCommand oCmd1 = new SqlCommand(this.sSQL, oCn1);
            oCn1.Open();
            // At this point, the connection is in the transaction scope,
            // which is a lightweight transaction.
            oCmd1.ExecuteNonQuery();
            oCn1.Close();
        }
        // Open a connection to SQL Server 2000
        using (SqlConnection oCn2 = new SqlConnection(this.sCn2))
        {
            SqlCommand oCmd2 = new SqlCommand(this.sSQL, oCn2);
            oCn2.Open();
            // The connection is enlisted in the transaction scope,
            // which is now promoted to a distributed transaction
            // controlled by MSDTC
            oCmd2.ExecuteNonQuery();
            oCn2.Close();
        }
        // Tell the transaction scope to commit when ready
        oTranScope.Consistent = true;
        // The following bracket completes and disposes the transaction
    }
}

You could enhance this code by adding more connections, perhaps to Oracle or even MSMQ, all inside the same transaction scope. Or you could take this further and call a method that performs database actions.

These method calls would also be included in the transaction scope, as long as they are invoked within the transaction scope's using statement. If you called a method and it defined its own TransactionScope, depending on its TransactionScopeOptions setting (see Figure 2), it may join the current transaction scope or even start a new one. This gives you the flexibility to nest distributed transactions against different types of resource managers.

Smart Transactions with SQL Server 2005

Lightweight transactions can be promoted to full, distributed transactions when necessary, without any additional code. When the transaction scope is created, it creates a LightweightTransaction object. When a SQL Server 2005 connection is enlisted in the transaction scope, the transaction remains lightweight and is not under the control of the DTC. If a second connection is made to any database in the same transaction scope, the LightweightTransaction is promoted to a distributed transaction under the domain of the DTC.

SQL Server 2005 has been designed to work well with the System.Transactions features so that it does not use a distributed transaction and all of the overhead that comes along with it unless it requires a distributed transaction. So in the example in Figure 4, when the first connection is created against the SQL Server 2005 database, the transaction stays lightweight. But when it connects to the second database, regardless of the type, the transaction is promoted to a distributed transaction.

You can monitor the transaction being promoted if you step through the code. Step over the first connection that is opened. Then view the transaction statistics of the first database server and you will not see a distributed transaction being created. Next, step over the second connection that is being opened and go back and view the transaction statistics to see that a distributed transaction is now active.

If the first connection in the transaction scope is a SQL Server 2005 database, then the transaction is lightweight but can be delegated for promotion. When the subsequent database connections are opened, the transaction is promoted to a distributed transaction. However, if the first connection is opened against a SQL Server 2000 or Oracle database, note that the transaction won't be promoted because the resource managers for those databases do not support delegated transactions at this time. Therefore when the first connection's database does not support the delegated promotion of transactions, the transaction is created as a full, distributed transaction instead.

Alternatives

You can also create explicit transactions and not use the implicit, automatic transactions. For example, you could manually create a transaction and then tell your connections to explicitly enlist in the transaction scope. The problem with this technique is that it requires more code than the implicit examples and, more importantly, it increases the risk that the transaction will not complete properly if an exception is thrown (you have to manually code the exception handling logic). But just to show how this works, the code in Figure 5 creates a transaction, manually enlists a connection into the transaction, and then commits the transaction.

Figure 5 Explicit Transactions (Close Figure 5)
// Create the Transaction (it is now lightweight)
// Note that for Beta 2 this has been changed
// to CommittableTransaction.
ICommittableTransaction oTran = Transaction.Create();

// Open a connection to SQL Server 2005
using (SqlConnection oCn1 = new SqlConnection(this.sCn1))
{
    SqlCommand oCmd1 = new SqlCommand(this.sSQL, oCn1);
    oCn1.Open();

    // The connection is not yet enlisted in the transaction scope
    oCn1.EnlistTransaction();

    // Now the connection is enlisted in the transaction scope
    oCmd1.ExecuteNonQuery((ITransaction)oTran);
    oCn1.Close();
}

// Tell the transaction to commit
oTran.Commit();

Regardless of how you implement transactions, it is important to make sure that you get in and out of your transactions quickly. Transactions lock valuable resources. In general, it is a good idea to create a transaction just before you need it, open your connections, execute your action queries, and complete the transaction. You also should avoid running too much non-database code inside the transaction so that other resources aren't locked any longer than it is absolutely necessary.

These transactions also have an effect on connection pooling. When a connection is enlisted in an active transaction scope, even if the connection is closed, it is not fully released to the connection pool to be reused right away. The closed connection is still enlisted in the active transaction scope, so it remains in a special subpool until the active transaction is completed. However, once the transaction is completed, all connections will be released back to their appropriate connection pools.

Conclusion

The System.Transactions namespace can make the management of transactions quick and easy without the need to inherit from a ServicedComponent. One of the greatest features of the LightweightTransaction object is that it can determine if it needs to promote itself to a distributed transaction. The lightweight transactions are also a faster alternative to using the DTC for local transactions. The System.Transactions namespace is a step in the right direction to make it easier to handle transactions.

I want to point out that all of these features were written using the .NET Framework Beta 1 and SQL Server 2005 Beta 2. Therefore, I would expect that some changes to the interfaces or the constructors (the usual suspects) will occur before the final releases of these products. Thus you may need to modify your code once the final products are released.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Cutting Edge: Adding a Context Menu to ASP.NET Controls

Cutting Edge
Adding a Context Menu to ASP.NET Controls
Dino Esposito

Code download available at: CuttingEdge0502.exe (138KB)
Contents
Outline of the ContextMenu Control
Usage of the ContextMenu Control
The Programming Interface
Implementation of the Control
Remarks on Absolute Positioning
Putting It All Together
The Menu in ASP.NET 2.0

Although the context menu is a common element of most desktop applications, it is still fairly uncommon in Web application names because it doesn't map well to a server-based technology like ASP.NET. To get context menu functionality, your browser needs strong DHTML support and a rich eventing model, both of which you get in Microsoft® Internet Explorer 5.0 and newer versions, as well as in Netscape 6.0 and others. However, the various browsers' object models, although nearly identical in functionality, feature different members and names, and require you to map events and objects from one model to the other.

In this column, I'll create an ASP.NET context menu that addresses the Internet Explorer object model. Along the way, I'll call out features that can work with other browsers with some modifications. The code download for this column works with ASP.NET 1.x, but can be compiled under ASP.NET 2.0 Beta 1 as well.

Outline of the ContextMenu Control

The MSDN® Library contains several examples of DHTML context menus. Each provides a different implementation of the same basic idea. A context menu is a modular piece of markup code that can move around the page. It consists of two distinct blocks—the user interface and the script code to connect the UI to whatever page control the user has right-clicked. The UI provides a list of clickable elements—the menu items—and defines their text, icon, command name, target URL, tooltip, and whatever else you feel is important to show. The context menu's UI is part of the page and takes up space in the page's control tree. (As a result, too many context menus can lead to performance issues as ASP.NET sends more content to the browser than will be needed by most users in a typical situation.) When the user right-clicks on an element in the page, some JavaScript fires, moves the menu's UI close to the coordinates of the clicked point, and toggles the visibility of the menu.

The context menu remains invisible until the user right-clicks on a page element bound to the context menu. The page element receives a script event and pops up a context menu in response. The script event depends on the browser's capabilities; in Internet Explorer 5.0, and newer versions, it's the OnContextMenu event. In Netscape 6.0 and newer versions you need to use OnMouseUp. (You can also make OnMouseUp work for Internet Explorer 5.0+ browsers but this requires a little more coding.) Upon receiving the event notification, the client script retrieves the block of UI code that forms the context menu and moves it to the exact location where the click occurred. It also toggles on the visibility property of the popup panel. When the user clicks on any of the menu's menu items, the page posts back and raises a server-side event. On the server, clicking on a context menu is no different from clicking on a traditional button.

What if the user decides to abort the operation once she has displayed the context menu? In Windows, the standard context menus are dismissed by hitting the Esc key or by clicking anywhere outside the menu region, so you'll need to build this behavior into your Web version. Note that for accessibility reasons, hitting Esc should only work when the context menu is actively selected since the Esc key could be used by other elements to provide keyboard shortcuts.

Here I'll also give the ContextMenu control the ability to hide when the user moves the mouse outside the control's UI. This is achieved using a script handler for the OnMouseLeave event.

Usage of the ContextMenu Control

Assuming the existence of my ASP.NET ContextMenu control, let's see how to use it to add one or more context menus to an ASP.NET page (you'll see how to build the control later in this column). You start by dropping one or more instances of the ContextMenu control onto a Web Form in Visual Studio® .NET. Next, add to each instance as many menu items as needed and configure them all with tooltips, command names, and whatever else you want to add, such as keyboard shortcuts and links to help topics. The command name will be used to track the clicked item once the page posts back in response to a click on the context menu; thus it must be a unique string within the collection of items for an instance of the control.

For the menu to pop up you need to attach some code to the HTML tag's OnContextMenu event handler. The actual code executed depends on a number of parameters, such as the x,y coordinates of the click, the element clicked, and the instance of the context menu you want to use. Note that this way you entirely replace the standard browser's context menu, if any exists. The JavaScript code to attach to the OnContextMenu event handler is determined dynamically at run time. The ContextMenu control will expose a collection property to contain references to bound controls, which will be given an oncontextmenu attribute at run time, thanks to the implementation of the ContextMenu control. With this completed, you're ready to test the control on some pages.

But first let me spend a minute on the applicability of a design-time binding mechanism between the ContextMenu control and any page elements that will display the menu. Ideally, you would have a ContextMenuId property on each Web control exposed directly by the base class Control. Then you could go to the Properties window, select the property, and see the list of ContextMenu controls on the page. Of course, the ContextMenu control I'm discussing is one I've created; ContextMenuId doesn't exist in ASP.NET 1.x or in the upcoming ASP.NET 2.0.

In the Visual Studio .NET 2003 IDE, an ASP.NET extender control would do the job quite nicely. Extender controls (see Cutting Edge November 2003) extend existing controls without creating new specific classes and, more importantly, can add the defined extensions to a variety of controls much like a common base class would do. The code that performs the extension is loaded into an external class that plugs into the base class at run time. There's no sophisticated interception layer in the middle. The excellent design-time support in Visual Studio .NET makes the presence of provider controls almost invisible and a few lines of auto-generated code bind the extender with the array of extendees.

I'm not going to use extenders in this column for a couple of reasons. One is that support for extenders in Visual Studio .NET 2003 is partially broken when it comes to extending the capabilities of controls on a Web Form. (See Extender provider components in ASP.NET: an IExtenderProvider implementation for more details and a workaround.) Second, there will be no Component Tray area in the ASP.NET designer of Visual Studio 2005. The Web Forms designer now supports only ASP.NET controls and ignores nonvisual components like extenders. Visual Studio 2005 no longer relies on an InitializeComponent section, and there's never any tool-generated code inserted into your code files. ASP.NET controls are not designed to have a context menu, so you can bind them to a custom context menu only through the implementation of the ContextMenu control itself. I've chosen a programmatic approach here, but a solution similar to how ASP.NET validator controls are linked to the controls being validated would also work.

The Programming Interface

Our ContextMenu control inherits from WebControl and acts as a naming container:

public class ContextMenu : WebControl, INamingContainer

Figure 1 details the members of the control. The key property is the ContextMenuItems collection, which contains objects of type ContextMenuItem, each of which represents a menu item. The source code of the ContextMenuItem class is shown in Figure 2. Each menu item has display text, a command name, and a tooltip. You can extend this class in various ways, for example by adding an image URL, a disabled state, or a target URL. The display text is the text rendered on the menu; the command name is a unique string that describes the command associated with the item. Finally, the tooltip describes the purpose of the currently selected item.

Figure 2 The ContextMenuItem Class (Close Figure 2)
[TypeConverter(typeof(ExpandableObjectConverter))]
public class ContextMenuItem
{
    public ContextMenuItem() {}

    public ContextMenuItem(string text, string commandName)
    {
        _text = text;
        _commandName = commandName;
    }

    private string _text;
    private string _commandName;
    private string _tooltip;

    public string Text
    {
        get {return _text;}
        set {_text = value;}
    }

    public string CommandName
    {
        get {return _commandName;}
        set {_commandName = value;}
    }

    public string Tooltip
    {
        get {return _tooltip;}
        set {_tooltip = value;}
    }
}
Figure 1 Members of the ContextMenu Control (Close Figure 1)

Property | Description
AutoHide | Indicates whether the context menu should be dismissed as the user moves out of the control's boundaries
BoundControls | Gets the collection of controls for which the context menu should be displayed
CellPadding | Gets and sets the number of pixels around each menu item
ContextMenuItems | Gets the collection of the menu items
RolloverColor | Gets and sets the background color to set when the mouse hovers over any menu items
Method | Description
GetEscReference | Returns the JavaScript code needed to dismiss the current context menu when the user hits the Esc key
GetMenuReference | Returns the JavaScript code needed to attach the current context menu to the specified HTML element
GetOnClickReference | Returns the JavaScript code needed to dismiss the current context menu when the user clicks outside the menu
Event | Description
ItemCommand | Occurs when the user clicks on a menu item

When the user clicks on a menu item, the page posts back and a server-side ItemCommand event is fired. By handling that event, the host page can execute some code in response to the user clicking the menu. Figure 3 shows the ContextMenu control in the Visual Studio .NET 2003 IDE in the sample project.

Figure 3 ContextMenu Control in the Visual Studio .NET 2003 IDE
Figure 3 ContextMenu Control in the Visual Studio .NET 2003 IDE

For the context menu to work, you need to fill the ContextMenuItems collection with menu item objects, adjust some visual styles, and add at least one control to the BoundControls collection. Then, point the browser to the sample page and right-click on any bound control. You should see something like Figure 4.

Figure 4 ContextMenu Control in Action
Figure 4 ContextMenu Control in Action

Each menu item consists of a LinkButton control with an internal handler bound to the Click event. When a click is detected, the page posts back and fires the Click event. In turn, the predefined handler bubbles the event up one level, changing its name to ItemCommand.

The control also features a few visual properties like CellPadding, RolloverColor, and AutoHide. As mentioned, in Windows a context menu disappears when you click outside its boundaries or when you hit the Esc key. For a Web-based context menu, the AutoHide property adds an OnMouseLeave script handler to the root tag of the ContextMenu control so that the subtree is hidden from view as soon as the user's mouse leaves the area of the control. A more fine-grained approach would implement AutoHide as a settable property so that the user can set the desired time for the mouse to hover outside of the context menu before it disappears.

To be able to hide the context menu by clicking or pressing Esc, you need to add event handlers:

<body onkeypress="..." onclick="...">

Script handlers can be programmatically added to virtually any page element, provided that the page element is marked with runat=server. This fact creates a logical dependency between the ContextMenu control and the page. In addition, it forces you to have an extra server control defined on the page. Sure, one extra control instantiated at run time does not have a huge effect on performance, but why have a useless control that you instantiate only because you want to consume another control more easily? In this alternative technique, the final effect is the same: the body catches both Esc hits and mouse clicks and you save the overhead of one server control:

  <body onkeypress="<% =
     ContextMenu1.GetEscReference() %>"
     onclick="<% =
     ContextMenu1.GetOnClickReference() %>">
Let's examine the control's implementation in more detail.

Implementation of the Control

The heart of the ContextMenu control lies in the overridden CreateChildControls method. In this method, the control creates its own UI and injects any needed script into the host page. As mentioned, the user interface of the ContextMenu control is divided into two parts—graphics and script. Let's consider graphics first.

CreateChildControls generates the HTML block that will then be moved around the page to serve as a popup menu on demand. From this perspective, a context menu is a <DIV> tag that contains a table with one row for each menu item. Using a table here is arbitrary but simplifies a number of development tasks (such as borders and overall layout) and lends itself very well to further enhancements like side images:

HtmlGenericControl div = new HtmlGenericControl("div");
div.ID = "Root";
div.Style["Display"] = "none";
div.Style["position"] = "absolute";
if (AutoHide)
   div.Attributes["onmouseleave"] = "this.style.display='none'";

The outermost <DIV> tag is hidden from view using cascading style sheets (CSS) styles and is marked for absolute positioning. If auto-hiding is enabled, the element also handles the mouse-leave event to hide itself when the mouse exits the area. It is interesting to note the difference between the onmouseout and onmouseleave events. The former occurs when the mouse is moved onto a new element. The latter fires only if the mouse is moved out of the boundaries of an object. For example, imagine you have a table with two rows. The onmouseout event on the table fires each time you move the mouse between the rows; the onmouseleave event fires only if you move the mouse outside the table.

The table contains as many rows as there are menu items to display. Each row has a single cell filled with a LinkButton object. The menu is built through a loop like this:

foreach(ContextMenuItem item in ContextMenuItems)
{
   TableRow menuItem = new TableRow();
   menuTable.Rows.Add(menuItem);

   TableCell container = new TableCell();
   menuItem.Cells.Add(container);

   LinkButton button = new LinkButton();
   container.Controls.Add(button);
   ...
}

The row cell has a couple of script handlers—onmouseover and onmouseout—to provide rollover effects. When the mouse hovers over the item, the background is set to a different color. The original background is restored when the mouse moves away. The BackColor property inherited from WebControl indicates the default background color. The new RolloverColor property gets and sets the highlight color:

string color = String.Format(ContextMenu.OnMouseOver,
   ColorTranslator.ToHtml(RolloverColor));
container.Attributes["onmouseover"] = color;
color = String.Format(ContextMenu.OnMouseOut,
   ColorTranslator.ToHtml(BackColor));
container.Attributes["onmouseout"] = color;

You need to translate the .NET System.Drawing.Color value into a string that represents a valid HTML color. Interestingly, neither the ToString method of the Color class nor the Name property on the same class returns the correct HTML string in all cases, nor is it intended to do so. The Name property works fine in all cases but one. When the color doesn't match any known color, the property returns the RGB components of the color plus the alpha channel. To obtain the correct HTML color string you must remove the alpha channel (usually a leading hex ff string) and replace it with the # symbol. Fortunately, the System.Drawing.ColorTranslator class does this automatically.

The link button requires some adjustments to work properly. In particular, you set the link width to 100 percent to ensure that the hand cursor displays throughout the whole row without additional settings. Likewise, you set text, tooltip, and command name getting data from the corresponding menu item object. Finally, you assign an internal handler to the Click event:

LinkButton button = new LinkButton();
container.Controls.Add(button);
button.Click += new EventHandler(ButtonClicked);
button.Width = Unit.Percentage(100);
button.ToolTip = item.Tooltip;
button.Text = item.Text;
button.CommandName = item.CommandName;

As a result, the page that hosts the context menu posts back when the user clicks a link button. On postback, the sender of the event is identified as the LinkButton control in the ContextMenu naming container and is given a chance to handle the postback event. The Click internal handler packs any information into a new ItemCommand event and raises it (see Figure 5).

Figure 5 Button Click Handler (Close Figure 5)
private void ButtonClicked(object sender, EventArgs e)
{
   LinkButton button = sender as LinkButton;
   if (button != null)
   {
      CommandEventArgs args = new CommandEventArgs(
         button.CommandName, button.CommandArgument);
      OnItemCommand(args);
   }
}

protected virtual void OnItemCommand(CommandEventArgs e)
{
   if (ItemCommand != null) ItemCommand(this, e);
}

The code in the host page that receives an ItemCommand event is passed the instance of the ContextMenu which fired the event and the command name associated with the clicked item.

The table built so far represents the user interface of the context menu. It is initially placed anywhere in the page and is hidden from view using CSS styles. This piece of HTML code (which has absolute positioning capabilities) will display a context menu exactly where the user right-clicks. JavaScript code to intercept the event and move the menu block is also required (see Figure 6). The __showContextMenu function sets the left and top properties of the ContextMenu's style object to make it display around the point of click. The small negative offset ensures that the mouse is already over the menu when the menu displays. This prevents small movements from causing the mouse to cross the boundaries of the element resulting in the hiding. The bubbling of the mouse event must also be stopped so that elements higher in the document object model hierarchy won't handle the right-click event.

Figure 6 JavaScript to Activate the Context Menu (Close Figure 6)
<script language="Javascript">

function __showContextMenu(menu)
{
    var menuOffset = 2
    menu.style.left = window.event.x - menuOffset;
    menu.style.top = window.event.y - menuOffset;
    menu.style.display = "";
    window.event.cancelBubble = true;

    return false;
}

function __trapESC(menu)
{
    var key = window.event.keyCode;
    if (key == 27)
    {
        menu.style.display = 'none';
    }
}

</script>

So, who calls the __showContextMenu function? The browser does when it finds the function linked to the oncontextmenu event of an HTML element. As mentioned earlier, oncontextmenu is an event specific to Internet Explorer that is not recognized by Netscape browsers. An alternative is to use onmouseup to catch any mouse release and check for the right button.

The ContextMenu control is also in charge of adding an oncontextmenu handler to each control or page element that registers to get one. I defined two ways for an element to get its own context menu. The BoundControls collection property (see Figure 1) is an array that the page's code fills with references to all page controls that require a particular context menu. Here's an example:

void Page_Load(object sender, EventArgs e)
{
   ContextMenu1.BoundControls.Add(Button1);
   ...
}

The preceding code produces the following markup:

<input type=submit ... oncontextmenu="__showContextMenu(...)" />

When the user right-clicks on the control, the context menu pops up. This approach requires that any page element with a context menu be a server control, a requirement that may or may not be acceptable. For example, suppose you want to replace the default context menu for an image. Must you mark the <img> tag as runat=server? Not necessarily. Here's an example:

<img oncontextmenu="<% = ContextMenu1.GetMenuReference() %>" src="...">
The GetMenuReference method on the control returns the script call that brings the context menu up. The page element still features the desired menu without defining a new server control.

Remarks on Absolute Positioning

The ContextMenu control as developed here requires absolute positioning that not all browsers support. However, a browser that supports a rich object model and a good deal of events will probably have advanced positioning capabilities as well.

As far as Internet Explorer is concerned, there's another way to implement a context menu. Instead of moving a DIV's position around the page you could just create a popup window and display it at the specified position. Then you dynamically load the DIV representing the context menu into the popup document's body. This technique is demonstrated in the article at Using the Popup Object.

While implementing this ContextMenu control, I first used the aforementioned technique and created and displayed a popup window. I discovered a couple of things about popups. One benefit is that the popup object automatically works like a desktop menu and disappears as soon as you click outside or hit Esc. There's no need to write a single of line of code for this behavior to occur.

On the downside, I experienced problems with the View Source function in Internet Explorer 6.0. According to my tests, having the context menu in the popup object alters the page's tree of elements and this seems to have repercussions on the View Source window; it just doesn't show up. Also, I want my context menu to work like an interactive ASP.NET control meaning that I'd like it to post back to the current page (bringing back view state and input fields) rather than jumping to a completely different URL. Posting to the parent window from inside a popup window is still possible but it requires changes at the page level and a bit of tricky scripting. (This is easier in ASP.NET 2.0, thanks to cross-page posting.) In the MSDN examples, it works just great because the context menu simply links to an external URL.

An ASP.NET postback is a little trickier; using absolute positioning maintains the context menu in the realm of the current page and doesn't apply changes to the page's elements tree. As a result, none of the aforementioned drawbacks apply. As for capturing the events that typically dismiss the popup menu, you can still consider capturing mouse events. This can be done using a number of Internet Explorer DHTML methods as described here: How To Create a Mouse Capture Context Menu.

Putting It All Together

The most important part of the ContextMenu control is the list of menu items. You can specify these items either programmatically or at design time. The following code snippet is an example of the design-time approach:

<cc1:contextmenu id="ContextMenu1" runat="server">
   <cc1:ContextMenuItem Text="Do This"
       CommandName="ThisCommand" Tooltip="..." />
   <cc1:ContextMenuItem Text="Do That"
       CommandName="ThatCommand" Tooltip="..." />
   <cc1:ContextMenuItem />
   <cc1:ContextMenuItem Text="Think ..."
       CommandName="ThinkCommand" Tooltip="... " />
</cc1:contextmenu>

The empty <cc1:ContextMenuItem> tag indicates a separator. Note that you can have Visual Studio .NET properly handle child tags simply by using a handful of well-placed attributes:

[DesignerSerializationVisibility(
 DesignerSerializationVisibility.Content)]
[PersistenceMode(PersistenceMode.InnerDefaultProperty)]
public ContextMenuItemCollection ContextMenuItems {...}

This configuration doesn't let you host other types of child tags, however. If you did that, a parser exception would be thrown. This means, for example, that you can't serialize the contents of the BoundControls in the body of the ContextMenu root tag. By using a different set of design-time attributes, you can work around it. (I expect to cover these aspects of control design in the future).

If you double-click on the context menu control in the Visual Studio .NET designer an event handler is added and is registered with the ContextMenu's ItemCommand event. You can then handle each action by name, filling in the context menu to look something like the following:

void ContextMenu1_ItemCommand(object sender, CommandEventArgs e)
{
   switch(e.CommandName)
   {
      case "ThinkCommand":
           ...
           break;
      case "ThisCommand":
           ...
           break;
      default:
           ...
           break;
   }
}

In Figure 3, you see the ContextMenu control rendered at design time. The ASP.NET designer just calls RenderControl for each control hosted in the page being designed. When this applies to ContextMenu, though, how could RenderControl render out a row to mimic the selected menu item? That is the effect of a custom designer for the ContextMenu control. You'll find the source code of this component in the companion code. In a nutshell, it captures the HTML string generated by RenderControl and modifies it by adding an extra row with a different background color. In this way, the user has a clearer idea of the control output.

The Menu in ASP.NET 2.0

The ContextMenu accompanying this column is written for ASP.NET 1.x, but it can be easily ported to ASP.NET 2.0. As you probably know, in ASP.NET 2.0 there's a brand new Menu control. However, you should not necessarily use it as a context menu in ASP.NET 2.0 applications. It's too heavy and designed for a different scenario. A ContextMenu control complements the ASP.NET 2.0 Menu control, which is specifically designed to operate as a static menu and lacks two key settings for performing the functions of a context menu: it can't be hidden and doesn't support absolute positioning. Both settings could be added declaratively by editing the control's markup. On the other hand, the ASP.NET 2.0 Menu control provides some key advantages, including support for multilevel, hierarchical menus, accessibility, and scrollbars for resized browser windows. Deciding which one best suits your needs is completely up to you.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the new book Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Wicked Code: Five Undiscovered Features on ASP.NET 2.0

Wicked Code
Five Undiscovered Features on ASP.NET 2.0
Jeff Prosise

Code download available at: WickedCode0502.exe (121KB)
Contents
Updating Browser Displays in (Almost) Real Time
Encrypted Configuration Sections
Auto-Culture Handling
Custom Expression Builders
Custom Web Events

By now, developers everywhere have had the opportunity to download the first beta of the Microsoft® .NET Framework 2.0. ASP.NET developers who have played with it are no doubt salivating at all the cool new features. From Master Pages to declarative data access to new controls to a new provider-based state management architecture, ASP.NET 2.0 offers myriad ways to do more with less code. And with Beta 2 just around the corner, now is the time to get serious about ASP.NET 2.0.

You may have read some of the many books and magazine articles previewing the upcoming features. You might even have seen a live demo at a conference or user group meeting. But how well do you really know ASP.NET 2.0? Did you know, for example, that those wonderful $ expressions used to declaratively load connection strings and other resources can be extended to create $ expressions of your own? Did you realize that the new ASP.NET 2.0 client callback manager provides an elegant solution to the problem of keeping browser displays in sync with constantly changing data on the server? Did you know that you can encrypt sections of Web.config to prevent connection strings and other potentially injurious data from being stored in plaintext?

Just underneath the surface of ASP.NET 2.0 lies a treasure trove of new features and capabilities that have received little coverage. This installment of Wicked Code presents five of them. All the code samples were tested against Beta 1; some may require modification for Beta 2. And as usual, remember things can change as these are beta versions.

Updating Browser Displays in (Almost) Real Time

In "An Overview of the New Services, Controls, and Features in ASP.NET 2.0" in the June 2004 issue of MSDN®Magazine, I wrote about the ASP.NET 2.0 new client callback manager and demonstrated how it can be used to transmit XML-HTTP callbacks to Web servers to convert ZIP codes into city names. Dino Esposito delved more deeply into XML-HTTP callbacks in his August 2004 Cutting Edge column ("Script Callbacks in ASP.NET").

XML-HTTP callbacks enable browsers to make calls to Web servers without performing full-blown postbacks. The benefits are numerous. XML-HTTP callbacks transmit less data over the wire, thereby using bandwidth more efficiently. XML-HTTP callbacks don't cause the page to flicker because they don't cause the browser to discard the page as postbacks do. Furthermore, XML-HTTP callbacks execute less code on the server because ASP.NET short-circuits the request so that it executes the minimum amount of code necessary. Inside an XML-HTTP callback, for example, a page's Render method isn't called, significantly reducing the time required to process the request on the server.

Once they learn about the ASP.NET 2.0 client callback manager, most developers can envision lots of different uses for it. But here's an application for XML-HTTP callbacks that you might not have thought of. I regularly receive e-mail from developers asking how to create two-way connections between browser clients and Web servers. The scenario generally involves an ASP.NET Web page displaying data that's continually updated on the server. The goal is to create a coupling between the browser and the Web server so that when the data changes on the server, it's automatically updated on the client, too.

It sounds reasonable enough, but transmitting asynchronous notifications from a Web server to a browser is a nontrivial problem. However, XML-HTTP callbacks provide a handy solution. Rather than try to contrive a mechanism for letting a Web server send notifications to a browser, you can have the browser poll the server in the background using efficient XML-HTTP callbacks. Unlike META REFRESH tags, an XML-HTTP solution causes no flashing in the browser, producing a superior user experience. And unlike more elaborate methods that rely on maintaining open ports, an XML-HTTP-based solution, properly implemented, has minimal impact on scalability.

The Web page shown in Figure 1 demonstrates how dynamic page updates using XML-HTTP callbacks work. To see for yourself, launch the page called DynamicUpdate.aspx in your browser. Then open the file named Stocks.xml in the Data directory. The fictitious stock prices displayed in DynamicUpdate.aspx come from Stocks.xml. Now change one of the stock prices in Stocks.xml and save your changes. After a brief pause (two or three seconds on average), the browser's display updates to reflect the change.

Figure 1 Dynamic Page Updates
Figure 1 Dynamic Page Updates

What's the magic that allowed the update to occur? XML-HTTP callbacks, of course. Figure 2 lists the source code for the codebehind class that serves DynamicUpdate.aspx. Page_Load uses the new ASP.NET 2.0 Page.GetCallbackEventReference method to obtain the name of a function it can call to initiate a callback. Then it registers its own __onCallbackCompleted function, which uses client-side script to update the client's display, to be called when a callback returns. Finally, it registers a block of startup script that uses window.setInterval to call the function that initiates callbacks—the function whose name was returned by GetCallbackEventReference. Once the page loads, it polls the server every five seconds for updated data. Any changes made to the data on the server appear in the browser after a short delay.

Figure 2 DynamicUpdate.aspx.cs (Close Figure 2)
using System;
using System.Data;
using System.Configuration;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;

public partial class DynamicUpdate_aspx
{
    static readonly string _script1 =
        "<script language=\"javascript\">\n" +
        "function __onCallbackCompleted (result, context)\n" +
        "{{\n" +
        "var args = result.split (';');\n" +
        "var gridView = document.getElementById('{0}');\n" +
        "gridView.rows[1].cells[1].childNodes[0].nodeValue = args[0];\n" +
        "gridView.rows[2].cells[1].childNodes[0].nodeValue = args[1];\n" +
        "gridView.rows[3].cells[1].childNodes[0].nodeValue = args[2];\n" +
        "}}\n" +
        "</script>";

    static readonly string _script2 =
        "<script language=\"javascript\">\n" +
        "window.setInterval (\"{0}\", 5000);\n" +
        "</script>";

    void Page_Load(object sender, EventArgs e)
    {
        // Get a callback event reference
        string cbref = GetCallbackEventReference (this, "null",
            "__onCallbackCompleted", "null", "null");

        // Register a block of client-side script containing
        // __onCallbackCompleted
        RegisterClientScriptBlock ("MyScript", String.Format(
            _script1, GridView1.ClientID));

        // Register a block of client-side script that launches
        // XML-HTTP callbacks at five-second intervals
        RegisterStartupScript ("StartupScript", String.Format(
            _script2, cbref));
    }

    // Server-side callback event handler
    string ICallbackEventHandler.RaiseCallbackEvent(string arg)
    {
        // Read the XML file into a DataSet
        DataSet ds = new DataSet ();
        ds.ReadXml(Server.MapPath ("~/Data/Stocks.xml"));

        // Extract the stock prices from the DataSet
        string amzn = ds.Tables[0].Rows[0]["Price"].ToString ();
        string intc = ds.Tables[0].Rows[1]["Price"].ToString();
        string msft = ds.Tables[0].Rows[2]["Price"].ToString();

        // Return a string containing all three stock prices
        // (for example, "10.0;20.0;30.0")
        return (amzn + ";" + intc + ";" + msft);
    }
}

Currently, DynamicUpdate.aspx.cs doesn't use bandwidth as efficiently as it could because every callback returns all three stock prices, even if the data hasn't changed. You could make DynamicUpdate.aspx.cs more efficient by modifying it to return only the data that has changed and to return nothing at all if no prices have changed. Then you'd have the best of both worlds: a scalable, lightweight mechanism for detecting updates on the server, and one that transmits only as much information as it must and not a single byte more. That's a win no matter how you look at it.

Encrypted Configuration Sections

ASP.NET 1.x texts frequently advise developers to put database connection strings in the <appSettings> section of Web.config. Doing so makes retrieving connection strings easy, and it centralizes the data so that changing a connection string in one place propagates the change throughout the application. Unfortunately, ASP.NET 1.x has no oblique support for encrypting connection strings (or any other data, for that matter) in Web.config. That leaves programmers with a Faustian choice: store connection strings in plaintext where they're vulnerable to hackers, or store them in encrypted form and write lots of code to decrypt them after you've already fetched them.

One of the tenets of writing secure ASP.NET code is to avoid storing as plaintext any secrets, passwords, connection strings, or other data that could be misused if divulged. To prevent such risky behavior, ASP.NET 2.0 lets you encrypt individual sections of Web.config. Encryption is transparent to the application. You don't have to do anything special to read an encrypted string from Web.config; you just read it as normal and if it's encrypted, it's automatically decrypted by ASP.NET. Not a single line of custom code is required. ASP.NET also offers you a choice of two encryption modes. One uses triple-DES encryption with a randomly generated key protected by RSA; the other encryption mode uses triple-DES encryption as implemented by the Windows® Data Protection API (DPAPI). You can add support for other encryption techniques by plugging in new data protection providers. Once encrypted, data stored in Web.config remains theoretically secure even if the Web server is compromised and the entire Web.config file falls into the wrong hands. Without the decryption key, the data can't be decoded.

Sometime before ASP.NET 2.0 ships, the ASP.NET page of the IIS Microsoft Management Console (MMC) snap-in will probably be upgraded with a GUI for encrypting and decrypting sections of Web.config. Beta 1 lacks such a tool. In fact, most ASP.NET developers I've talked to aren't even aware that Web.config supports encrypted configuration sections.

The good news is that you don't have to wait for tool support to take advantage of encrypted configuration sections. The new ASP.NET 2.0 configuration API has methods you can use to build a tool of your own. One call to ConfigurationSection.ProtectSection is sufficient to encrypt a configuration section; a subsequent call to ConfigurationSection.UnProtectSection decrypts it. Following a successful call to either method, you call Configuration.Update to write changes to disk. (Configuration.Update will probably be renamed to Configuration.Save in Beta 2.) Additionally, the aspnet_regiis.exe tool in Beta 1 provides some command-line support (see the -p* options).

The page in Figure 3 offers a simple UI for encrypting and decrypting the <connectionStrings> section of Web.config. If you'd like, you can modify it to support encryption of other configuration sections, too. To encrypt <connectionStrings>, simply click the Encrypt button. To decrypt, click Decrypt (tricky, huh?).

Figure 3 Encrypting and Decrypting <connectionStrings> (Close Figure 3)

ProtectSection.aspx
<%@ Page Language="C#" CompileWith="ProtectSection.aspx.cs"
  ClassName="ProtectSection_aspx" %>

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Untitled Page</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <asp:Button ID="Button1" Runat="server"
            Text="Encrypt Connection Strings" Width="214px"
            Height="73px" OnClick="Button1_Click" />
        <asp:Button ID="Button2" Runat="server"
            Text="Decrypt Connection Strings" Width="214px"
            Height="73px" OnClick="Button2_Click" />
    </div>
    </form>
</body>
</html>
ProtectSection.aspx.cs
using System;
using System.Data;
using System.Configuration;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;

public partial class ProtectSection_aspx
{
    void Button1_Click(object sender, EventArgs e)
    {
        Configuration config = Configuration.GetWebConfiguration(
            Request.ApplicationPath);
        ConfigurationSection section =
            config.Sections["connectionStrings"];
        section.ProtectSection ("DataProtectionConfigurationProvider");
        config.Update ();
    }

    void Button2_Click(object sender, EventArgs e)
    {
        Configuration config = Configuration.GetWebConfiguration(
            Request.ApplicationPath);
        ConfigurationSection section =
            config.Sections["connectionStrings"];
        section.UnProtectSection ();
        config.Update();
    }
}

Figure 4 shows the <connectionStrings> section of the accompanying Web.config file before and after encryption (note that the encrypted string is really more than a thousand characters long and has been excerpted here). You should also notice the <protectedData> section added to Web.config containing information needed to decrypt the connection strings. Significantly, <protectedData> doesn't contain the decryption key. When the Windows DPAPI is used to perform the encryption as it is here, the decryption key is autogenerated and locked away in the Windows Local Security Authority (LSA).

Figure 4 <connectionStrings> Before and After Encryption (Close Figure 4)

connectionStrings Before Encryption
<connectionStrings>
  <add name="Pubs" connectionString=" Server=localhost;Integrated
    Security=True;Database=Pubs" providerName="System.Data.SqlClient" />
  <add name="Northwind" connectionString="Server=localhost;Integrated
    Security=True;Database=Northwind"
    providerName="System.Data.SqlClient" />
</connectionStrings>
connectionStrings After Encryption
<connectionStrings>
  <EncryptedData>
    <CipherData>
      <CipherValue>AQAAANCMnd8BfdERjHoAw ...</CipherValue>
      </CipherData>
    </EncryptedData>
  </connectionStrings>

<protectedData>
  <protectedDataSections>
    <add name="connectionStrings"
      provider="DataProtectionConfigurationProvider" />
  </protectedDataSections>
</protectedData>

ASP.NET 2.0 permits all but a handful of configuration sections to be encrypted. The <httpRuntime> section, for example, doesn't support encryption because it's accessed by the tiny fraction of ASP.NET that's built from unmanaged code. But everything that matters can be encrypted, and by using encrypted configuration sections judiciously, you can erect an additional barrier for hackers attempting to steal secrets from your site.

Auto-Culture Handling

Developers charged with the task of localizing Web sites in ASP.NET 1.x often found themselves writing code into Global.asax to inspect Accept-Language headers and attach CultureInfo objects representing language preferences to the threads that handle individual requests. The code they wrote to do that frequently took the form shown in Figure 5.

Figure 5 Attach Language Preference to Incoming Requests (Close Figure 5)
void Application_BeginRequest (Object sender, EventArgs e)
{
    try
    {
        if (Request.UserLanguages.Length > 0)
        {
            CultureInfo ci = CultureInfo.CreateSpecificCulture(
              Request.UserLanguages[0]);
            Thread.CurrentThread.CurrentCulture = ci;
            Thread.CurrentThread.CurrentUICulture = ci;
        }
    }
    catch (ArgumentException)
    {
        // Do nothing if CreateSpecificCulture fails
    }
}

A new feature of ASP.NET called auto-culture handling obviates the need for such code. Auto-culture handling is enabled for individual pages by including Culture="auto" and UICulture="auto" attributes in the pages's @ Page directives. It's enabled site-wide by including a <globalization culture="auto" uiCulture="auto" /> element in Web.config. However you choose to enable it, auto-culture handling has an interesting effect: it maps Accept-Language headers to CultureInfo objects and attaches them to the current thread, just like the code in Figure 5.

To demonstrate, check out the page in Figure 6. Its output consists of a Calendar control and a text string showing today's date. The latter is generated by a call to DateTime.ToShortDateString. If a user who has configured her browser to transmit Accept-Language headers specifying French as her preferred language visits the page, she sees the page depicted in Figure 7. If auto-culture handling were not enabled, the page would appear no different to French users than it would to other users. The key is the @ Page directive turning auto-culture handling on. ASP.NET does the hard part; you do the rest.

Figure 6 Auto-Culture Handling (Close Figure 6)

AutoCulture.aspx
<%@ Page Culture="auto" UICulture="auto" Language="C#"
  CompileWith="AutoCulture.aspx.cs" ClassName="AutoCulture_aspx" %>

<html>
  <body>
    <form runat="server">
      <asp:Calendar ID="Calendar1" Runat="server" ...>
        ...
      </asp:Calendar>
      <h2><asp:Label ID="Label1" Runat="server" /></h2>
    </form>
  </body>
</html>
AutoCulture.aspx.cs
using System;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.HtmlControls;

public partial class AutoCulture_aspx
{
    void Page_Load (object sender, EventArgs e)
    {
        Label1.Text = "Today's date is " +
            DateTime.Now.ToShortDateString();
    }
}
Figure 7 Auto-Culture Handling Enabled
Figure 7 Auto-Culture Handling Enabled

Obviously, there's more to localizing an entire Web site than simply enabling auto-culture handling. Auto-culture handling does not, for example, localize static Web site content. But ASP.NET 2.0 offers other new localization features as well, including the new <localize> tag for localizing static content and $ Resources expressions for loading localization resources declaratively. Put simply, ASP.NET 2.0 makes it dramatically easier than ASP.NET 1.x to provide localized content to international users.

Custom Expression Builders

ASP.NET 2.0 developers are encouraged to store database connection strings in the <connectionStrings> section of the registry. Connection strings stored that way can be loaded declaratively, as demonstrated by the following tag declaring a SqlDataSource:

<asp:SqlDataSource ... Runat="server"
  ConnectionString="<%$ ConnectionStrings:Northwind %>"
/>
"<%$...%>" is a new expression type in ASP.NET. It can also be used to load resources with statements like this one:
<asp:Literal Runat="server"
  Text="<%$ Resources:MyResources, MyText %>"
/>
And it can be used to load strings from the <appSettings> configuration section, as shown here:
<asp:Literal Runat="server"
  Text="<%$ AppSettings:MyText %>"
/>

What is less widely known is that $ expressions are extensible. That is, you can add support for $ expressions of your own by writing custom expression builder classes. System.Web.Compilation.ExpressionBuilder in the Framework provides the basic plumbing needed and can be derived from in order to create custom expression builders.

Here is the source code for a simple page named Version.aspx:

<%@ Page Language="C#" CompileWith="Version.aspx.cs"
  ClassName="Version_aspx" %>

<html>
  <body>
    <h1>Powered by ASP.NET
      <asp:Literal Text='<%$ Version:MajorMinor %>' Runat="server" />
    </h1>
  </body>
</html>
The page's output shows the version of ASP.NET that's running. The version number is generated by this expression:
<%$ Version:MajorMinor %>

On its own, ASP.NET has no idea what to do with this expression. It works because the application's Code directory contains the source code for a custom expression builder named VersionExpressionBuilder (shown in Figure 8). VersionExpressionBuilder derives from System.Web.Compilation.ExpressionBuilder and overrides one virtual method, GetCodeExpression, which is called at run time by ASP.NET to evaluate the $ Version expression.

Figure 8 VersionExpressionBuilder.cs (Close Figure 8)
using System;
using System.Web.UI;
using System.Web.Compilation;
using System.CodeDom;

public class VersionExpressionBuilder : ExpressionBuilder
{
    public override CodeExpression GetCodeExpression(
        BoundPropertyEntry entry, object parsedData,
        ExpressionBuilderContext context)
    {
        string param = entry.Expression;
        if (String.Compare(param, "All", true) == 0)
            return new CodePrimitiveExpression
                (String.Format("{0}.{1}.{2}.{3}",
                Environment.Version.Major,
                Environment.Version.Minor,
                Environment.Version.Build,
                Environment.Version.Revision));
        else if (String.Compare(param, "MajorMinor", true) == 0)
            return new CodePrimitiveExpression
                (String.Format("{0}.{1}",
                 Environment.Version.Major,
                 Environment.Version.Minor));
        else
            throw new InvalidOperationException
                ("Use $ Version:All or $ Version:MajorMinor");
    }
}

The Expression property of the BoundPropertyEntry parameter passed to GetCodeExpression contains the text to the right of the colon in the expression: in this particular case, "MajorMinor." GetCodeExpression responds by a CodePrimitiveExpression encapsulating the string "2.0". If you write the $ Version expression this way instead

<%$ Version:All %>
then the string returned contains build and revision numbers as well as major and minor version numbers.

Custom expression builders must be registered and mapped to expression prefixes so that ASP.NET knows what class to instantiate when it encounters a $ expression. Registration is accomplished by adding an <expressionBuilders> section to Web.config's <compilation> section, as shown here:

<!-- From Web.config -->
<compilation>
  <expressionBuilders>
    <add expressionPrefix="Version" type="VersionExpressionBuilder"/>
  </expressionBuilders>
</compilation>

Now that you know about custom expression builders, you can probably envision other uses for them. Imagine, for example, $ XPath expressions that extract data from XML files, or $ Password expressions that retrieve passwords or other secrets from ACLed registry keys. The possibilities are endless.

Custom Web Events

One of the major new services featured in ASP.NET 2.0 is the one provided by the health monitoring subsystem. With a few simple statements in Web.config, an ASP.NET 2.0 application can be configured to log failed logins, unhandled exceptions, expired forms authentication tickets, and more.

Logging is accomplished by mapping the "Web events" fired by the health monitoring subsystem to Web event providers. Each provider corresponds to a specific logging medium. For example, the built-in EventLogProvider logs Web events in the Windows event log, while SqlWebEventProvider logs them in a SQL Server™ database. Other providers supplied with ASP.NET 2.0 permit Web events to be transmitted in e-mail messages, redirected to the WMI subsystem, and even forwarded to registered trace listeners.

Out of the box, the health monitoring subsystem presents a world of possibilities for monitoring the health and well-being of running ASP.NET applications and for leaving paper trails for use in failure diagnostics. But what's really great about health monitoring is that it is entirely extensible. You can define custom Web events and fire them at appropriate junctures in an application's lifetime. Imagine that your application fired a Web event every time database contention resulted in a concurrency error. You could map these events to a provider and check the log at the end of each day to detect and correct excessive concurrency errors. Or what if a financial app fired a Web event every time it performed a monetary transaction? You could keep a running log of all such transactions simply by adding a few statements to Web.config.

The page shown in Figure 9 fires a custom Web event each time you click the "Fire Custom Web Event" button. The custom Web event is simple: it notifies any providers that are connected to it that the button was clicked.

Figure 9 Firing Custom Web Events (Close Figure 9)

WebEvent.aspx
<%@ Page Language="C#" CompileWith="WebEvent.aspx.cs"
  ClassName="WebEvent_aspx" %>

<html>
  <body>
    <form runat="server">
      <asp:Button ID="Button1" Runat="server"
        Text="Fire Custom Web Event" Width="219px"
        Height="104px" OnClick="Button1_Click" />
    </form>
  </body>
</html>
WebEvent.aspx.cs
using System;
using System.Web;
using System.Web.Management;

public partial class WebEvent_aspx
{
    void Button1_Click (object sender, EventArgs e)
    {
        MyWebEvent mwe = new MyWebEvent ("Click!",
            null, 100001, DateTime.Now);
        WebBaseEvent.Raise (mwe);
    }
}

Figure 10 shows how the event appears in the Windows Event Viewer if the event, which is named simply "MyWebEvent," is mapped to the Windows event log.

Figure 10 Windows Event Viewer
Figure 10 Windows Event Viewer

How do custom Web events work? You begin by defining a custom Web event class by deriving from System.Web.Management.WebBaseEvent, as shown in Figure 11.

Figure 11 MyWebEvent.cs (Close Figure 11)
using System;
using System.Web.Management;

public class MyWebEvent : WebBaseEvent
{
    DateTime _time;

    public MyWebEvent (string message, object source, int eventCode,
        DateTime time) : base (message, source, eventCode)
    {
        _time = time;
    }

    public override void FormatCustomEventDetails(
        WebEventFormatter formatter)
    {
        formatter.AppendLine ("Button clicked at " +_time.ToString());
    }
}

Derived classes typically override FormatCustomEventDetails, which gives them the opportunity to append output of their own to the output generated by the base class. The base class's output contains key statistics about the event, such as its name and the time and date it was fired. MyWebEvent adds a line of its own—"Button clicked at [date and time]"—that appears at the end of the log entry. After you've defined a custom Web event in this manner, you fire it by instantiating it and passing it to the static WebBaseEvent.Raise method, as seen in Figure 11.

You must register custom Web events before firing them. The following code shows the registration entries in Web.config for MyWebEvent:

<!-- From Web.config -->
<healthMonitoring enabled="true">
  <eventMappings>
    <add name="My Web Events" type="MyWebEvent, __code" />
  </eventMappings>
  <rules>
    <add name="My Web Events" eventName="My Web Events"
      provider="EventLogProvider" />
  </rules>
</healthMonitoring>

The enabled="true" attribute in the <healthMonitoring> tag enables the health monitoring subsystem. The <eventMappings> section defines an event named "My Web Event" and maps it to the MyWebEvent class. Finally, the <rules> section maps MyWebEvent to EventLogProvider, directing instances of MyWebEvent to the Windows event log. If you wanted to log MyWebEvents in other storage media, you could do so simply by changing the provider specified in <rules>.

One nuance to be aware of regarding custom Web events is that in Beta 1, you need to add the source code file containing the custom Web event class to the application's Code directory and run the application once before registering the event and mapping it to a provider in Web.config. Otherwise, ASP.NET complains that the Web event class is undefined, apparently because ASP.NET parses Web.config before its autocompilation engine gets a chance to compile the files in the Code directory. I presume this will be fixed in Beta 2, but it's only a minor annoyance if you stage deployment so that autocompilation happens first.

ASP.NET 2.0 is loaded with new features designed to make building cutting-edge Web apps easier and less time consuming. But beyond the features you read about, a host of "lesser" features makes ASP.NET 2.0 more powerful and more extensible than its predecessors. Exploiting these hidden gems is one of the keys to writing great ASP.NET 2.0 code.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (2002, Microsoft Press). He's also a cofounder of Wintellect, a software consulting and education firm that specializes in Microsoft .NET.


Service Station: SOA: More Integration, Less Renovation

Service Station
SOA: More Integration, Less Renovation
Aaron Skonnard

Contents
Motivations of an Enterprise
More Integration, Less Renovation
Designing Systems to Last
Objects, Components, and Contracts
The Road to Service Orientation
What About Interoperability?
Service Orientation vs. Web Services
Thinking the Service-Oriented Way
Contract-First Development
Contracts as Building Blocks
Looking Ahead

Since the focus of this column is service-oriented architecture (SOA), I thought now might be a good time to step back and take a wide-angle look at the general concept and what it means to developers. As most readers are painfully aware, much has already been written on the subject of service orientation. In fact, the current frenzy around the topic makes many developers wonder if service orientation will be the next buzz-word casualty.

I don't think it will. The principles of service orientation are here to stay. I believe they'll help enterprises build more successful distributed systems that can adapt over time. So this month, I'll give you my take on service orientation and what it means to the enterprise as well as to the developer.

Motivations of an Enterprise

Service orientation is simple: it more effectively addresses the realities of today's enterprises than any other design paradigm. In order to understand why service orientation is such a good fit, let's first consider the most important needs of an enterprise. When looking at distributed systems from a business perspective, most enterprises share a few fundamental concerns:

  1. How well does the architecture meet specific business needs?
  2. How much is it going to cost to manage and maintain?
  3. Will it adapt to the growth of our businesses over time?
  4. How long until it is necessary to start over and rewrite?

Within an enterprise, there are a variety of stakeholders. You have top-level executives focused on company direction, vision, and strategy. You have CTO types, architects, and project managers tasked with aligning business goals with technology choices—these folks are usually the ones most directly focused on the questions I listed. You also have the engineers and developers tasked with implementing, testing, and deploying the system. Then there's the IT staff tasked with managing and maintaining the system after deployment. And finally, you have the users of the system.

All of these stakeholders have different, and often conflicting, goals and concerns. Applying the principles of service orientation can improve the overall experience for everyone involved.

More Integration, Less Renovation

The main reason service orientation will improve the enterprise experience is that it lets you leverage existing investments. Complete renovation—making significant changes to the technological makeup, architecture, and design of a distributed system—can be a nasty temptation. You think it'll be easier to just start over. It's common for developers to face compressed timelines with very little time, if any, allocated to the design process. Some may find themselves in a never-ending state of renovation, which usually has a negative impact on the business concerns.

In addition to general evolution, mergers and acquisitions complicate matters further by creating a melting pot of technologies that must all work together. Legacy hardware and software is obviously here to stay and will probably always complicate distributed system development.

Service-oriented design can alleviate these age-old problems, and designing for SOA from the beginning can seriously improve the future for all enterprise stakeholders. It will make systems more malleable to business needs, easier to develop, and more reasonable to maintain and manage so you can plan ahead rather than respond defensively.

Designing Systems to Last

If you design systems to last, it doesn't mean they're stuck in the past. Increasing the longevity of a distributed system requires a focus on the flexibility of its individual parts. If individual components are free to evolve and if new components can be introduced without requiring a major renovation, overall agility and manageability improves as does longevity. Focusing on integration early in the development process (perhaps even before it's needed) will give you a competitive edge.

Service orientation represents the next evolutionary step in distributed design methodologies, with the necessary adjustments to address the enterprise realities surrounding integration.

Objects, Components, and Contracts

Let's briefly retrace the steps leading up to the latest abstraction of service orientation. Distributed computing has consistently improved with more powerful and flexible abstractions, as each evolutionary step encouraged reuse and reduced coupling.

In the beginning software was monolithic. Then we moved to functional programming. Object orientation (OO), which quickly became the norm in the 1980s, with its focus on inheritance, encapsulation, and polymorphism, soon followed. Object orientation encouraged reuse, but its whitebox characteristics led to tight coupling and brittle inflexibility.

Components and interface-based development followed in the 1990s. This new approach provided a blackbox abstraction over objects. Components introduced the concept of shielding objects behind a boundary, only allowing interactions through a shared contract (the interface). Implementation details remained completely hidden from view. Only interface details and documentation were available to the consumer. Components were deployed independently as binary units of code, which made them somewhat autonomous. They were loaded dynamically by the component runtime, which provided additional services such as security, reliability, transactions, and so forth.

Component-oriented development was the first significant step towards reducing coupling between objects. It did this by focusing on contracts, not classes. This became even more important when developers began using components across machine boundaries in a distributed fashion.

When designing components, developers usually started by defining the contract. Interface definitions were written using the Interface Definition Language (IDL). The IDL was then used to generate class definitions and runtime behaviors. Focusing on the interface definition required developers to think about the data types shared across the component boundary. This turned out to be especially important when different languages were used to create and consume the component. Language interoperability often suffered when not enough attention was paid to the contract during design. Components were a significant move towards a more loosely coupled distributed system.

The Road to Service Orientation

Although component technologies were a significant advancement, they possessed a serious flaw: they made too many assumptions about the technologies used, assumptions about platform dependencies, homogeny, and proprietary communication protocols. They assumed that everyone using the component framework would be running the same operating system and component runtime. They also made serious assumptions about messaging models, assuming everyone would be satisfied with remote procedure calls (RPC). Although these assumptions are reasonable for some business scenarios, they negatively impact the agility and longevity of any evolving organization.

Service orientation takes component technologies to the next level by stripping them of these assumptions. You can think of service orientation as a black-box abstraction over the traditional component framework (in fact, it will often be used that way).

In this new paradigm, a "service" is an application you interact with by exchanging messages over specific communications protocols. A service adheres to these fundamental principles: it has explicit boundaries and is autonomous, it shares contract rather than implementation, and compatibility is based on policy.

In order to interact with a service, you simply send messages to its specific address. For this to work, both the client and the service must speak the same communication protocols and share an understanding of the required message format. This means there's an explicit boundary to the service—the only way you can interact with it is by sending it messages. Exactly how you interact with a service, and what type of message you must use, is defined by a shared schema and contract definitions.

Services also benefit from being autonomous, which basically means they can stand alone and take care of themselves (although the precise meaning is a matter of debate). Despite the emphasis on autonomy, however, services can also be composed of other services. For example, a workflow coordinates the work and interactions of multiple individual services—imagine the workflow itself exposed as a service.

Given the focus on loose coupling, services must be able to figure out if they're compatible with other services somehow. This is accomplished through a shared policy that both sides understand and can interpret at run time. For more information on the tenets of service orientation, see the "Service-Oriented Architecture Resources" sidebar.

Service orientation does not impose any platform dependencies, component technology, or programming language. The only shared assumptions are the message formats, description languages, and communication protocols. This is a huge step in the right direction but it begs the obvious question: what technologies should be used for these remaining shared assumptions?

What About Interoperability?

Before turning to assumed technologies, I'll discuss interoperability. Service orientation focuses on independence and loose-coupling, but there's no mention of interoperability. Although you could argue that the tenets of service orientation don't necessarily require interoperability, I'd argue that it's intrinsically tied to this paradigm and is nonnegotiable in the real world.

It was the lack of interoperability, mind you, that prevented component technologies from keeping up with the Internet and Web advancements. It was the lack of interoperability that motivated the need for something better than traditional component development, leading to a focus on services built with Web-based technologies (Web services). It is also the lack of interoperability that hurts enterprises more than anything else in their efforts to build flexible systems that last. Interoperability is absolutely necessary to achieve more integration and less renovation.

Indeed, interoperability is a fundamental factor in building long-lasting distributed systems. Coalitions like the WS-I have been formed with the specific goal of helping to ensure interoperability. Interoperability must not be forgotten, but rather used to enhance the flexible abstractions of service orientation that I described earlier in this column.

Service Orientation vs. Web Services

If you only focus on the official tenets of service orientation, it's quite possible, in fact, to build a service-oriented application using proprietary message formats and communication protocols. Doing so gives you a flexible system that can't talk to anything but itself. However, since interoperability is a fundamental requirement, the technology choices for these shared assumptions must facilitate interoperability, not reduce the ability to work together.Service-Oriented Architecture Resources

The following is a list of resources that will help you learn more about service-oriented architecture and design and put you on your way to better planning.

MSDN .NET Architecture Center—Service-Oriented Architecture

Data on the Outside vs. Data on the Inside

Metropolis: Envisioning the Service-Oriented Enterprise

Service Orientation and Its Role in Your Connected Systems Strategy

MSDN Web Services Interoperability and Integration

Service Station, MSDN Magazine, November 2004

This is precisely where Web services fit in. There is a common misconception that Web services and service orientation are synonymous. They are not. Service orientation is a paradigm while Web services represents a suite of implementation technologies. Web services is the best implementation of service orientation today because it's built on a foundation of standards that truly facilitate interoperability in the real world (see Figure 1).

Figure 1 Web Services Protocol Stack
Figure 1 Web Services Protocol Stack

Standards are vitally important when interoperability is at stake. And it's extremely difficult for the industry at large to agree on anything, not to mention an entire suite of data formats and protocols. Producing standards requires debate, negotiation, and, most importantly, compromise from all sides. After a standard solidifies, vendors must remain committed to it in their products. The real test of their commitment comes when a given standard doesn't meet their exact needs or business objectives.

Overall, it's extremely tricky to achieve widespread agreement and true commitment around a suite of technologies. So the value of standards, even when the technologies themselves don't seem that brilliant, cannot be overstated. The software industry has been able to achieve remarkable consensus around a suite of simple protocols and technologies, most of which helped build the distributed Web. This was not possible before, at least not to this degree. It is a remarkable achievement.

For example, we have achieved widespread interoperability around several communication and application protocols including TCP, HTTP, SMTP, and many others. Disparate platforms and applications can effectively communicate using any of these standard transports. There has been mass adoption of XML 1.0 as a standard syntax for representing information sent over these transports and a more recent adoption of additional XML-based dialects like SOAP, XML Schema, and Web Services Description Language (WSDL) that lay the foundation for an interoperable Web services architecture. These technologies work well for the kinds of communications that occur at the service boundary.

SOAP defines a standard mechanism for building application protocols in a transport-neutral manner. It defines a simple framework for sending and receiving XML messages between systems using a variety of message exchange patterns. SOAP lays a solid foundation on top of which service-oriented systems can be built.

XML Schema provides a standard language for defining application-specific types or schemas that can be used in a variety of different programming environments. WSDL is another language for defining service contracts that use XML Schema types for input and output. These two languages together are appropriate for sharing the schemas and contracts of a given service.

Figure 2 Determining Compatability
Figure 2 Determining Compatability

In addition to these foundational technologies, there are other specifications under development that provide layered services such as security, reliability, and transaction support. These WS-* specifications play an important role in building sophisticated Web services infrastructures that will simplify the building of robust, real-world distributed systems. When services begin to use these different layered services, questions of compatibility quickly arise (that is, does service A support the same security required by service B?). Determining compatibility requires policies that describe service requirements and behaviors (see Figure 2). The Web Services Policy language (WS-Policy) is another specification with significant adoption that makes it possible to accomplish this.

The widespread agreement around Web services technologies makes it the only viable choice for implementing service-oriented systems today. Applying Web services to this flexible architectural paradigm is expected to be a powerful combination.

Thinking the Service-Oriented Way

Designing distributed systems around the principles of service orientation using Web services technologies requires a different way of thinking about the world. It requires you to think more about connecting the dots within a distributed system—again more integration, less renovation.

Before you write a single line of code for a service-oriented system, you must identify the logical services and the necessary interactions that will make up the system. When modeling these services, you must make sure to consider real-world business and technology dependencies and the overall goal of autonomy.

As with objects, services should be modeled in a way that facilitates and encourages reuse across applications and environments. Thought must go into where the code will be hosted and what protocols will be used to talk to it. Exactly how you draw lines around your code to delineate what's contained in a service is arguably the least understood aspect of this new paradigm, but you can surely expect advances in this area.

After the scope of a service has been defined, a service-oriented developer would focus next on defining the contracts that will be needed to interact with the service. In practical terms, this means defining the XML Schema and WSDL definitions used by the service. Focusing on how to move data in and out of the service is an excellent exercise in flushing out what you need it to do before you get caught up in details. What data is needed inside the service versus outside the service? What message exchange patterns are required? How you move data across a service boundary is the primary focus of this stage, and it's completely disconnected from how you treat data within a service (see Pat Helland's recent MSDN® article entitled "Data on the Outside vs. Data on the Inside").

After the service contracts have been designed, focus should turn to the service policies. This requires you to think about the various security and reliability requirements of the service and how high you're going to set the bar for others. Policy decisions not only affect how hard it is for you to implement, but more importantly, how hard it is for others to consume. The decisions made with regards to policy can definitely impact the reach of a service.

When you're designing the service-oriented way, you do all of these things during the design phase before you begin implementation. Once you have the contracts and policies nailed, you can then take advantage of tools to generate the necessary code to implement the service framework, helping return some of the time invested in design.

Contract-First Development

Probably the most significant change for developers embracing this new model has to do with the impact it has on their development process. Most developers of distributed apps are used to focusing on objects and writing a lot of code from the starting gate. This habit must change in a service-oriented world.

There are two common techniques that developers use to define service contracts today. One is to start by defining a WSDL contract before implementing the code—I refer to this as "contract-first" development. The other is to start by writing the service implementation in code, such as C# or Visual Basic® .NET, and then rely on the underlying infrastructure (for example, ASP.NET) to produce the contract for you—I refer to this as code-first development. Both approaches have their pros and cons, but code-first is more common today.

Contract-first forces you to work with the shared language that everyone agrees on (WSDL, for example). Today working directly with WSDL is extremely difficult and uncomfortable because we lack proper tool support and the language is difficult. But there is a silver lining: working directly with WSDL prevents developers from using language-specific types or OO techniques that don't map nicely to a service contract. The code-first approach is more pleasant, but it introduces an extra level of indirection—a translation layer that can be deceiving and troubling for interoperability.

Although today's frameworks don't accomplish this, it will be interesting to see if future Web services platforms can make it more feasible to define service contracts in a developer-friendly language (like C#) without hampering interoperability. I believe it's possible with proper constraints and restrictions built into the framework. However, even if the next version of Windows®, code-named "Longhorn," makes it possible to define contracts in an easier manner, say through an attributed interface definition, the same principles apply to that approach. It simply eliminates the need to work directly with WSDL, which is something that any developer would certainly welcome.

If developers truly want to embrace service orientation and Web services, they must also be willing to embrace contract-first development. Otherwise, they've missed the entire point. Contracts are the building blocks of a composable, service-oriented system. Hence, everything hinges on the contract definitions.

Contracts as Building Blocks

Objects used to be the building blocks of distributed systems, but not anymore. Contracts are the new building block in a service-oriented system. They facilitate composition, integration, and the overall adaptability of a system. In the end, the contracts tie the system together, bridging a wide range of platforms, operating systems, component technologies, and programming languages. Therefore, it's extremely important to invest time and energy in contract design in order to produce something that facilitates interoperability and integration, the fundamental goals of this model.

One thing to keep in mind is that you're not always in control of the contract definition. In some situations you're the one providing the service so you control the contracts and others are forced to use them. In other situations, you may find yourself on the receiving end, forced to use a contract that someone else has defined. And you may find yourself collaborating with other individuals, departments, or entire organizations to develop standard contracts that will be shared in a given vertical segment.

When you're forced to use a contract defined by someone else, all you're going to care about is that it's easy to use in your development environment. If you run into problems, you may be inclined to look elsewhere for the same functionality. Most enterprises would do well by adopting the general principle of making it as easy as possible for others to integrate with them. In a service-oriented world, the true value of the system is found in the overall quality and ease-of-use of its contracts.

Developing an easy-to-consume contract is definitely harder and more time consuming than producing one indirectly through a toolkit, but you'll be making your partners happier and improving your business objectives at the same time. And when you find yourself on the receiving end of a contract, you'll wish those who developed it had followed the same principles.

Looking Ahead

Overall service orientation offers tremendous opportunities if you're willing to change your mindset and embrace contract-first development. The key paradigm shift for developers is to focus on contracts, the new building blocks, rather than on objects. Doing so will help produce long-lasting distributed systems that meet the enterprise goals I presented at the beginning of this column.

In a future column, I'll walk you through a practical example of building a service-oriented system using contract-first development techniques, and I'll show you some of the tools available today for making the technique more approachable.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, an education and content creation company, where he focuses on XML and Web services technologies. He's a contributing editor to MSDN Magazine and the author of several books, including Essential XML Quick Reference (Addison-Wesley, 2001). Reach him at www.pluralsight.com/aaron.


.NET Matters: File Copy Progress, Custom Thread Pools

.NET Matters
File Copy Progress, Custom Thread Pools
Stephen Toub

Code download available at: NETMatters0502.exe (214KB)

Q I'm copying a lot of large files in my application, and I'd like to give the user the option to cancel these actions. Can I use the same dialog that the shell provides when dragging and dropping files from one place to another? Alternatively, is there any way my application can receive status notifications during a file copy so that I can create my own progress dialog with an option to cancel?

A If you're using Visual Studio® 2005 Beta 1, this is a piece of cake. While not exposed as part of the base class libraries, the Visual Basic team has wrapped the shell's file copy functionality (the SHFileOperation function exposed from shell32.dll) in the My namespace (for more information on My, see Duncan Mackenzie's article in the May 2004 issue of MSDN®Magazine). When using Visual Basic®, the CopyFile and CopyDirectory methods on My.Computer.FileSystem provide a showUI Boolean parameter. When set to true, the copy file dialog will be displayed to show the file transfer's progress. This dialog also allows the user to cancel the operation. If you're using Visual C#® 2005, you can reference the Microsoft.VisualBasic.dll library from your app. Instead of this code in Visual Basic

My.Computer.FileSystem.CopyFile(...)
you can use the functionally equivalent C#:
new Microsoft.VisualBasic.MyServices.MyServerComputer()
    .FileSystem.CopyFile(...);
Of course, this is only valid with the .NET Framework 2.0. If you're using the .NET Framework 1.x, you'll have to use P/Invoke to access the SHFileOperation manually, a sample for which is available at C# does Shell, Part 2.

That addresses your first question about whether it's possible to use the shell's copy operation. Your second question about receiving copy status notifications is answered with one function, CopyFileEx, exposed from Kernel32.dll. Under the covers, the System.IO.File class's copy operations use the related Win32® CopyFile function, also exposed from Kernel32.dll, but unfortunately there is no wrapper provided for CopyFileEx.

CopyFileEx can call a specified callback function each time a portion of the copy operation is completed, exactly the functionality you need. So, I've created the wrapper class shown in Figure 1 to expose this functionality to your application through the FileRoutines.CopyFile method. The primary overload of this method accepts five parameters: a FileInfo for the source file to be copied, a FileInfo for the destination location, a CopyFileOptions enumeration that further configures the copy operation, a CopyFileCallback delegate that is to be called to inform your application of the copy's progress, and an object state parameter that will be passed to this callback delegate.

Figure 1 FileRoutines.CopyFile Using the Win32 CopyFileEx (Close Figure 1)
public sealed class FileRoutines
{
    public static void CopyFile(FileInfo source, FileInfo destination)
    {
        CopyFile(source, destination, CopyFileOptions.None);
    }

    public static void CopyFile(FileInfo source, FileInfo destination,
        CopyFileOptions options)
    {
        CopyFile(source, destination, options, null);
    }

    public static void CopyFile(FileInfo source, FileInfo destination,
        CopyFileOptions options, CopyFileCallback callback)
    {
        CopyFile(source, destination, options, callback, null);
    }

    public static void CopyFile(FileInfo source, FileInfo destination,
        CopyFileOptions options, CopyFileCallback callback, object state)
    {
        if (source == null) throw new ArgumentNullException("source");
        if (destination == null)
            throw new ArgumentNullException("destination");
        if ((options & ~CopyFileOptions.All) != 0)
            throw new ArgumentOutOfRangeException("options");

        new FileIOPermission(
            FileIOPermissionAccess.Read, source.FullName).Demand();
        new FileIOPermission(
            FileIOPermissionAccess.Write, destination.FullName).Demand();

        CopyProgressRoutine cpr = callback == null ?
            null : new CopyProgressRoutine(new CopyProgressData(
                source, destination, callback, state).CallbackHandler);

        bool cancel = false;
        if (!CopyFileEx(source.FullName, destination.FullName, cpr,
            IntPtr.Zero, ref cancel, (int)options))
        {
            throw new IOException(new Win32Exception().Message);
        }
    }

    private class CopyProgressData
    {
        private FileInfo _source = null;
        private FileInfo _destination = null;
        private CopyFileCallback _callback = null;
        private object _state = null;

        public CopyProgressData(FileInfo source, FileInfo destination,
            CopyFileCallback callback, object state)
        {
            _source = source;
            _destination = destination;
            _callback = callback;
            _state = state;
        }

        public int CallbackHandler(
            long totalFileSize, long totalBytesTransferred,
            long streamSize, long streamBytesTransferred,
            int streamNumber, int callbackReason,
            IntPtr sourceFile, IntPtr destinationFile, IntPtr data)
        {
            return (int)_callback(_source, _destination, _state,
                totalFileSize, totalBytesTransferred);
        }
    }

    private delegate int CopyProgressRoutine(
        long totalFileSize, long TotalBytesTransferred, long streamSize,
        long streamBytesTransferred, int streamNumber, int callbackReason,
        IntPtr sourceFile, IntPtr destinationFile, IntPtr data);

    [SuppressUnmanagedCodeSecurity]
    [DllImport("Kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
    private static extern bool CopyFileEx(
        string lpExistingFileName, string lpNewFileName,
        CopyProgressRoutine lpProgressRoutine,
        IntPtr lpData, ref bool pbCancel, int dwCopyFlags);
}

public delegate CopyFileCallbackAction CopyFileCallback(
    FileInfo source, FileInfo destination, object state,
    long totalFileSize, long totalBytesTransferred);

public enum CopyFileCallbackAction
{
    Continue = 0,
    Cancel = 1,
    Stop = 2,
    Quiet = 3
}

[Flags]
public enum CopyFileOptions
{
    None = 0x0,
    FailIfDestinationExists = 0x1,
    Restartable = 0x2,
    AllowDecryptedDestination = 0x8,
    All = FailIfDestinationExists | Restartable | AllowDecryptedDestination
}

To start, the parameters are validated, ensuring that the source and destination FileInfo parameters are not null and that the specified CopyFileOptions enumeration value is legitimate. There are a few ways to check whether an enumeration value is valid based on that enumeration's definition. The simplest way involves the Enum.IsDefined method; it is robust and easy to use, but also slow when compared to other mechanisms, which is important to take into consideration if the API using this method will be used frequently. When I have a small Flags-attributed enumeration, especially one that has an all-value (a value in the enumeration that is the Boolean Or of all of the other values), the simplest way to check whether a value falls within that enumeration is to negate the all-value and take the Boolean And of it with the value being checked for inclusion. By negating the all-value, I'm creating a number with one-bits in places where one-bits are invalid. Thus, if taking the Boolean And of that with the value in question yields a non-zero value, the number being checked has bits in invalid positions, and is thus an invalid value (you should note, however, that this solution might not work as intended if the various values in the enumeration have more than one-bit set).

You'll note that I don't call File.Exists to ensure that the specified source file does in fact exist; this would be an exercise in futility as there's nothing to prevent the file from being deleted between the time I call File.Exists and the time the actual copy operation commences. Thus, I rely on Win32 to provide the appropriate error response if the specified source file doesn't exist.

After validating the parameters, I ensure that the caller has the necessary permissions to read the source file and to write the destination file by demanding the relevant FileIOPermission. I've also marked the declaration of CopyFileEx with the SuppressUnmanagedCodeSecurityAttribute so that calls to it do not cause a stack walk searching for the UnmanagedCode security permission.

With validation and security checks out of the way, I can get to the meat of the operation which is to invoke the CopyFileEx function. Besides the source and destination path parameters, the most important parameter for your purposes is lpProgressRoutine. In the Win32 API, this parameter is a function pointer to a function that will be called when the CopyFileEx routine has a progress update to report. In the managed world, I can pass a delegate with the corresponding signature such that a managed method will be called as the callback from CopyFileEx. In order to get additional state passed to the callback (such as the original FileInfo objects), the method I wrap with the CopyProgressRoutine delegate is an instance member of a state class which I populate with all of the relevant information about the copy operation. Now, when CopyFileEx calls back to the lpProgressRoutine, I have all of the user-supplied state in addition to that provided to me by CopyFileEx, such as the total number of bytes being copied and the number of bytes copied thus far.

The wrapper could end here, but I don't want to provide too much unnecessary information to the user of the wrapper. Thus, instead of exposing to the user the CopyProgressRoutine callback delegate, I've exposed the simpler CopyFileCallback. The user-supplied CopyFileCallback passed to the FileRoutines.CopyFile method is invoked when the CopyProgressData.CallbackHandler method (the actual callback for CopyFileEx) is invoked, receiving information I feel is most important to the user (the source and destination FileInfo, the original user-supplied state, the total number of bytes in the file to copy, and the total number of bytes successfully copied thus far). In addition, the CopyFileCallback delegate returns a CopyFileCallbackAction value which is, in turn, cast to an integer and returned directly to CopyFileEx. This value controls how CopyFileEx proceeds, whether it should continue the copy, cancel the copy, stop the copy, or continue the copy but stop notification callbacks for updates.

The difference between cancel and stop is very important. CopyFileCallbackAction.Cancel and CopyFileCallbackAction.Stop both prevent the copy operation from continuing; however, whereas Cancel deletes the portion of the target file already copied, Stop leaves it. This is helpful when using the CopyFileOptions.Restartable option. Restartable causes the progress of the copy operation to be tracked in the target file in case the copy is stopped or fails. The operation can then be restarted at a later time by specifying source and destination FileInfo objects that represent the same paths. More information on the CopyFileEx Win32 function is available in the MSDN library.

Q I have a situation where I'd like to have multiple thread pools, each of which has a small number of threads dedicated to processing a certain task. Unfortunately, it looks like the ThreadPool in the System.Threading namespace is only usable in a static context, meaning that there's only one of them and I can't create more. Any suggestions? How can I create multiple thread pools?

A Before embarking down this path, make sure you really need this for your solution. The .NET Framework team spent a lot of time optimizing the ThreadPool for many different scenarios, and it would be a shame for you to reinvent the wheel. And make sure you have the performance scenario you think you have. Assuming that a bottleneck is being caused by a particular piece of code can be a frustrating situation when you spend hours "fixing" it, only to realize that the area in question wasn't the problem.

If, in fact, you determine by profiling that you really will benefit from multiple, small thread pools, you have a couple of options. The first would be to use the .NET ThreadPool, but throttle requests to it in order to simulate multiple smaller pools (I implemented a simple ThreadPool throttling mechanism in my December 2004 column). This would allow you to limit the number of threads from the pool that could be allotted to each task. A second option is, of course, to create your own custom pool.

Thread pool implementations run the gamut from very simple to very complex and feature rich. The common language runtime (CLR) thread pool falls into the latter category, providing functionality such as delayed thread creation, thread teardown after a period of nonuse, completion callbacks, and special support for I/O completion ports. There's little reason to attempt to duplicate its complexity in your own custom thread pool, especially given that the CLR team has spent a lot of time and resources getting its sophisticated implementation right—not an easy task.

So, here I'll show you how to create a very simple thread pool that creates the requested pool size on startup and keeps them around until you're done with the pool. Each thread waits for work items to be supplied, and when work is available, it executes the work. When a thread is finished executing a work item, it waits for more work. A side benefit of this approach is that it allows you to create a "pool" of only one thread, which means you can easily implement serialized processing: queue up a bunch of delegates, and only one will be executed at a time.

My implementation is shown in Figure 2. An instance of the CustomThreadPool class maintains three private member variables: a semaphore used to block threads in the pool until work is available, a queue of all work waiting to be processed, and a list of the threads in the pool (the list is only necessary for pool management operations, such as shutting it down). For a CustomThreadPool to be instantiated, the user must specify the number of threads she wants to exist in the pool. The constructor then spins up that number of new threads, each of which starts running the private Run method and waits for work to be queued.

Figure 2 Custom Thread Pool (Close Figure 2)
public sealed class CustomThreadPool : IDisposable
{
    private Semaphore _workWaiting;
    private Queue<WaitQueueItem> _queue;
    private List<Thread> _threads;

    public CustomThreadPool(int numThreads)
    {
        if (numThreads <= 0)
            throw new ArgumentOutOfRangeException("numThreads");

        _threads = new List<Thread>(numThreads);
        _queue = new Queue<WaitQueueItem>();
        _workWaiting = new Semaphore(0, int.MaxValue);

        for (int i = 0; i < numThreads; i++)
        {
            Thread t = new Thread(Run);
            t.IsBackground = true;
            _threads.Add(t);
            t.Start;
        }
    }

    public void Dispose()
    {
        if (_threads != null)
        {
            _threads.ForEach(delegate(Thread t) { t.Interrupt(); });
            _threads = null;
        }
    }

    public void QueueUserWorkItem(WaitCallback callback, object state)
    {
        if (_threads == null)
            throw new ObjectDisposedException(GetType().Name);
        if (callback == null) throw new ArgumentNullException("callback");

        WaitQueueItem item = new WaitQueueItem();
        item.Callback = callback;
        item.State = state;
        item.Context = ExecutionContext.Capture();

        lock(_queue) _queue.Enqueue(item);
        _workWaiting.Release();
    }

    private void Run()
    {
        try
        {
            while (true)
            {
                _workWaiting.WaitOne();
                WaitQueueItem item;
                lock(_queue) item = _queue.Dequeue();
                ExecutionContext.Run(item.Context,
                    new ContextCallback(item.Callback), item.State);
            }
        }
        catch(ThreadInterruptedException){}
    }

    private class WaitQueueItem
    {
        public WaitCallback Callback;
        public object State;
        public ExecutionContext Context;
    }
}

The Run method is very straightforward. The worker thread blocks waiting for work items to be added to the instance using QueueUserWorkItem. When a work item shows up, one of the waiting threads wakes up, retrieves the work item from the work queue (under a lock on the queue so that access to the queue is synchronized), and executes it. QueueUserWorkItem is just as straightforward. The user-supplied WaitCallback and object state, along with the current ExecutionContext, are stored in a data structure that is appended to the work queue (again, synchronized around the queue). The semaphore is then signaled to indicate that another work item is available.

I discussed ExecutionContext in my November 2004 column, but suffice it to say that ExecutionContext provides a single container for all information relevant to a logical thread of execution. This includes security context, call context, synchronization context, localization context, and transaction context. The class provides the ability to transfer this context from one thread to another, and as such I use it to transfer the execution context from the thread that queues a work item to the worker thread that actually invokes the work item. If the thread that queues the work item is the same thread that created the pool, this adds little functionality, as the threads created in the pool's constructor will have already had the original execution context flowed to them. If, however, the thread queuing the work item has a different context than the thread that created the pool at the time it created it, this simple addition can greatly increase the security of the pool. Given it only took three additional lines of code to support the transfer of the ExecutionContext from QueueUserWorkItem to run, it's well worth the effort!

As I'm a big fan of C# and the improvements that have been made to it for version 2.0, I'd like to highlight some here (I'm using Beta 1). First and foremost, the work queue I've implemented uses a generic Queue of WaitQueueItem objects, and the list of Thread instances is a generic List of Thread objects. This improves both the efficiency and readability of the code (in addition to improving IntelliSense® in Visual Studio 2005 while I wrote the code). Of course, generics are applicable to more than just C#, as languages like MSIL, Visual Basic, and C++ also support them (in fact, C++ supports both generics and templates, and it supports them in combination). I've also taken advantage of delegate inference, as you can see in the CustomThreadPool's constructor where I construct the Thread instances, and of anonymous delegates, which you can see in the call to _threads.ForEach (which executes the specified delegate for each item in the List).

With this custom thread pool in place, there are lots of enhancements you can add to it to better suit your particular requirements. For example, currently all threads are created during the construction of the pool, but the pool could be modified to create them on demand up to the maximum number allowed, and to tear down extra threads after some period of inactivity. Work item priorities and work item wait functionality could be added (similar to how I demonstrated in my October 2004 and November 2004 columns, but more easily given that you don't have to write it as an extension to the pool). You could add the ability to cancel a queued work item such that it wouldn't be executed if it hadn't already, you could add properties to expose how many threads are currently in use and how many are available for work, and you could add additional events to signal when work has been started and when work has been completed. These are just a few ways a pool can be extended, and many possibilities exist for you to tailor the class for your exact needs.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


Pure C++: Hello, C++/CLI

[ Editor's Update - 1/20/2006: This article refers to a beta version of Visual Studio 2005. An updated version of the article, reflecting features found in the final release of Visual Studio 2005, can be found at Pure C++: Hello, C++/CLI.]
Pure C++
Hello, C++/CLI
Stanley B. Lippman

Contents
What is C++/CLI?
Learning C++/CLI
Mapping C++/CLI to the CTS?
CLI Level of Detail
What Are the Issues?
Additional Functionality
So, What Did You Say About C++/CLI?

Welcome to my first column for MSDN®Magazine. I had actually written a completely different first column—one on the new generic programming support in Visual Studio® 2005. Reviewing it, though, I realized that it left far too many unanswered questions. So I put that column aside and wrote this one to provide an overview of my team's work on C++/CLI for Visual C++® 2005. I'll follow this with a fistful of columns on .NET generic programming. Don't forget that this column is based on a prerelease version and may be subject to change.

What is C++/CLI?

C++/CLI represents a tuple. C++ refers, of course, to the C++ programming language invented by Bjarne Stroustrup at Bell Laboratories. It supports a static object model that is optimized for the speed and size of its executables. However, it doesn't support runtime modification of the program other than heap allocation. It allows unlimited access to the underlying machine, but very little access to the types active in the running program and no real access to the associated infrastructure of that program. Herb Sutter, a colleague of mine here at Microsoft and the chief architect of C++/CLI, refers to C++ as a concrete language.

CLI refers to the Common Language Infrastructure, a multitiered architecture supporting a dynamic component programming model. In many ways, this represents a complete reversal of the C++ object model. A runtime software layer, the virtual execution system, runs between the program and the underlying operating system. Access to the underlying machine is fairly constrained. Access to the types active in the executing program and the associated program infrastructure—both as discovery and construction—is supported. The slash (/) represents a binding between C++ and the CLI. The details surrounding this binding make up the general topic of this column.

So, a first approximation of an answer to what is C++/CLI is that it is a binding of the static C++ object model to the dynamic component object model of the CLI. In short, it is how you do .NET programming using C++ rather than C# or Visual Basic®. Like C# and the CLI itself, C++/CLI is undergoing standardization under the European Computer Manufacturers Association (ECMA) and eventually under ISO.

The common language runtime (CLR) is the Microsoft version of the CLI that is specific to the Windows® operating system. Similarly, Visual C++ 2005 is the implementation of C++/CLI.

As a second approximation of an answer, I would say that C++/CLI integrates the .NET programming model within C++ in the same way as, back at Bell Laboratories, we integrated generic programming using templates within the then existing C++. In both of these cases your investment in an existing C++ codebase and in your existing C++ expertise are preserved. This was an essential baseline requirement of the design of C++/CLI.

Learning C++/CLI

There are three aspects in the design of a CLI language that hold true across all languages: a mapping of language-level syntax to the underlying Common Type System (CTS), the choice of a level of detail to expose the underlying CLI infrastructure to the direct manipulation of the programmer, and the choice of additional functionality to provide, above and beyond that supported directly by the CLI.

The first item is largely the same across all CLI languages. The second and third items are where one CLI language distinguishes itself from another. Depending on the kinds of problems you need to solve, you'll choose one or another language, or possibly combine multiple CLI languages. Learning C++/CLI involves understanding each of these aspects of its design.

Mapping C++/CLI to the CTS?

It is important when programming C++/CLI to learn the underlying CTS, which includes these three general class types:

  • The polymorphic reference type, which is what you use for all class inheritance.
  • The non-polymorphic value type, which is used for implementing concrete types requiring runtime efficiency, such as the numeric types.
  • The abstract interface type, which is used for defining a set of operations common to a set of either reference or value types that implement the interface.

This design aspect, the mapping of the CTS to a set of built-in language types, is common across all CLI languages although, of course, the syntax varies in each CLI language. So, for example, in C#, you would write

abstract class Shape { ... } // C#
to define an abstract Shape base class from which specific geometric objects are to be derived, while in C++/CLI you write
ref class Shape abstract { ... }; // C++/CLI
to indicate the exact same underlying CLI reference type. The two declarations are represented exactly the same way in the underlying IL. Similarly, in C#, you write
struct Point2D { ... } // C#
to define a concrete Point2D class, while in C++/CLI you write:
value class Point2D { ... }; // C++/CLI

The family of class types supported with C++/CLI represents an integration of the CTS with the native facilities, and that determines your choice of syntax. For example:

class native {};
value class V {};
ref class R {};
interface class I {};

The CTS also supports an enumeration class type that behaves somewhat differently from the native enumeration, and support is provided for both of those as well:

enum native { fail, pass };
enum class CLIEnum : char { fail, pass};
Similarly, the CTS supports its own array type that again behaves differently from the native array. And again Microsoft provides support for both:
int native[] = { 1,1,2,3,5,8 };
array<int>^ managed = { 1,1,2,3,5,8 };

It is not accurate to think of any one CLI language as closer to or more nearly a mapping to the underlying CTS than is another. Rather, each CLI language represents a view into the underlying CTS object model. You'll see this more clearly in the next section.

CLI Level of Detail

The second design aspect that must be considered when designing a CLI language is the level of detail of the underlying CLI implementation model to incorporate into the language. What kind of problems will the language be tasked to solve? Does the language have the tools necessary to do this? Also, what sort of programmers is the language likely to attract?

Take, for example, the issue of value types occurring on the managed heap. Value types can find themselves on the managed heap in a number of circumstances:

  • Through implicit boxing, when an instance of a value type is assigned to an Object or when a virtual method is invoked through a value type that is not overridden.
  • When that value type is serving as a member of a reference class type.
  • When that value type is being stored as the element type of a CLI array.
Whether the programmer should be allowed to manipulate the address of a value type of this sort is a CLI language design consideration that must be addressed.

What Are the Issues?

Any object located on the managed heap is subject to relocation during the compaction phase of a sweep of the garbage collector. Any pointers to that object must be tracked and updated by the runtime; the programmer cannot manually track it herself. Therefore, if you were allowed to take the address of a value type that might be on the managed heap, there would need to be a tracking form of pointer in addition to the existing native pointer.

What are the trade-offs to consider? On the one hand, there's simplicity and safety. Directly introducing support in the language for either one or a family of tracking pointers makes it a more complicated language. By not supporting this, the available pool of programmers is expanded because less sophistication is required. In addition, allowing the programmer access to these ephemeral value types increases the possibility of programmer error—she may purposely or inadvertently do dangerous things to the memory. By not supporting tracking pointers, a potentially safer runtime environment is created.

On the other hand, efficiency and flexibility must be considered. Each time you assign the same Object with a value type, a new boxing of the value occurs. Allowing access to the boxed value type allows in-memory update, which may provide significant performance improvements. Without a form of tracking pointer, you cannot iterate over a CLI array using pointer arithmetic. This means that the CLI array cannot participate in the Standard Template Library (STL) iterator pattern and work with the generic algorithms. Allowing access to the boxed value type allows significant design flexibility.

Microsoft chose to provide a collection of addressing modes that handle value types on the managed heap in C++/CLI:

int ival = 1024;
int^ boxedi = ival;

array<int>^ ia = gcnew array<int>{1,1,2,3,5,8};
interior_ptr<int> begin = &ia[0];

value struct smallInt { int m_ival; ... } si;
pin_ptr<int> ppi = &si.m_ival;

The typical C++/CLI developer is a sophisticated system programmer tasked with providing infrastructure and organizationally critical applications that serve as the foundation over which a business builds its future. She must address both scalability and performance concerns and must therefore have a system-level view into the underlying CLI. The level of detail of a CLI language reflects the face of its programmer.

Complexity is not in itself a negative quality. Human beings are more complicated than single-cell bacteria, and that is certainly not a bad thing. However, when the expression of a simple concept is made complicated, that is usually considered to be a bad thing. In C++/CLI, the CLI team has tried to provide an elegant way to express complex subject matter.

Additional Functionality

A third design aspect is a language-specific layer of functionality above and beyond what is directly supported by the CLI. This may require a mapping between the language-level support and the underlying implementation model of the CLI. In some cases, this just isn't possible because the language cannot intercede with the behavior of the CLI. One example of this is the virtual function resolution in the constructor and destructor of a base class. To reflect ISO-C++ semantics in this case would require a resetting of the virtual table within each base class constructor and destructor. This is not possible because virtual table handling is managed by the runtime and not by the individual language.

So this design aspect is a compromise between what would be preferable to do, and what is feasible. The three primary areas of additional functionality provided by C++/CLI are the following:

  • A form of Resource Acquisition is Initialization (RAII) for reference types, in particular, to provide an automated facility for what is referred to as deterministic finalization of garbage- collected types that hold scarce resources.
  • A form of deep-copy semantics associated with the C++ copy constructor and copy assignment operator; however, this could not be extended to value types.
  • Direct support of C++ templates for CTS types in addition to the CLI generic mechanism—originally, this had been the topic of my first column. In addition, a verifiable version of the STL for CLI types is provided.

Let's look at a brief example: the issue of deterministic finalization. Before the memory associated with an object is reclaimed by the garbage collector, an associated Finalize method, if present, is invoked. You can think of this method as a kind of super-destructor since it is not tied to the program lifetime of the object. This is called finalization. The timing of just when or even whether a Finalize method is invoked is undefined. This is what is meant by nondeterministic finalization of the garbage collector.

Nondeterministic finalization works well with dynamic memory management. When available memory gets sufficiently scarce, the garbage collector kicks in and solves the problem. Nondeterministic finalization does not work well, however, when an object maintains a critical resource such as a database connection, a lock of some sort, or perhaps native heap memory. In this case, it would be great to release the resource as soon as it is no longer needed. The solution that is currently supported by the CLI is for a class to free the resources in its implementation of the Dispose method of the IDisposable interface. The problem here is that Dispose requires an explicit invocation, and therefore is not likely to be invoked.

A fundamental design pattern in C++ is the aforementioned Resource Acquisition is Initialization, which means that a class acquires resources within its constructor. Conversely, a class frees its resources within its destructor. This is managed automatically within the lifetime of the class object.

Here's what reference types should do in terms of the freeing of scarce resources:

  • Use the destructor to encapsulate the necessary code for the freeing of any resources associated with the class.
  • Have the destructor invoked automatically, tied with the lifetime of the class object.

The CLI has no notion of the class destructor for a reference type. So the destructor has to be mapped to something else in the underlying implementation. Internally, then, the compiler performs the following transformations:

  • The class has its base class list extended to inherit from the IDisposable interface.
  • The destructor is transformed into the Dispose method of IDisposable.

That represents half of the goal. A way to automate the invocation of the destructor is still needed. A special stack-based notation for a reference type is supported; that is, one in which its lifetime is associated within the scope of its declaration. Internally, the compiler transforms the notation to allocate the reference object on the managed heap. With the termination of the scope, the compiler inserts an invocation of the Dispose method—the user-defined destructor. Reclamation of the actual memory associated with the object remains under the control of the garbage collector. Figure 1 shows an example.

Figure 1 Invoking the Destructor (Close Figure 1)
ref class Wrapper {
    Native *pn;
public:
    // resource acquisition is initialization
    Wrapper( int val ) { pn = new Native( val ); }

    // this will do our disposition of the native memory
    ~Wrapper(){ delete pn; }

    void mfunc();
protected:

    // an explicit Finalize() method—as a failsafe
    !Wrapper() { delete pn; }
};

void f1()
{
   // normal treatment of a reference type
   Wrapper^ w1 = gcnew Wrapper( 1024 );

   // mapping a reference type to a lifetime
   Wrapper w2( 2048 ); // no ^ token !

   // just illustrating a semantic difference
   w1->mfunc();
   w2.mfunc();

   // w2 is disposed of here
}

//
// ... later, w1 is finalized at some point, maybe

C++/CLI is not just an extension of C++ into the managed world. Rather, it represents a fully integrated programming paradigm similar in extent to the earlier integration of the multiple inheritance and generic programming paradigms into the language. I think the team has done an outstanding job.

So, What Did You Say About C++/CLI?

C++/CLI represents an integration of the Kansas and Oz of native and managed programming. In this iteration, that has been done through a kind of separate but equal community of source-level and binary elements, including Mixed mode (source-level mix of native and CTS types, plus a binary mix of native and CIL object files), Pure mode (source-level mix of native and CTS types, all compiled to CIL object files), Native classes (can hold CTS types through a special wrapper class only), and CTS classes (can hold native types only as pointers).

Of course, the C++/CLI programmer can also choose to program in the CLI types alone, and in this way provide verifiable code that can be hosted, for example, as a SQL stored procedure in SQL Server™ 2005.

So, returning to the question, what is C++/CLI? It is a first-class entry visa into the .NET programming model. With C++/CLI, there is a C++ migration path not just for the C++ source base but for C++ expertise as well. I find great satisfaction in that.

Send your questions and comments to purecpp@microsoft.com.

Stanley B. Lippman is an Architect with the Visual C++ team at Microsoft. He began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. In between, he worked in feature animation at Disney and DreamWorks, was a Distinguished Consultant with JPL, and was a Software Technical Director on Fantasia 2000.


C++ at Work: Persisting View State Update, Using Managed Extensions in a DLL

C++ At Work
Persisting View State Update, Using Managed Extensions in a DLL
Paul DiLascia

Code download available at: CATWork0502.exe (253KB)

This month marks the 11th anniversary of my column and the inauguration of a new title: C++ At Work. We're also adding a new bimonthly column, "Pure C++," by my pal Stan Lippman, one of the great C++ Masters. Stan will cover more of the pure C++/CLI language stuff (he can tell you more himself), and I'll continue to write about the everyday application of C++, MFC (and now the Managed Extensions) to real-world Windows® programming, just as I've done for 11 years. Stan's new column represents the recognition by Microsoft of C++ as one of the most popular and vibrant programming languages alive today, and MSDN®Magazine's commitment to provide our readers even more great C++ coverage. Yeah!

I chose the title C++ At Work because it has a double meaning. C++ at work could mean "C++ on-the-job," as in: for people who use C++ at work. But for me the other more important meaning is putting C++ to work—that is, making C++ do work for you. That's what my column has always been about, and this won't change. The only formal change you can expect is that since the letters "Q&A" no longer appear in the moniker, I reserve the right to occasionally write about some C++ topic or technique I consider important or interesting or valuable, but doesn't necessarily come directly from a reader question. But mostly I plan to keep the Q&A format, so by all means—keep your questions coming!

Q I read your November 2004 column on how to persist the Open dialog view state across user sessions with great interest, but I think you overlooked something. Your CListViewShellWnd (m_wndListViewShell in the dialog) will be destroyed at the moment the user changes folders. So when you close the dialog, it won't save the current view mode, rather the view mode it was in when the user changed to another folder. Is there some way to fix this?

John E. Noël

A Well tie me up and put me in solitary confinement! You're absolutely right, there's a bug in my November code. For readers who missed that issue, I described how to implement a customized Open dialog (CPersistOpenDlg) that remembers the view mode across user sessions. So if the user selects Thumbnails in the Open dialog, then runs your program again tomorrow, the dialog opens in thumbnail view. I did this by subclassing a special window SHELLDLL_DefView, which the Open dialog uses to display files and folders. I used Spy++ to discover the magic command IDs to set the view mode, and I showed how to use LVM_GETITEMSPACING to distinguish between icon and thumbnail views—both of which return LV_VIEW_ICON from LVM_GETVIEW/GetView.

Whenever you want to save your window's state, whether it be size/position or something like the view mode, the natural place to do it is just before the window is destroyed, in your WM_DESTROY/OnDestroy handler. So that's what I did in CListViewShellWnd, my MFC class for SHELLDLL_DefView:

void CListViewShellWnd::OnDestroy() {
   m_lastViewMode = GetViewMode(); // remember current view mode
}

The dialog object (CPersistOpenDlg) then writes m_lastViewMode to the user's profile as it's destroyed, in its destructor. This all makes perfect sense—it's the normal way to do this sort of thing. But as John discovered, the Open dialog doesn't destroy the shell view only when the user closes the dialog; it destroys it every time the user changes folders. As if the only way to change a list is to blow up the world and rebuild it from scratch. (Now you know why the Open dialog can be so slow changing folders.) To see the folder view destroyed, all you have to do is add a TRACE:

void CListViewShellWnd::OnDestroy() {
   TRACE(_T("CListViewShellWnd::OnDestroy\n"));
   ...
}

Now run the dialog and change folders. Sure enough, the folder view is clobbered. So CListViewShellWnd faithfully saves its view mode, but when the user changes folders, the Open dialog destroys its SHELLDLL_DefView, which causes MFC to automatically unsubclass it (in CWnd::OnNcDestroy) and sets my m_hWnd to NULL. The Open dialog then creates a brand new SHELLDLL_DefView which CPersistOpenDlg never sees, so when CPersistOpenDlg writes the view mode to the user's profile, it writes the old mode before the user changed folders. Oops.

Fortunately, this is one of those bugs that points the way to its own solution. Fixing CPersistOpenDlg isn't difficult, it just requires a little thought. The obvious thing would be to trap OnFolderChange and resubclass the folder view. Resubclassing is the right idea, but why do it in OnFolderChange? Whenever you discover something unexpected in Windows, you should always ask: where else might things go awry? If the Open dialog destroys its folder view when the user changes folders, how do you know it doesn't destroy it some other time? Like when the clock strikes midnight on Halloween, or when the Red Sox win the World Series during a lunar eclipse? Rather than patch every event, it's better to find a solution closer to the problem.

CPersistOpenDlg already has a private message, MYWM_POSTINIT, to initialize the dialog. CPersistOpenDlg posts this message to itself in OnInitDialog when the dialog first starts up; the handler subclasses the folder view. All I have to do is post MYWM_POSTINIT again when the folder view is destroyed:

void CListViewShellWnd::OnDestroy() {
   m_lastViewMode = GetViewMode(); // as before
   m_pDialog->PostMessage(MYWM_POSTINIT); // re-post init msg
}

Now if the Open dialog decides to destroy its folder view for whatever reason (folder change or Red Sox), CListViewShellWnd posts the initialization message, causing my dialog to resubclass the new folder view. Clever, eh? It's important to post—not send—MYWM_POSTINIT, so the Open dialog can create the new folder view before CPersistOpenDlg::OnPostInit subclasses it. Fixing the problem in CListViewShellWnd::OnDestroy is guaranteed to catch all places the Open dialog might destroy its folder view, and has the added benefit of not requiring any new event handlers. Never write more code than you have to. If the code doesn't exist, it can't be riddled with obscure bugs.

Astute readers may have noticed I added a data member, m_pDialog, which points to the "parent" CPersistOpenDlg object. Posting MYWM_POSTINIT to the view's parent (GetParent) would be cleaner (because it doesn't require adding a data member), but doesn't work here due to the peculiar way CPersistOpenDlg is actually a child of the real dialog. So I added a back pointer, initialized at construction.

I also had to modify CPersistOpenDlg::OnPostInit slightly. The original handler not only subclassed the folder view, it also initialized the view mode from the user's profile. That's what I want when the dialog first starts—but I don't want to reset the view mode to the saved state every time the user changes folders. So I need a way to distinguish first-time initialization from subsequent reinitialization. Since I wasn't using WPARAM for anything, I used that. In the revised code, OnInitDialog posts MYWM_POSTINIT with WPARAM set to TRUE while CListViewShellWnd::OnDestroy posts it with WPARAM set to FALSE. Figure 1 shows the new OnPostInit handler in action.

Figure 1 OnPostInit.cpp (Close Figure 1)
// Handle MYWN_POSTINIT: not until now is the list view created, so I can
// set it to the remembered value from profile settings.
// WPARAM is TRUE for first-time init; FALSE otherwise.
LRESULT CPersistOpenDlg::OnPostInit(WPARAM wp, LPARAM lp) {
   // Subclass folder view shell window. Real dialog is my parent
   VERIFY(m_wndListViewShell.SubclassDlgItem(lst2, GetParent()));
   if (wp) { // if first-time init:
      // Set view mode based on value saved in profile settings.
      UINT nMode = AfxGetApp()->GetProfileInt(
         PROFILE_SETTINGS, PROFILE_VIEWMODE, 0);
      m_wndListViewShell.SetViewMode(nMode);
   }
   return 0;
}

Finally, some of you may be wondering—what happens when the Open dialog destroys the folder view because the dialog is really shutting down? What happens to the MYWM_POSTINIT message posted from CListViewShellWnd::OnDestroy? Nothing. CListViewShellWnd::OnDestroy posts its message into the void where, like the philosophical tree falling in the woods, it makes no sound because nobody's listening. The entire message queue is destroyed along with the dialog.

Because I'm so nice I rewrote CPersistOpenDlg to include the fixes described here. You can download this updated sample from the MSDN Magazine Web site.

Q I have a DLL that uses MFC and now I want to use the Managed Extensions to call classes in the Microsoft® .NET Framework. When I compile, I get a linker warning LNK4243: "DLL containing objects compiled with /clr is not linked with /NOENTRY; image may not run correctly." I don't really understand what this means so I ignore it—but now my app crashes when I run. Can't I use Managed Extensions in a DLL?

Jimmy Preston

A Just when programming has gotten so easy your Aunt Wendy is writing Web services in C#, you do something simple and suddenly the universe topples down on your poor little head. Fortunately, in this case the solution is described in the article "Converting Managed Extensions for C++ Projects from Pure Intermediate Language to Mixed Mode," which can be found in the MSDN Library. But since you're probably not the only programmer in readerland who's been hit by this train, I'll go over it again here with a little more emphasis on what goes on behind the scenes. You must delve into DLLs, but it's good for you—like eating your peas.Mixed Code Initialization in Visual C++ 2005 and Martyn Lovell

As Paul demonstrated in his column, getting mixed DLLs to initialize in Visual Studio® .NET 2003 is a difficult manual process. For Visual Studio 2005, the Visual C++® and common language runtime (CLR) teams designed a new model for initialization that is simpler and more automatic.

The fundamental problem with the Visual Studio .NET 2003 architecture was that it mixed native code and managed code initializers together into a single "pool." This pool was then unsafe to run during DllMain, since managed code can't ever run at that point.

Our new model collects static initialization into two separate pools. One pool contains static initializers from native compilation; the other contains those from managed compilation.

The first pool is initialized during DllMainCRTStartup, just as for native code. To initialize the second pool, we inserted a new phase of initialization for managed code. When a module containing managed code is loaded into an AppDomain, the CLR will run a special .cctor function (which we call the "module constructor") before any other managed code is executed.

The managed C Runtime (msvcmrt.lib) includes a module constructor which initializes the second pool. It also installs an AppDomain unload event to run managed static destructors and exit functions.

This new model means that migrating to managed code or adding managed code to an existing application will be easier as there will be no need to add a custom initialization entry point. During migration, you may still find cases where native code calls into managed code. This needs to be fixed if your code is to start up and shut down reliably. It will manifest either as an R6031 error from the C Runtime Libraries (CRT), or as a CLR warning when running under the debugger.

To resolve the problem in Visual Studio 2005 Beta 2 and later, use the debugger's call stack to find the offending function, and move that function either to a header file or a native source file so that it is compiled into native code. You may see advice to resolve problems like this through use of #pragma managed. Avoid this wherever possible. It's much harder to make this construct work correctly and efficiently. You should especially ensure that, if you have your own DllMain or RawDllMain, you put this function in a native compilation unit.

If you've used the solution Paul describes, migrating projects from Visual C++ 7.1 to 8.0 shouldn't be much trouble. When you recompile, you'll see deprecation warnings about your use of functions from "_vcclrit.h". You should remove references to that file and manual initialization calls. To fully reinstate use of the CRT, you should also remove /Zl from the compiler switches and /noentry from the linker switches.

This is one of a large number of changes in Visual Studio 2005 intended to ease migration of C++ applications to use of managed code. With our latest builds, we've seen large applications moved to managed compilation in a short amount of time. Take a look at Visual Studio 2005 Beta 2 when it comes out and send us feedback on how well it works.

Let's start with a plain old C DLL. No C++, nothing fancy, just a bunch of exported functions that apps can call. Remember: a DLL is fundamentally just a library of subroutines that get linked at run time as opposed to compile time. (That's the "Dynamic" in DLL.) The same way every C program has an entry point called main, every DLL has an entry called DllMain. (At least, in the vanilla case. As you'll see, DllMain isn't always required.) DllMain's only role in life is to provide a place for you to initialize your DLL. Say you need to create some global state that all your functions use. DllMain is the place to do it. The system calls DllMain whenever a process attaches to, or detaches from, your DLL. Figure 2 shows the basic structure for DllMain.

Figure 2 DllMain.cpp (Close Figure 2)
BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD fdwReason,
    LPVOID lpReserved)
{
    switch( fdwReason ) {
    case DLL_PROCESS_ATTACH:
       // Initialize once for each new process, return FALSE if failure.
       // MFC initializes itself and calls your app's InitInstance.
       break;

    case DLL_THREAD_ATTACH:
       // Thread-specific initialization. Rarely used.
       break;

    case DLL_THREAD_DETACH:
       // Thread-specific cleanup. Rarely used.
       break;

    case DLL_PROCESS_DETACH:
       // Perform any necessary cleanup.
       // MFC terminates itself and calls your app's ExitInstance.
       break;
    }
    return TRUE;  // success!
}

So far, so good. Turn the calendar several years ahead to C++, and now your DLL can have classes. Say you have a Bobble class like the one shown here:

class Bobble {
public:
   Bobble()  { /* create  */ }
   ~Bobble() { /* destroy */ }
};
And suppose your Bobble DLL has a static Bobble instance defined at global scope:
Bobble MyBobble;

MyBobble is a global object all your functions can use, like theApp in MFC. Somehow the compiler must now arrange to have your Bobble constructor called before apps can call any functions in bobble.dll. This never happened in C, where the only static initialization is something like:

int GlobalVal=0;
That is, initialization of primitive types to constant values, which the compiler can manage by itself. But now initialization requires calling functions that execute at run time, not compile time. In C++, you can even write the following line of code to initialize an int by calling a function:
UINT WM_MYFOOMSG = RegisterWindowMessage("MYFOOMSG");
Who calls these functions and when? Answer: the startup code does it just before it calls DllMain. You may think life begins with DllMain, but even before that, the C runtime DLL startup code calls your constructors. If you peer inside crtdll.c, the source code for the CRT's DLL initialization sequence (it's in \VS.NET\VC7\crt\src), you'll find the following function:
BOOL WINAPI _DllMainCRTStartup(...) {
   if ( /* DLL_PROCESS_ATTACH */ ) {
      _CRT_INIT(...); // initialize CRT including ctors
      DllMain(...);
   }
   ...
}

I've simplified to highlight what happens: _DllMainCRTStartup calls another crtdll function, _CRT_INIT, then DllMain. _CRT_INIT initializes the C runtime, then calls all your static object constructors. How does _CRT_INIT know which constructors to call? The compiler has generated a list. So when an application invokes your C++ DLL, the load sequence is as follows:

  1. The app loads your DLL (LoadLibrary)
  2. LoadLibrary calls _DllMainCRTStartup
  3. _DllMainCRTStartup calls _CRT_INIT
  4. _CRT_INIT calls static ctors
  5. _DllMainCRTStartup calls DllMain
  6. DllMain does more initialization
  7. The app calls DLL functions

If you're using MFC, you don't even know DllMain exists because MFC provides it for you. In MFC, life begins with CWinApp::InitInstance. But that's only because MFC has its own DllMain stub that calls it for you.

Okay, now let's spin the time dial forward once more to .NET and the Managed Extensions. Your MFC DLL is working fine, you didn't even know it had a DllMain, and now you want to call the Framework. So you #include <mscorlib.dll>, compile with /clr, and you get the enigmatic /NOENTRY warning. What's going on?

To answer that question, I have to explain one more piece of the puzzle. DllMain runs at a critical time in your DLL's life, when it's in the middle of loading. You can't do anything you want inside DllMain. In particular, you can't call other DLLs because they might not be loaded yet. Calling other DLLs would require loading them first, which could end up through a circular chain of references trying to load your DLL, which is still in the process of loading. Can you say "infinite loop, stack overflow?" This restriction applies even to system DLLs like user32, shell32, and COM objects. Many programmers are dismayed to discover that their DLL crashes when they call MessageBox from DllMain, hoping to display some information during debugging. The only DLL that's guaranteed to be loaded and safe to call from DllMain is kernel32.dll. (For some more information on what you can and can't do in DllMain, see DllMain.)

Having explained that, you might guess what goes wrong with managed DLLs. Once you flip the /clr switch, your DLL becomes managed. Initializing a managed DLL could require running unsafe code during DllMain. The compiler generates Microsoft intermediate language (MSIL) now, not native machine instructions. Only the Redmondtonians know exactly what happens and how many DLLs the system invokes when it encounters the first MSIL instruction in your DLL. Whatever happens, you can bet it involves more than calling the kernel. So by default, managed DLLs don't link with the C runtime, msvcrt.lib. They don't have _DllMainCRTStartup and they don't call DllMain—even if it exists. In other words, in order to prevent calling unsafe code during DLL initialization, the Redmondtonians have simply decreed that managed DLLs shall be /NOENTRY DLLs. A /NOENTRY DLL is one that has no entry point. If your DLL doesn't need to initialize or terminate itself, it doesn't need DllMain and so you can use the /NOENTRY linker option. Such DLLs can't have static objects that require initialization; all they can do is export functions.

But MFC has static objects that require initialization all over the place; it needs for them to work. Does that mean you can't use MFC in your mixed-mode DLL? Of course not! The Redmontonians offer a way out. The solution is described in the "Converting Managed Extensions" article mentioned earlier, and varies slightly depending on whether you're using LoadLibrary or import libs and whether you're writing a normal DLL or a COM object.

Since you asked about MFC, I'll describe what to do for C++/MFC DLLs that use import libraries, which is the more common case. The basic idea is simple: first make your DLL a NOENTRY DLL by adding /NOENTRY to the linker options. That keeps the linker happy. To initialize your objects, you have to write your own initialization/termination functions and export them. Any application that uses your DLL must call your init function before it calls any functions in your DLL, and must likewise call your term function when it's finished.

That sounds like a good plan, but how do you implement the init/term functions? Lucky for you, the Redmondtonians have provided a file with a couple of functions that make life easy; all you have to do is include the file and call them:

#include <_vcclrit.h>
BOOL __declspec(dllexport) MyDllInit() {
   return __crt_dll_initialize();
}
BOOL__declspec(dllexport) MyDllTerm() {
   return __crt_dll_terminate();
}

That wasn't hard, was it? If you peek inside _vcclrit.h and read past all the gnarly multithreading stuff, you'll see all __crt_dll_ initialize does is call _DllMainCRTStartup(DLL_ PROCESS_ ATTACH). Likewise, __crt_dll_terminate calls _DllMainCRTStartup (DLL_PROCESS_DETACH). If you're writing in C++ and you don't mind losing the Boolean return code, you can write an auto-init class whose constructor/destructor encapsulates the __crt_dll_xxx calls, so all client apps have to do is instantiate an auto-init object somewhere at global scope:

CMyLibInit initlib; // in main app

Figure 4 LibTest
Figure 4 LibTest

Note that the CMyLibInit approach works only when your DLL is consumed by an EXE; if your client is another DLL and you try to instantiate CMyLibInit in the DLL's global scope, __crt_dll_initialize will be called under loader lock again, potentially triggering the same DLL-deadlock problem you're trying to avoid. To initialize a mixed DLL from another native DLL, you have to find an appropriate place to call the mixed DLL's init/term functions—for example, if the native DLL is a COM object, you could do it from DllGetClassObject/DllCanUnloadNow. Otherwise you need to export yet another layer of init/term functions from your native DLL that in turn call the constructors or DllMain since these are under loader lock.

Figure 3 shows a DLL I wrote that uses the CMyLibInit approach. ManLib exports a function, FormatRunningProcesses, that calls the Process class in the .NET Framework to format a list of running processes and return it in a CString. The test app calls FormatRunningProcesses and displays the list in a message box (see Figure 4). I sprinkled ManLib with TRACE diagnostics to help you see which functions get called when. Figure 5 is a sample run that shows the order of events. You can compare this output with the TRACE statements in Figure 3 to better understand the call order. If you implement something like CManLibInit at home, remember: you can't do anything before calling __crt_dll_initialize (or after calling __crt_dll_terminate). The first time I wrote ManLib, I put my TRACE at the top without thinking, like the following:

CManLibInit::CManLibInit() {
   TRACE(...); // Oops!
   __crt_dll_initialize();
}
This crashed my program because MFC's trace mechanism isn't initialized yet. That's what __crt_dll_ initialize does!

Figure 3 MainLib (Close Figure 3)

MainLib.h
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
#pragma once

#if defined(BUILD_MANLIBDLL)
#  define MANLIBEXP __declspec ( dllexport )
#elif defined(USE_MANLIBDLL)
#  define MANLIBEXP __declspec ( dllimport )
#else
#  define MANLIBEXP
#endif

// Instantiate one of these somewhere in your app to initialize the DLL.
class MANLIBEXP CManLibInit {
public:
   CManLibInit();
   ~CManLibInit();
};

extern CString MANLIBEXP WINAPI FormatRunningProcesses(UINT nMax=0);
MainLib.cpp
#include "stdafx.h"
#include "ManLib.h"
#include "TraceWin.h"
#include <_vcclrit.h>

using namespace System::Diagnostics; // Process class lives here

// App class displays TRACE diagnostics to show what happens.
class CManLibDLL : public CWinApp {
public:
   CManLibDLL() {
      TRACE(_T(" CManLibDLL::CManLibDLL\n"));
   }
   virtual BOOL InitInstance() {
      TRACE(_T(" CManLibDLL::InitInstance\n"));
      return TRUE;
   }
   ~CManLibDLL() {
      TRACE(_T(" CManLibDLL::~CManLibDLL\n"));
   }
};

// one-and-only global instance.
CManLibDLL theDLL;

// Special initialization/termination required for managed DLL.
// The __crt_xxx fns are in _vcclrit.h.
CManLibInit::CManLibInit() {
   __crt_dll_initialize();
   TRACE(_T(" CManLibInit::CManLibInit—initialized DLL\n"));
}
CManLibInit::~CManLibInit() {
   TRACE(_T(" CManLibInit::~CManLibInit—terminating DLL\n"));
   __crt_dll_terminate();
}

// Get process list in a CString. Calls .NET Framework Process class.
CString MANLIBEXP WINAPI FormatRunningProcesses(UINT nMax) {
   TRACE(_T(" FormatRunningProcesses\r\n"));
   if (nMax==0) nMax—; // 0==>show all

   CString ret;
   Process* allprocs[] = Process::GetProcesses();
   UINT len = allprocs->Length;
   for (UINT i=0; i<len; i++) {
      if (i>=nMax) {
         ret += _T("More...\r\n");
         break;
      }
      Process* p = allprocs[i];
      CString pinfo;
      try {
         ProcessModule* pm = p->MainModule;
         pinfo.Format(_T("Process %p: %s\r\n"),
                       p->Id, CString(pm->ModuleName));
      } catch (Exception* e) {
         pinfo.Format(_T("Process %p: %s\r\n"), p->Id,
                       CString(e->Message));
      }
      p->Dispose();
      ret += pinfo;
   }
   return ret;
}
Figure 5 TraceWin
Figure 5 TraceWin

One more thing before I go. The solution described in the MSDN Library article tells you to add msvcrt.lib to your link libraries and __DllMainCRTStartup@12 (the mangled name) under "Force Symbol References" in your project settings. If you're starting with a DLL that already uses MFC, these steps aren't necessary because MFC already links what you need. All that you have to do is add /NOENTRY to your linker options, write the init and term functions, and call them from your application (see the sidebar Mixed Code Initialization in Visual C++ 2005). Happy programming!

Send your questions and comments for Paul to cppwork@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at www.dilascia.com.


{ End Bracket }: C# and VBA: Like Oil and Water

{ End Bracket }
C# and VBA: Like Oil and Water
Ken Getz

Some things just don't mix as well as you would like. Take C# and Microsoft® Excel 2003 or Word 2003, for example. Not only are these applications huge productivity tools, but they both also provide access to large object models that you can program against from your own applications. The problem is none of the documentation for the object models includes code samples for developers writing managed code, and writing Visual Basic® for Applications (VBA) code is quite unlike writing C# code.

I spent much of summer 2004 creating managed code samples exercising many of the methods of both Word and Excel, providing samples in both C# and Visual Basic .NET. The task seemed simple—400 code snippets, all converted from Visual Basic .NET to C#—but was much harder than you might imagine.

It's not that it's difficult to convert code from Visual Basic .NET to C#, or to simply create code that automates Word or Excel in C#. What caused me pain in this particular project was that when you attempt to communicate with objects designed to be consumed by VBA clients, you're living in a world that didn't take C# and its design goals into account when it was being created. One thing is for sure: because of its VBA lineage, it's a lot easier accomplishing Office automation from Visual Basic .NET than it is from C#. I can't dig into the details here, so I'll just point out a few delectable tidbits, with links to more info later on.

From my perspective, the least pleasant issue is that of passing parameters to Word members. Before taking Word and its design to task, remember that it started its career including Word Basic rather than VBA, back in the Pleistocene era of computing. In those days, it was typical for scripting languages to accept and expect all parameters to be passed by reference, such that incoming parameters were simply full-featured aliases of the original variables, and this habit remains unbroken in Word to this day (a lot of design decisions are also a relic of designing for COM).

For the most part, all parameters must be passed by reference when calling a method in Word, so code that looks like the following Visual Basic .NET snippet needs to be converted for use in C#:

' Visual Basic .NET
' wdApp refers to the Word Application object.
wdApp.Windows.Arrange(Word.WdArrangeStyle.wdTiled)

// C#
object value = Word.WdArrangeStyle.wdTiled;
wdApp.Windows.Arrange(ref value);
It's not a terrible burden, but it does force you to create a variable each and every time you want to pass a parameter.

Optional parameters provide separate challenges when working with both Excel and Word. When programming Excel, you simply have to accept the fact that although VBA welcomes optional parameters, C# doesn't handle them in the same way (for an explanation of why this is the case, you can take a look at Anders Hejlsberg's TechEd 2004 discussion of the subject at Whiteboard with Anders Hejlsberg). Many Excel methods take advantage of optional parameters to ease the burden for VBA developers, and you must supply a value for each of the missing parameters. Luckily, the Type.Missing value indicates to the Primary Interop Assembly (PIA) that your code intends to accept the default value for the optional parameter.

Passing optional parameters to Word adds an extra challenge because they're passed by reference. Luckily, the PIA takes care of marshaling the values, so you needn't worry about "by reference" side effects passing a single variable to handle all the optional parameters. The following snippets demonstrate a technique for handling optional parameters when programming against the Word object model:

' Visual Basic .NET
' wdApp refers to the Word Application object.
ThisApplication.Documents.Open("C:\Test\MyNewDocument")

// C#
object filename = @"C:\Test\MyNewDocument";
object missing = Type.Missing;
wdApp.Documents.Open(ref filename,
    ref missing, ref missing, ref missing,
    ref missing, ref missing, ref missing,
    ref missing, ref missing, ref missing,
    ref missing, ref missing, ref missing,
    ref missing, ref missing, ref missing);

Of course, there are more issues and many subtleties. For more information, check out a longer article I wrote previously on the topic (Programming Office Applications Using Microsoft Visual C#), and for the real scoop, you should take a look at Eric Carter's blog (Type.Missing, C#, and Word). He's the expert on this particular subject.

Ken Getz is a senior consultant with MCW Technologies. He is coauthor of ASP .NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2001), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.