 MSDN Magazine April 2008

MSDN Magazine April 2008

Talk Back: Voice Response Workflows with Speech Server 2007

Speech Server 2007 lets you create sophisticated voice-response applications with Microsoft .NET Framework and Visual Studio tool integration. Here’s how.

Michael Dunn Performance: Scaling Strategies for ASP.NET Applications

Performance problems can creep into your Web app as it scales up, and when they do, you need to find the causes and the best strategies to address them.

Richard Campbell and Kent Alstad Silverlight: Building Advanced 3D Animations with Silverlight 2.0

Animating with Silverlight is easier than you think. Here we create a 3D app that folds a polyhedron using XAML, C#, and by emulating the DirectX math libraries.

Declan Brennan Interview++: Bjarne Stroustrup on the Evolution of Languages

Howard Dierking talks to the inventor of C++, Bjarne Stroustrup, about language zealots, the evolution of programming, and what’s in the future of programming.

Howard Dierking Office Development: Manage Metadata with Document Information Panels

Here the author uses Document Information Panels in the Microsoft 2007 Office system to manipulate metadata from Office docs for better discovery and management.

Ashish Ghoda Editor's Note: A Service-Oriented Editor's Note

Here is what Howard Dierking has to say about the differences between services and distributed applications and how he has approached services in his projects.

Howard Dierking Toolbox: Logging Web App Errors, Learning LINQ, and More

See how to log Web application errors for better health monitoring, what to read if you’re thinking about LINQ, and which blog Scott recommends reading this month.

Scott Mitchell CLR Inside Out: Measure Early and Often for Performance, Part 1

In this month’s column, get the inside scoop on how to build performance into your apps from the start, rather than dealing with the fallout after you deploy them.

Vance Morrison Basic Instincts: My Namespace Extensions with My Extensibility

In this month’s column, learn how to integrate your own My Namespace extension into the Visual Basic development environment using the My Extensibility feature.

Matthew DeVore Cutting Edge: ListView Tips and Tricks

This month, use nested ListView controls to create hierarchical views of data and extend the eventing model of the ListView by deriving a custom ListView class.

Dino Esposito Foundations: Code Access Security in WCF, Part 1

Here we discuss code-access security in Windows Communication Foundation (WCF) and present a solution for enabling partially trusted clients for WCF services.

Juval Lowy Test Run: Testing SQL Stored Procedures Using LINQ

Language Integrated Query makes lots of things easier. Here we put LINQ, or more specifically the LINQ to SQL provider, to use testing SQL stored procedures.

Dr. James McCaffrey Service Station: Building a WCF Router, Part 1

Here we take a look at WCF clients and services, and explain how to tweak the addressing and message filtering behaviors so you can route services between them.

Michele Leroux Bustamante Windows with C++: Windows Imaging Component Basics

Windows Imaging Component (WIC) is an extensible framework for encoding, decoding, and manipulating images. See how to use WIC to encode and decode different image formats.

Kenny Kerr Going Places: Provisioning Mobile Devices

Learn how you can set up every mobile device in your company with a few lines of code and some XML--thanks to the provisioning APIs in the Windows Mobile SDK.

Mike Calligaro { End Bracket }: The Magic of Software

Are the transparent computer screens in the movie Minority Report or the flashy fingerprint analysis software programs used on CSI closer than we think? Maybe so.

Julia Lerman


Talk Back: Voice Response Workflows with Speech Server 2007

Talk Back

Voice Response Workflows with Speech Server 2007

Michael Dunn
This article discusses:
  • Voice response application basics
  • Creating a voice response workflow
  • Prompts, keywords, and grammars
  • Handling user responses
This article uses the following technologies:
Speech Server 2007, .NET Framework
Code download available at: SpeechServerAndWF2008_04.exe (188KB)
Contents
Voice App Basics
Voice Response Workflow
Building Prompts
Prerecorded Prompts
Keyword and Conversational Grammars
Handling Responses
Debugging and Testing

E
mbedded presence, instant messaging (IM), audio and video conferencing, and telephony are among the unified communications features offered by Microsoft® Office Communications Server (OCS) 2007. Developers can build on an array of OCS APIs to include these and other features in their own applications. But OCS 2007 adds one new developer-centric feature you may not have heard about yet—interactive voice response (IVR) workflows based on the Microsoft Speech Server platform.

Not sure what an IVR is? If you've ever called a company and heard a message like "press 1 for sales, press 2 for customer service," then you've encountered an IVR system.

Voice App Basics

Microsoft Speech Server 2004 only supported IVR development via the Speech Application Language Tags (SALT) API. The new version, OCS 2007 Speech Server (Speech Server 2007 for short), not only supports both SALT and VoiceXML but introduces a Microsoft .NET Framework API for creating IVR applications. Speech Server 2007 also includes a visual IVR application designer based on Windows® Workflow Foundation (Windows WF) called the Voice Response Workflow Designer.

Current IVR development standards such as SALT and VoiceXML focus on using a Web-based development model relying on XML tags and JavaScript. Both of these standards focus on delivering the user interface of an IVR application. However, any non-trivial IVR application still needs to perform tasks such as accessing data from a database. For these tasks, server-side code, typically developed in a .NET-compliant language or Java, is still needed. And this somewhat diminishes the main benefit that SALT and VoiceXML intend to provide: cross-platform compatibility for IVR applications.

From a developer perspective, using the .NET Framework as the basis for your IVR application development has significant benefits over SALT or VoiceXML, the main advantage being the ability to use an object-oriented programming language instead of XML tags. From a business perspective, a .NET Framework-based IVR allows the business to utilize investments they have already made in Microsoft technologies by being able to reuse existing business and data-access logic.

Another benefit is finding key resources. Try asking a hiring manager or recruiter to find you a developer using VoiceXML and a developer using .NET. Which one do you think they'll be able to find first? There are just more developers around who use .NET.

Voice Response Workflow

Much like an ASP.NET page, the logic of a Voice Response Workflow consists of a partial class in two files, one containing the Visual Studio®-generated code and the other containing your code (called the code-beside). Voice Response Workflows use Speech Dialog activities as well as standard Windows WF activities. The Visual Studio toolbox contains both activity types.

The first step in developing an IVR application is to lay out the call flow. In Visual Studio you do that by dragging the Speech Workflow activities onto the Speech Workflow canvas and arranging them in the order you want the activities to execute, as shown in Figure 1.

Figure 1 Voice Response Workflow Designer
Figure 1 Voice Response Workflow Designer(Click the image for a smaller view)
Figure 1 Voice Response Workflow Designer
Figure 1 Voice Response Workflow Designer (Click the image for a larger view)

For an IVR application that responds to inbound calls, the first activity is typically AnswerCall and the last activity is typically DisconnectCall, so these activities are added to the Speech Workflow automatically upon creating a new Voice Response Workflow. Speech activities execute in order, from top to bottom, by default. Although Voice Response Workflow applications are based on Windows WF, they only support sequential workflows, not state machine workflows.

The two most commonly used activities are Statement and QuestionAnswer. Using just these two activities you can develop a fairly simple IVR application with minimal effort. Both activities speak to the user either using Text-to-Speech (TTS) or prerecorded prompts. The QuestionAnswer activity varies from the Statement activity in that while the Statement activity only speaks to the user, the QuestionAnswer activity lets the user give a response either through speech recognition or touch-tone (Dual Tone Multi-Frequency or DTMF) key presses.

Building Prompts

Each activity that provides information to the user through either TTS or prerecorded prompts has a property named MainPrompt, which is of type PromptBuilder. MainPrompt defines what is spoken to the user when the activity is first executed. The PromptBuilder type provides multiple methods for fine-tuning how a prompt will sound. The three most commonly used methods are SetText, ClearContent, and AppendText. Here is an example of using SetText to create the message "Thank you for calling.":

this.statementActivity.MainPrompt.SetText("Thank you for calling.");

These two lines use ClearContent and AppendText to send the same "Thank you for calling." message:

this.statementActivity.MainPrompt.ClearContent();
this.statementActivity.MainPrompt.AppendText("Thank you for calling.");

Both of these snippets accomplish the same objective, so you can use either one. My general rule is, if I have a line containing ClearContent immediately followed by an AppendText method, I'll replace it with the SetText method for the simple reason that one line of code is easier to maintain than two.

In a Statement activity (see Figure 2), there are a standard set of events that fire each time the prompt is rendered. Setting a static prompt in any of these events would cause your app to write the same value of the prompt each time the prompt was rendered.

Figure 2 Statement Activity Order of Event Execution
Figure 2 Statement Activity Order of Event Execution(Click the image for a smaller view)
Figure 2 Statement Activity Order of Event Execution
Figure 2 Statement Activity Order of Event Execution (Click the image for a larger view)

It's more efficient to set static prompts before the activity is initialized, such as in the Initialize event of the workflow instead of the activity. If the prompt is dependent on a dynamic value of some sort, it's still more efficient to set the value before the activity is initialized. In fact, the most efficient method for setting static prompts is to use the Visual Studio tools, which put the prompt text in a resource file that is loaded only once when the constructor of the workflow calls the InitializeComponent method.

As for dynamic prompts, the story starts to change when you look at activities that not only speak to the user but also look for user input. A good example is the QuestionAnswer activity. While QuestionAnswer follows the same order of execution, it also accepts user input. This means that if the user says something the system doesn't recognize, the TurnStarting event fires again. This would be inefficient, as you are setting the same dynamic prompt again as well as duplicating the use of the dynamic resource, such as a database lookup. If the dynamic prompt must be set at the time of activity execution, set it during the Executing event over the TurnStarting event. This is because the Executing event is only fired once per session, while the TurnStarting event is fired every time the QuestionAnswer activity is rendered.

The text that is actually spoken is dependent on how many times the activity has been played to the user and what kind of response they gave. You can define what is spoken at a given point in time by assigning the text you want spoken for a corresponding prompt type.

The QuestionAnswer activity has four more prompt types than a Statement activity: Silence, Escalated Silence, No Recognition, and Escalated No Recognition. While MainPrompt is mandatory to set, these prompts are optional. The Silence prompt is spoken to the user when a user doesn't give any response before the time specified in the InitialTimeout property. If the user still doesn't give a response, the Escalated Silence prompt will play. The No Recognition and Escalated No Recognition prompts follow the same rules as the Silence and Escalated Silence prompts, but are used when the user gives a response the application doesn't recognize.

Dynamic prompts create some problems when converting the text resource into spoken words, and two of the most common problems are dates and times. Typical data sources store dates in a readable format such as 01/01/2008. However, when spoken, you want to hear "January first two thousand eight." The solution for translating this commonly understood date format into a commonly understood spoken format is the AppendTextWithHint method. This method accepts the same string parameter as AppendText does, but it also takes a SayAs parameter. The SayAs parameter is a simple Enum containing the different options for manipulating readable text into spoken text. In the example of converting a date, the code would look like the following:

this.statementActivity.MainPrompt.AppendTextWithHint(
    "01/01/2008", Microsoft.SpeechServer.Synthesis.SayAs.Date);

Prerecorded Prompts

By default, your Voice Response Workflow will use the TTS engine's voice to speak the text defined in your prompts. Using the TTS engine, however, requires a lot of processing power and can end up sounding like every other system using TTS.

Your other option is to use prerecorded prompts. While you can code your prompts to play a WAV file instead of assigning text, it's better to simply add a prompt database project to the solution. A prompt database enables you to easily create, edit, and manage the WAV files used in your application as well as storing them in a compressed format. This compressed format allows the prompts to be loaded faster. If you use a prompt database, you won't have to make any coding changes to your application's prompts.

When you import or create WAV files in the prompt database, you can assign the transcription of the WAV file. When the application encounters text to be spoken, it looks in the prompt data for a match on the transcription field. For example, if the application needs to speak "Thank you for calling." and there is a transcription of "Thank you for calling." in the prompt database, it will play the associated WAV file instead of rendering the text using the TTS.

The process of creating and importing these WAV files can be tedious if your application has numerous prompts. Let's consider the following prompts:

  • "I'll transfer you to a representative."
  • "I'll transfer you to a loan officer."
  • "A representative will be with your shortly."
  • "A loan officer will be with you shortly."

Instead of creating four different prerecorded prompts you could use an extraction technique. Extraction allows you to combine all or part of a phrase from one WAV file with all or part of a phrase from another. Extraction doesn't happen automatically; you must specify in the transcription field which part of the transcription can use extraction. So instead of having four different WAV files in the prompt database you can simply have two WAV files with extraction brackets, as shown in Figure 3.

Figure 3 Prompts with Extraction Annotated
Figure 3 Prompts with Extraction Annotated(Click the image for a smaller view)
Figure 3 Prompts with Extraction Annotated
Figure 3 Prompts with Extraction Annotated (Click the image for a larger view)

When the application encounters the phrase "I'll transfer you to a," it will automatically combine that specific audio portion with another portion of audio. It will even mix prerecorded prompts with TTS if part of the encountered prompt isn't in the prompt database. So if you later add another prompt that says, "I'll transfer you to a mortgage specialist," the phrase "I'll transfer you to a" will use the WAV file, while the "mortgage specialist" phrase can be rendered using the TTS engine.

Keyword and Conversational Grammars

Now that your application can talk, you'll want it to be able to accept responses from the user either through speech recognition or DTMF. Speech Server 2007 does not automatically recognize everything the user says, so you must specify acceptable user responses. These are defined in your application's grammar. In a speech recognition app, there are two types of grammars: keyword and conversational. Keyword grammars are based on the use of specific words in the response. Keyword grammars are good for asking the user a very direct question, such as, "Do you want Account Information, Loan Inquiry, or to speak with a representative?"

If the user says anything other than the list of acceptable answers, even if the user's response is very similar to what the application is looking for, the application would probably respond with something like, "I didn't understand what you said. Please say ..."

Conversational grammars, on the other hand, try to address this problem by taking some of the responsibility off the user and putting it back on the application to figure out what the user wants. Instead of asking a direct question, you ask the user an open-ended question: "How can I help you?"

This is a more natural way of asking a question, and you will probably get a more natural and complex answer back from the user. The Conversional Grammar Builder allows you to build conversational grammar easily, taking much of the effort out of working with a statistical language model yourself.

The Conversational Grammar Builder, shown in Figure 4, is divided in two sections: Keywords and Answers. In the Answers section, you have three options: Concept Answer, Keyword Answer, and Command. Concept Answers differ from the Keyword Answers in that Keyword Answers require the caller to say the exact phrase to trigger recognition while Concept Answers allow the user to give a response similar to one of the Answer phrases you have predefined.

Figure 4 Conversational Grammar Builder
Figure 4 Conversational Grammar Builder(Click the image for a smaller view)
Figure 4 Conversational Grammar Builder
Figure 4 Conversational Grammar Builder (Click the image for a larger view)

You'll want to add one Concept Answer for the main menu prompt for this example (see Figure 4). The Concept Answer typically represents one question you are asking the user to give a response to, in this case the main menu question, "How can I help you?"

Next, you'll need to add one Concept to the Concept Answer for each of the possible choices the application will allow for this prompt. For the Main Menu QuestionAnswer activity created earlier, I would have one Concept Answer named MainMenu and three concepts named AccountInformation, LoanInquiry, and Representative.

Now consider this phrase: "I'd like to pay my loan." If this phrase was evaluated simply on keywords, it might be recognized as an inquiry for a new loan, when really the user is trying to make a payment on their account, which probably resides as a secondary menu choice under AccountInformation.

The Answer Examples pane is where you can add phrases that a user is likely to say. You'll need an example phrase for each of the defined concepts. In the Representative node, you might have a list of the following phrases:

  • "I need to talk to a person."
  • "Can I speak with someone?"
  • "I'd like to talk with a representative."

Next, you need to consider the use of keywords, not for recognition, but to expand the Answer Examples phrases that you've entered. Notice that words person, someone, and representative all refer to the same object, a representative. What if the user says "I need to talk to a representative"? Instead of creating three different sentences for each Answer Example, you can create keywords so the sentences are more dynamic. The use of keywords is not necessary when using a Concept node in your conversational grammar, but it will save you time when developing the answer phrases. Figure 4 shows how we have consolidated three phrases into one phrase by using keywords.

In the Keywords area of the Conversational Grammar Builder, you'll need a keyword container that corresponds with the Representative node under Answers. In this case, I called it RepresentativeKeys. Under this node, you might want to have a keyword for Person, and in the Keyword Phrase area you add words or phrases that fall under the main keyword. For example, the Person keyword might have a list of words which a user might use to describe a representative, including person, representative, and someone.

The goal of the keyword is not to distinguish the between these words, but a list of words that a user might use to describe a representative. If the application needs to determine the difference between the words person and representative, each would be an additional keyword container node under Keywords.

Now on the Representative concept node you can reference the RepresentativeKeys keyword node, and then click on the Parse button. This will replace the keywords found in the concept phrases and replace them with brackets indicating that the word isn't static, but a variable belonging to the referenced keyword. A green checkmark indicates that a keyword is being used in a phrase, while a red "x" indicates that no keywords are being used. For concept type grammars, keywords are not required, so it's acceptable and very likely that the phrases may not reference any keywords.

You can reference the completed grammar in a QuestionAnswer activity or any other activity that accepts user input. This can be done through the Visual Studio environment or from your code-beside. Like prompts, if you set grammar via your code and not through Visual Studio, keep in mind the order of event execution; you don't need to repeatedly set the grammar.

Handling Responses

Now that your application can talk and listen, you need to get the results of what your user said. To get these results via your code, you employ the RecognitionResult property of the QuestionAnswer activity. You can retrieve the exact phrase the user said via the Text property:

this.questionAnswerActivity.RecognitionResults.Text;

If the user were to respond, "I need to talk to a person," the Text property of the recognition results would contain "I need to talk to a person." How does your app know that the phrase means that they want to speak with a representative? This is where Semantic values come into play. Your app doesn't care whether the user said, "I need to talk to a person" or "Can I speak with someone?" It only needs to know that they want to speak with a representative.

Instead of the Text property, you can use the SemanticResult property and provide a key value. In this case, the Concept Answer name, MainMenu, is the key value:

this.questionAnswerActivity.RecognitionResults.Semantics[
  "MainMenu"].Value.ToString();

This code would return the name of the Concept. In the case of the user saying, "Can I speak with someone?," it would return "Representative."

You have retrieved the results, but now you need to do something with those results. Typically in Voice Response Applications you'll want to create branches, representing each of the possible choices. If you referenced the grammar via Visual Studio you can right-click on the QuestionAnswer activity and choose Generate Branching. This will automatically create each of the branches and create the criteria for each branch. Alternatively, you can create each branch yourself and set the criteria regarding when the user will enter the branch based on the recognition results. For example, from the MainMenu you might have a Representative branch, which would transfer the user to a representative upon successful recognition.

Debugging and Testing

Voice Response Workflow applications give you the same debugging experience you are already accustomed to. Unlike most other applications, however, you need to be able to hear and speak to an IVR application. Speech Server adds a Voice Response Debugging Window into Visual Studio, as shown in Figure 5.

Figure 5 Debugging Window
Figure 5 Debugging Window

This debugging phone lets you test your app without actually deploying it or even having any telephone lines connected. You can test your grammars using a microphone or simply enter the text you want the app to test its grammars against. While it seems like a simple feature, most other IVR platforms overlook the ability to test and debug without deploying. To learn more about Speech Server 2007 and download the tools, see the Microsoft Speech Technologies site at microsoft.com/uc/products/speechserver.mspx.

Michael Dunn is a Senior Consultant at Microsoft. He is also the author of the book Pro: Microsoft Speech Server 2007. You can contact him via his blog at blogs.msdn.com/midunn.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Performance: Scaling Strategies for ASP.NET Applications

Performance

Scaling Strategies for ASP.NET Applications

Richard Campbell and Kent Alstad
This article discusses:
  • Scaling ASP.NET applications and databases
  • Optimizing code
  • Efficient caching
  • Affinity and load balancing
This article uses the following technologies:
ASP.NET
Contents
The Performance Equation
Scaling Problems
Optimizing the Code
Load Balancing
Affinity
Minimizing Payload
Caching
Scaling Databases
The Endless Scaling Effort

A
ASP.NET performance advisors, we are typically brought into a project when it's already in trouble. In many cases, the call doesn't come until after the application has been put into production. What worked great for the developers isn't working well for users. The complaint: the site is too slow. Management wants to know why this wasn't discovered in testing. Development can't reproduce the problem. At least one person is saying that ASP.NET can't scale. Sound familiar?

Some of the busiest Web sites in the world run on ASP.NET. MySpace is a great example; in fact, it was migrated to ASP.NET after running on a number of different platforms. The fact is, performance problems can creep into your app as it scales up, and when they do, you need to determine what the actual problem is and find the best strategies to address it. The biggest challenge you'll face is creating a set of measurements that cover the performance of your application from end to end. Unless you're looking at the whole problem, you won't know where to focus your energy.

The Performance Equation

In September 2006, Peter Sevcik and Rebecca Wetzel of NetForecast published a paper called "Field Guide to Application Delivery Systems." The paper focused on improving wide area network (WAN) application performance and included the equation in Figure 1. The equation looks at WAN performance, but with a few minor modifications it can be used to measure Web application performance. The modified equation is shown in Figure 2 and each element is explained in Figure 3.

Figure 3 Elements of the Performance Equation (Close Figure 3)

Variable | Definition
R | Response time. The total time from the user requesting a page (by clicking a link, and so on) to when the full page is rendered on the user's computer. Typically measured in seconds.
Payload | Total bytes sent to the browser, including markup and all resources (such as CSS, JS, and image files).
Bandwidth | Rate of transfer to and from the browser. This may be asymmetrical and might represent multiple speeds if a given page is generated from multiple sources. Usually, it is averaged together to create a single bandwidth expressed in bytes per second.
AppTurns | The number of resource files a given page needs. These resource files will include CSS, JS, images, and any other files retrieved by the browser in the process of rendering the page. In the equation, the HTML page is accounted for separately by adding in round-trip time (RTT) before the AppTurns expression.
RTT | The time it takes to round-trip, regardless of bytes transferred. Every request pays a minimum of one RTT for the page itself. Typically measured in milliseconds.
Concurrent Requests | Number of simultaneous requests a browser will make for resource files. By default, Internet Explorer performs two concurrent requests. This setting can be adjusted but rarely is.
Cs | Compute time on the server. This is the time it takes for code to run, retrieve data from the database, and compose the response to be sent to the browser. Measured in milliseconds.
Cc | Compute time on the client. This is the time it takes for a browser to actually render the HTML on the screen, execute JavaScript, implement CSS rules, and so on.
Figure 1 The Original Performance Equation
Figure 1 The Original Performance Equation(Click the image for a smaller view)
Figure 1 The Original Performance Equation
Figure 1 The Original Performance Equation (Click the image for a larger view)
Figure 2 The Web Version of the Performance Equation
Figure 2 The Web Version of the Performance Equation(Click the image for a smaller view)
Figure 2 The Web Version of the Performance Equation
Figure 2 The Web Version of the Performance Equation (Click the image for a larger view)

Now that you have the formula, the challenge lies in measuring each element. The ending value, response time, is relatively easy to measure; there are a number of tools that will time exactly how long the entire process takes.

Payload can be measured using various tools (websiteoptimization.com/services/analyze is a great option), as can Bandwidth (see speedtest.net) and round-trip time (using Ping). Tools like websiteoptimization.com/services/analyze will also report the size of a Web page's HTML, CSS, JavaScript, images, and so forth. Concurrent Requests is essentially a constantly (Internet Explorer® defaults to 2).

That leaves Cs and Cc, which need some additional development effort. It's relatively straightforward to write code in an ASP.NET page that notes the exact second execution of the page begins and subtracts that time from the current time when execution completes. The same is true on the client side; a bit of JavaScript can execute right at the top of the HTML page to note the time and then subtract the time at the point that the OnLoad event fires when the page is completed.

In fact, all these elements can be coded for if you want to build in a debug mode of your Web site that utilizes the performance equation. And there's good reason to do so: if you can routinely render the performance equation elements on the browser, then you can easily detect where your performance problems lie.

For example, suppose you have an ASP.NET application whose users are on another continent and have low bandwidth. With high ping times (> 200ms) and low bandwidth (< 500kbps), your users would be highly sensitive to the total payload and the number of round-trips in your application. Looking at your application in the context of those users is vital since their experience will be very different from yours.

Scaling Problems

As consultants, we know we're probably dealing with a scaling issue when the application performs well in the test environment but poorly in the real world. Usually, the only difference between the two is the number of simultaneous users. If the application performed poorly all the time, you would have a performance problem rather than a scaling problem.

There are three strategies available that you can employ in order to improve scaling: specialization, optimization, and distribution. How you apply them will vary, but the actual strategies are straightforward and consistent.

The goal of specialization is to break your application into smaller pieces in order to isolate the problem. For example, you might want to consider moving static resource files such as images, CSS, and JS files off of the ASP.NET servers. A server well-tuned for ASP.NET is not especially well-suited to serving those sorts of files. For this reason, a separate group of IIS servers tuned to serve resource files can make a substantial difference in the scalability of the application you are running.

If you perform a lot of compression or encryption (for SSL), setting up servers dedicated to SSL can help. You should be aware that there are even specialized hardware devices available for compression and SSL termination.

While more traditional strategies for decomposing server tiers might have you considering separate servers for data access, complex computations, and so on, independent of the actual generation of the Web pages, I'd rather have five Web servers that do everything than three Web servers and two business object servers. All those out-of-process calls between the Web servers and business object servers create a lot of overhead.

Specialization should only be done for a known and expected benefit. And the fastest solution is not always the best. The goal of scalability is consistency of performance. You want to narrow the performance range as the load increases; whether there's one user or one thousand, you want a given page rendered for all users in the same amount of time.

Eventually you will need to optimize your server code to scale more effectively. Virtually every aspect of the performance equation scales linearly, except for compute time on the server; you can always add more bandwidth (and it's fairly easy to know when), and the compute time on the client doesn't change as the number of clients increases. The other elements of the performance equation also remain consistent as you scale. But compute time on the server will need to be tuned as the number of users increases.

Optimizing the Code

The trick to optimizing server code is to use testing to be sure you're actually making a difference. You should use profiling tools to analyze your application and find out where the application is spending the most time. The entire process should be empirical: use tools to find the code to improve, improve the code, test to see that you have actually improved performance, rinse, lather, repeat. In really large-scale sites, you'll often hear performance tuning like this compared to the job of painting the Golden Gate Bridge: once you finish painting the entire thing, it's time to go back to the beginning and start again.

I'm always amazed at the number of people who believe that the starting point of scaling is distribution. "Throw more hardware at it," they yell. Don't get me wrong; without a doubt, adding hardware can help. But without specialization and optimization, the return can be small indeed.

Specialization lets you distribute smaller parts of your application as needed. If you've separated out your image servers, for example, it's easy to scale your image services independently of the rest of the application.

Optimization also provides dividends for distribution by reducing the amount of work needed for a given operation. This translates directly into fewer servers needed to scale to the same number of users.

Load Balancing

To implement distribution you need to add servers, duplicate the application across them, and implement load balancing. For load balancing, you can use Network Load Balancing (NLB), a service included with all editions of Windows Server® 2003. With NLB, every server is an equal partner in the load-balancing relationship. They all use the same algorithm for balancing, and they all listen on a shared virtual IP address for all traffic. Based on the load-balancing algorithm, each server knows which server should be working on a given request. Each server in the cluster sends out a heartbeat to let the other servers know it is alive. When a server fails, the heartbeat for that server stops and the other servers compensate automatically.

NLB works well when you have a large number of users making fairly similar requests. However, the compensation mechanism does not work as well in a scenario when some requests create a much greater load than others. Fortunately, for that type of situation there are hardware load-balancing solutions available.

Affinity

Ultimately, the challenge of effective distribution lies in eliminating affinity. For example, when you have only one Web server, storing session data there makes perfect sense. But if you have more than one Web server, where do you keep session information?

One approach is to keep it on the Web server and use affinity. Essentially, this means the first request from a given user is load balanced, and after that, all subsequent requests from that user/session are sent to the same server as the first request. This is a simple approach, every load balancing solution supports it, and in some cases it even makes sense.

In the long run, however, affinity creates grief. Keeping session data in-process may be fast, but if the ASP.NET worker process recycles, all those sessions are dead. And worker processes recycle for a lot of reasons. Under high load, IIS might recycle the worker process of ASP.NET because it thinks it's stuck. In fact, by default in IIS 6.0 a worker process is recycled every 23 hours. You can adjust it, but either way, your users are vulnerable to losing their session data while it is in-process. When you're small, this isn't that big a deal, but as your site gets bigger and busier it becomes a more significant issue. And there's more.

If you're load balancing by IP address, one server is going to get hit by a megaproxy (like AOL) and be unable to service that entire load on its own. Plus, updating your servers with a new version of your application becomes more difficult; you must either wait for hours to let users to finish up on your site or annoy those users by knocking them out of their sessions. And your reliability becomes an issue: lose a server and you lose a lot of sessions.

Getting rid of affinity is a key goal of distribution. This requires moving session state data out of process, which means taking a performance decrease to provide a scalability increase. When you move session out of process, session data is recorded in a place where all the Web servers can access it—located either on SQL Server® or the ASP.NET State Server. This is configured in web.config.

There's also a coding effort needed to support out-of-process session. Any classes that will be stored in the Session object need to be marked with the Serializable attribute. That means that all data in the class needs to either be serializable or be marked as NonSerialized so it will be ignored. If you don't mark up your classes, when the serializer runs to store your session data out of process, you'll get errors.

Finally, moving session out of process is a great way to find out you have too much data in your session object, because you're now paying a price for shipping that great big blob of data back and forth across the network twice (once to retrieve it at the beginning of the page, once to return it at the end of the page) for every page request.

Once you nail down the Session object, go after other affinity issues like Membership and Role Manager. Each one has its own challenges for eliminating affinity. But for your ASP.NET application to really scale up, you'll need to hunt down every form of affinity you can find and eliminate it.

All of the strategies we've discussed thus far are applicable to practically every Web application that needs to scale. In fact, those strategies would apply to scaling virtually any application using any technology. Now let's look at some ASP.NET-specific techniques.

Minimizing Payload

Looking at the performance equation, you can see payload plays a significant role, especially when you're dealing with limited bandwidth. Reducing the size of your payload will improve your response time, you'll gain some scaling benefits from moving fewer bytes around, and you could even save some money on your bandwidth costs.

One of the simplest things you can do to decrease the size of your payload is to turn on compression. In IIS 6.0, you can specify whether to compress static files, dynamically generated responses (ASP.NET pages, for example), or both (see Figure 4).

Figure 4 Configuring Compression Server-Wide in IIS 6.0
Figure 4 Configuring Compression Server-Wide in IIS 6.0(Click the image for a smaller view)
Figure 4 Configuring Compression Server-Wide in IIS 6.0
Figure 4 Configuring Compression Server-Wide in IIS 6.0 (Click the image for a larger view)

IIS 6.0 compresses static files on demand, storing them in a compressed files cache that you specify. For dynamically generated responses, no copy is stored; they're compressed every time. IIS 7.0 is smarter about what it compresses, only compressing files that are used frequently.

Compression costs processor cycles, but you typically have plenty of extra processor capacity on a dedicated Web server. IIS 7.0 is further optimized, however, so that when the processor gets really busy, it will suspend compression efforts. There are also dedicated devices for doing compression independent of the Web server itself.

Another area ripe for payload reduction is ViewState. During development, it's quite easy for ViewState usage to get out of hand. Most Web controls use some ViewState, and on control-intensive pages, ViewState can grow to thousands of bytes. To reduce ViewState usage, turn it off on controls where it isn't needed. In some cases, developers will even eliminate controls to reduce the ViewState. But that's not always necessary. Most modern Web controls are sensitive to the problem of excessive ViewState and thus provide granular control over its size. There are also hardware devices that can remove and replace ViewState for you without altering your code or how your application runs.

One of the most effective technologies for reducing payload size is AJAX. Except that AJAX doesn't really reduce payload size—it simply reduces the perceived size of the payload while increasing the total number of bytes sent to the browser. Using AJAX, the parent page is smaller, so initial render times are faster. Individual elements in that page then make their own requests to the server to populate data.

Effectively, AJAX spreads the payload out over time, giving the user something to look at while other bits load. So using AJAX will improve your user experience overall, but refer back to the performance equation to measure the real costs of your effort. AJAX typically increases compute time on the client, sometimes dramatically, to the point that performance can be unacceptable.

If the AJAX round-trips to the server to populate individual elements are replacing entire page requests, you'll have a net decrease in round-trips. But in many cases, you'll find the total number of round-trips for a given user will increase. You just need to be diligent about your testing so you know whether AJAX has improved performance or reduced it.

Caching

Experts in scaling ASP.NET applications talk a great deal about caching. Fundamentally, caching is about moving data closer to the user. In a typical ASP.NET application, before any significant optimization work has been done, virtually all the data the user needs is in the database and retrieved from the database with every request. Caching changes that behavior. ASP.NET actually supports three forms of caching: page caching (also known as output caching), partial-page caching, and programmatic (also known as data) caching.

Page caching is by far the simplest form of caching. To use it, you add an @OutputCache directive to your ASP.NET page and include a rule for when to expire it. For example, you could specify that the page should be cached for 60 seconds. With that directive in place, the first request of that page will process normally, accessing the database and whatever other resources are needed to generate the page. After that, the page is held in memory on the Web server for 60 seconds and all requests during that time are served directly from memory.

Unfortunately, while this example is straightforward, it ignores a fundamental reality of page caching: virtually no ASP.NET page is so static that you can cache the entire thing for any length of time. That's where partial-page caching comes in. With partial-page caching, you're able to mark portions of an ASP.NET page as cachable so that only the parts of the page that do change regularly are computed. It's more complicated but effective.

Arguably, the most powerful (and most complex) form of caching is programmatic caching, which focuses on the objects used by the page. The most common use of programmatic caching is to store data retrieved from the database.

The most obvious problem with caching data is that the underlying data may have changed since you cached it. Expiration of caching is the biggest challenge you'll face in implementing caching of any form. But there's also memory to consider.

On a busy ASP.NET server, memory becomes a significant issue for a variety of reasons. Whenever an ASP.NET page is computed, it uses some memory. And the Microsoft® .NET Framework is set up to allocate memory very quickly but release it relatively slowly, through garbage collection. The discussion around garbage collection and .NET memory allocation is an article unto itself, one that has been written a number of times. But suffice it to say that on a busy Web server, the 2GB memory space available for your ASP.NET application is in high demand. Ideally, most of that memory usage is temporary, as it is allocated for variables and structures used in computing a Web page.

When it comes to persistent memory objects, however, like in-process session and cache objects, memory usage becomes much more problematic. And of course, these problems only surface when your application is really busy.

Consider this scenario: your Web site is hopping from some new marketing promotion, there are thousands of users hitting the site, and you're making loads of money. To maintain good response times, you're caching portions of pages and groups of data objects wherever possible. Each page request from a user consumes a bit of memory, so the bar of consumed memory keeps sliding upward. The more users, the faster that bar moves. There are also big jumps from the cache and session objects.

As the total memory used gets close to 90 percent of ASP.NET's default cache memory limit, a garbage collection event is called. The garbage collector works its way through the memory space, shuffling down persisted memory objects (like cache objects and session objects) and freeing up memory that's no longer used (the memory that was used to compute the Web pages). Freeing up unused memory is fast, but the shuffling of persisted objects is slow. So the more persisted objects you have, the harder time the garbage collector has doing its job. This type of problem can be identified in perform.exe by a high number of gen-2 collections.

And recall that while garbage collection is going on, no pages can be served by that ASP.NET server; everything is held in a queue, waiting for the garbage collection process to complete. And IIS is watching, too. If it thinks the process is taking too long and might be hung, it will recycle the worker thread. And while this frees up a lot of memory really quickly because all of those persisted memory objects are thrown out, you'll have some annoyed customers.

There is now a patch for ASP.NET that will automatically remove objects from the programmatic cache if you get low on memory, which sounds like a good idea on the surface. It's better than crashing. Just remember that every time you remove something from the cache, your code will eventually put it back.

The moment you cache something, you run the risk of it being wrong. Take, for example, a widgets database and corresponding order page. In the initial incarnation of the widget page, every rendering of that page will involve a request from the database for the number of widgets still in inventory. If you analyze those requests, you'll likely find that 99 percent of the time, you're retrieving the same number over and over again. So why not cache it?

A simple way to cache it would be over time. So you cache the inventory of the widgets for an hour. The drawback to this technique is that someone will buy a widget, then go back to the page and see that the inventory is still the same. You'll get complaints about that. But far more challenging is when someone goes to buy your widget and sees that the inventory is there, when it's actually sold out. You could build a backorder system, but either way, you're dealing with a disappointed customer.

Perhaps the problem is your expiration scheme: Time isn't good enough. You could cache the inventory count until someone buys a widget and then expire the cache object. That's more logical, but what happens if there is more than one ASP.NET server? Depending on which server you go to, you'll get different inventory counts for the widget. Consider that receiving new inventory (which adds to the count) doesn't even go through your Web application, and you have a whole new way to be wrong.

Synchronizing expirations among ASP.NET servers can be done, but you have to be careful. The amount of chatter you can generate among Web servers goes up geometrically as the number of cache objects and Web servers increases.

The impact of cache expiration on performance needs to be studied carefully, too. Under high load conditions, expiring a cache object can cause a lot of grief. For example, suppose you have an expensive query that takes 30 seconds to return from the database. You've cached that query to save that high expense because under load, that page is requested once every second.

The code for handling cache objects is pretty simple. Instead of retrieving the data from the database when needed, the application first checks to see if the cache object is populated. If it is, it uses the data from the cache object. If it is not, it executes the code to retrieve the data from the database and then populates the cache object with that data; code then continues to execute as normal.

The problem is that if you've got a query that takes 30 seconds and you're executing the page every second, in the time it takes to populate the cache item, 29 other requests will come in, all of which will attempt to populate the cache item with their own queries to the database. To solve this problem, you can add a thread lock to stop the other page executions from requesting the data from the database.

But run through the scenario again: the first request comes in, discovers the cache item is not populated, applies a lock to the code, and runs the query to populate the cache object. The second request arrives a second later while the first is still running, finds the cache object is not populated but the lock is in place, so it blocks. As does the next 28 requests. Then the first one finishes its processing, removes the lock, and continues. What happens to the other 29 requests? They're no longer blocked, so they continue executing as well. But they've already run through the check to see if the cache object is populated (and it wasn't at the time). So they'll try and grab a lock, and one will succeed and run the query again.

See the problem? Other requests that arrive after the first request completed populating the cache object will run normally, but those requests that come in while the query is running are in a tough spot. You have to write code to deal with this. If a request hits a lock, when the lock is lifted it should check again to see if the cache object is populated, as shown in Figure 5. Likely, the cache object will be populated now; that was why the lock was taken in the first place. Although it's possible that it isn't, because in the meantime some other chunk of code has expired the cache object again.

Figure 5 Checking, Locking, and Rechecking a Cache Object (Close Figure 5)
// check for cached results
object cachedResults = ctx.Cache["PersonList"];
ArrayList results = new ArrayList();

if  (cachedResults == null)
{
  // lock this section of the code
  // while we populate the list
  lock(lockObject)
  {
    // only populate if list was not populated by
    // another thread while this thread was waiting
    if (cachedResults == null)
    {
      ...
    }
  }
}

Writing caching code that works well is hard work, but the returns can be tremendous. Caching does add complexity, however, so use it judiciously. Make sure you're really going to benefit from the complexity. Always test your caching code for these complex scenarios. What happens on multiple simultaneous requests? What happens if expirations come quickly? You need to know the answers to these questions. You don't want your caching code to make your scaling problems worse.

Scaling Databases

The normal approach for scaling Web sites is to scale out, rather than up. This is largely due to ASP.NET thread and memory limitations combined with the short-term nature of Web requests.

When it comes to scaling databases, however, the normal practice is to scale up—one gigantic box, perhaps two in a cluster configuration (although only one is actually running the database at any given time). Eventually, though, in every large-scale Web application, a single database cannot handle the load. You have to scale out. It's possible; you just need to apply the same strategies applied to the Web application itself. The first step is always specialization—breaking the database into logical partitions. Those partitions could be datacentric, perhaps by region. So you would have multiple databases, each containing a portion of the whole database. One server would have East Coast data, for example, while the other has West Coast data.

Really large-scale Web applications, however, partition their databases into readers and writers (see Figure 6). The reader databases are read-only; they receive their data from the writer databases via replication. All data queries go to the reader databases, which are optimized for reading data as fast as possible. Reader databases are by their nature very distributable.

Figure 6 Distributed Database Architecture
Figure 6 Distributed Database Architecture(Click the image for a smaller view)
Figure 6 Distributed Database Architecture
Figure 6 Distributed Database Architecture (Click the image for a larger view)

All data write requests are sent to the writer databases, which are partitioned and tuned to write efficiently. Replication moves the new data from the writer to the reader databases.

The consequence of creating such specialized databases is latency: a write is now going to take time to be distributed to the reader databases. But if you can deal with the latency, the scaling potential is huge.

The Endless Scaling Effort

As long as your application continues to grow, your efforts to scale it are going to continue to grow as well. The ASP.NET techniques that work effectively for 10,000 simultaneous users aren't as effective with 100,000 users, and the rules change again with 1 million users. Of course, performance can completely depend on your application; we've seen applications that had scaling challenges with less than a thousand users!

The key to effective scaling is to measure before you cut: Use testing to be sure you're spending effort where it's needed. Test your work to be sure that you've actually made an improvement, not just a change. Even at the end of a development cycle focused on optimizing for scalability, you should know where your slowest bits are. Hopefully, however, they're fast enough for the users today, so that you can work on what your users will need tomorrow.

Richard Campbell is a Microsoft Regional Director, MVP in ASP.NET, and the co-host of .NET Rocks, the Internet Audio Talkshow for .NET Developers (dotnetrocks.com). He has spent years consulting with companies on the performance and scaling of ASP.NET and is also one of the co-founders of Strangeloop Networks.

Kent Alstad is the CTO of Strangeloop Networks (strangeloopnetworks.com) and the principal or contributing author on all of Strangeloop's pending patents. Before helping to create Strangeloop, he built and consulted on many high-performance, high-scaling ASP.NET applications.


Silverlight: Building Advanced 3D Animations with Silverlight 2.0

Silverlight

Building Advanced 3D Animations with Silverlight 2.0

Declan Brennan
Code download available at: SilverlightAnimation2008_04.exe (228KB)
This article discusses:
  • XAML basics
  • Building elements in XAML
  • How to fold a polyhedron
  • Emulating DirectX math
This article uses the following technologies:
Silverlight
Contents
Using XAML
Some XAML Examples
Tips for Working with XAML
How to Fold a Polyhedron
Emulating DirectX Math
More to Explore

I
f, by some incredible mischance, all the publicity in the past few months about SilverlightTM has passed you by, let me bring you up to date: Silverlight is a new cross-browser plug-in from Microsoft that brings the power of the Microsoft® .NET Framework to bear on an area that was previously reserved for Flash or Java Applets. Silverlight has a wealth of useful features out of the box. It supports a lean-and-mean version of the .NET Framework 3.5 that includes, among other things, XML and Extensible Application Markup Language (XAML), generic collections, Web services, and LINQ. Silverlight also supports a range of .NET-compliant languages, but here I'll be sticking to C#.

I think the best way to come to grips with a new technology is to have a little fun with it. So when the Silverlight 1.1 alpha was released and I saw the exciting presentation by Tim Sneath at the IMT conference in Dublin, I decided to throw together a little educational application demonstrating how various 3D shapes (called polyhedra) can be assembled by folding a flat template. Silverlight doesn't support 3D by default, so this involved building an emulation of the DirectX® math libraries to do the 3D stuff.

A polyhedron is a three-dimensional object with flat faces. This Silverlight sample explores those regular and semi-regular polyhedra called the Platonic and Archimedian polyhedra. The faces of these polyhedra are all regular polygons—their sides are all the same length—such as equilateral triangles or squares. They are also convex, meaning they have no pieces that stick out. As you might guess from the ancient Greek names, these objects have fascinated humanity for a long time. If you're interested, you can find out a lot more about them on George Hart's Web site: georgehart.com/virtual-polyhedra/vp.html.

You can see a demo of the finished application in Figure 1 or at picturespice.com/ps/Polyhedra/default.html. Basically, the application allows you to select a shape (a polyhedron) by moving the mouse over it. You are then presented with some information about your selection in the top-right corner of the window and also an animation of a flat template folding to form your chosen polyhedron. Finally, if you click the Cycle button, the program automatically cycles through each of the shapes in turn.

Figure 1 Silverlight Demonstration of Polyhedra
Figure 1 Silverlight Demonstration of Polyhedra(Click the image for a smaller view)
Figure 1 Silverlight Demonstration of Polyhedra
Figure 1 Silverlight Demonstration of Polyhedra (Click the image for a larger view)

Using XAML

Like many Silverlight applications, Polyhedra makes heavy use of XAML, which is a content definition language—a sort of HTML equivalent, but more flexible. Continuing the analogy, while it is possible to create an HTML page using only the HTML document object model (DOM), this is rarely a sensible way to produce content—it's often time-consuming to code, and it produces pages that are slow to initialize. It's nearly always best to keep as much of the page as possible as HTML markup and then augment this using JavaScript and the DOM where flexibility is needed.

A very similar approach applies to XAML. The fastest way to get content together is to use XAML markup as much as possible and augment this where necessary using a .NET-compliant language such as C# and the Silverlight Media API. The XAML may be hand coded, produced by a design package such as Expression BlendTM, generated from a program that is run during the development process, or even dynamically generated on the server. This can require a change of mind-set. It's all too easy as a C# programmer to end up coding functionality in your natural environment that is best left to XAML.

Anything but a very brief taste of XAML would be beyond the scope of this article. However, Charles Petzold covers XAML in great detail in his book Applications= Code+Markup.

Here's the XAML equivalent of the "Hello World" example that we've all come to expect when learning new languages:

<UserControl x:Class="Polyhedra.Page"
  xmlns="http://schemas.microsoft.com/client/2007"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  Width="400" Height="300">
  <Grid x:Name="LayoutRoot" Background="White">
    <TextBlock>Hello World</TextBlock>
  </Grid>
</UserControl>

The root element is a UserControl. This contains a Grid, which, in turn, contains a TextBlock element with the "Hello World" text.

Have a quick look at the UserControl attributes. Without going into too much detail, this defines the equivalent of a codebehind class for the XAML. This class is instantiated at the same time that the XAML is parsed and loaded. You can perform various bits of initialization in the constructor, but to get more elaborate behavior it is often necessary to employ event handlers. This is a key feature of Silverlight. Event handlers can be attached to various XAML objects and implemented in the .NET-compliant language of your choice—a language that, unlike JavaScript, is compiled and therefore opens up all sorts of possibilities that would be otherwise impractical.

Returning to the HTML analogy, elements are often grouped inside various DIVs to arrange their location on the page. Similarly, in XAML, the shapes are grouped inside Canvases (or other elements such as Grids that are types of Canvas). Just as DIVs are often nested in HTML, so Canvases can be nested in XAML. Most elements in HTML are rectangular. However, XAML supports a whole range of shapes including TextBlock, Rectangle, Polygon, Ellipse, and the very flexible Path, which allows for user-defined shapes. Elements in HTML are identified with an ID attribute, and the equivalent attribute to identify an element in XAML is x:Name, where x is the alias for the XAML namespace.

XAML does a lot more than just produce static pages. One of its most powerful features is the use of storyboards as a way to animate changes to the initial UI specified by the XAML. (Something similar, called HTML+Time, was added to HTML in Internet Explorer® 5.0) An example might be to animate a change to the color, visibility, or transparency of an object. If used in conjunction with transforms, storyboards can also rotate, scale, or move objects.

Storyboards can generate all sorts of animation effects with little or no conventional code. When used in conjunction with triggers, it is, in theory, possible to start various animations automatically when an event such as MouseEnter occurs on an object. Alas, as of the March 2008 Silverlight 2.0 Beta,, the only event handled by a trigger is Loaded. For other events, a small amount of plumbing code is required in the form of an event handler. I expect this situation will change soon, if it hasn't already done so.

One nice feature of storyboards in Silverlight is that they are time-based rather than frame-based. Using multiple independent storyboards tied to different times and events can make a complex behavior simple to implement. After all, the real world doesn't operate in frames—it's a hangover from the way that moving pictures were achieved on celluloid. If independent objects are implemented with independent behaviors, it makes for a far simpler life.

Some XAML Examples

The folding animation in the middle of the Polyhedra application is generated using C# code. However, the bulk of the remainder is defined in XAML. This includes the circle of sample polyhedra, the way the currently selected polyhedron is emphasized in a variety of ways, and the Cycle button, which shows that it is activated by animating a rotating arrow.

Let's look at a couple of excerpts from Page.xaml to see how these animations are achieved, starting with a look at the Cycle button, shown in Figure 2. Have a look at the Path named Cycle. The data attribute specifies a series of operations including M for move, A for arc, and L for line. As this is a pretty simple glyph, I just drew the shape I wanted on a sheet of paper and manually worked out the operations required. In most cases, you'll be better off using a tool for this such as Expression Design.

Figure 2 Cycle Button (Close Figure 2)
<Canvas x:Name="CycleButton"
  Canvas.Top="250"
  MouseLeftButtonDown="CycleButtonLeftMouseDown" >
  <Path x:Name="Cycle" Stroke="#000033"
    Fill="#FFB47C0D" Canvas.Left="10" Canvas.Top="10"
    Data="M 25,35 A 10,10 180 1 0 25,15 L 25,20 L 12.5,10 L 25,0 L 25,
         5 A 10,10 180 1 1 25,45 Z"
    Width="50" Height="50">
    <Path.RenderTransform>
      <RotateTransform x:Name="CycleRotate" Angle="0"
        CenterX="25" CenterY="25"/>
    </Path.RenderTransform>
    <Path.Resources>
      <Storyboard x:Name="CycleLatched">
        <DoubleAnimation Storyboard.TargetName="CycleRotate"
          Storyboard.TargetProperty="Angle" From="360" To="0"
          Duration="00:00:02" RepeatBehavior="Forever"/>
      </Storyboard>
    </Path.Resources>
  </Path>
  <TextBlock x:Name="CycleCaption" Canvas.Left="65" Canvas.Top="10"
    Foreground="#FFB47C0D" FontSize="30" FontWeight="Bold"
        Text="Cycle" />
  <Rectangle Width="200" Height="70" RadiusX="30" RadiusY="30"
    Stroke="#FFB47C0D" StrokeThickness="4" Fill="Transparent"/>
</Canvas>

This Path also has a RotateTransform named CycleRotate. Initially, this transform does nothing, as its angle is set to 0. However, there is a Storyboard named CycleLatched that, when active, continuously changes the angle in small increments, which causes the arrow to rotate.

Figure 3 shows how one of the polyhedron samples is defined. At the bottom of this XAML, you can see that the sample consists of the four Polygons that define the tetrahedron. They are inside their own Canvas, called Model0, and are surrounded by a ring or Ellipse called Ring0 that is initially invisible. There are two Storyboard animations defined, one for MouseEnter and one for MouseLeave. The MouseEnter animation makes the ring visible immediately, inflates the size of the Model0 canvas, and makes it more opaque over a period of .7 seconds. The MouseLeave animation reverses each of these changes.

Figure 3 Polyhedron Sample (Close Figure 3)
<Canvas x:Name="Canvas0" Width="116.376" Height="116.376"
  Canvas.Left="671.812" Canvas.Top="341.812"
  MouseEnter="TriggerMouseEnter" MouseLeave="TriggerMouseLeave">
  <Canvas.Resources>
    <Storyboard x:Name="Canvas0MouseEnter">
      <DoubleAnimation Duration="00:00:00" Storyboard.TargetName="Ring0"
        Storyboard.TargetProperty="Opacity" From="0.0" To="1.0" />
      <DoubleAnimation Duration="00:00:00.7"
        Storyboard.TargetName="Model0"
        Storyboard.TargetProperty="Opacity" From="0.7" To="1.0" />
      <DoubleAnimation Duration="00:00:00.7"
        Storyboard.TargetName="Model0Scale"
        Storyboard.TargetProperty="ScaleX" From="0.7" To="1.0" />
      <DoubleAnimation Duration="00:00:00.7"
        Storyboard.TargetName="Model0Scale"
        Storyboard.TargetProperty="ScaleY" From="0.7" To="1.0" />
    </Storyboard>
    <Storyboard x:Name="Canvas0MouseLeave">
      <DoubleAnimation Duration="00:00:00" Storyboard.TargetName="Ring0"
        Storyboard.TargetProperty="Opacity" From="1.0" To="0.0" />
      <DoubleAnimation Duration="00:00:00.7"
        Storyboard.TargetName="Model0"
        Storyboard.TargetProperty="Opacity" From="1.0" To="0.7" />
      <DoubleAnimation Duration="00:00:00.7"
        Storyboard.TargetName="Model0Scale"
        Storyboard.TargetProperty="ScaleX" From="1.0" To="0.7" />
      <DoubleAnimation Duration="00:00:00.7"
        Storyboard.TargetName="Model0Scale"
        Storyboard.TargetProperty="ScaleY" From="1.0" To="0.7" />
    </Storyboard>
  </Canvas.Resources>
  <Canvas x:Name="Model0" Opacity="0.7"
    Width="116.376" Height="116.376" >
    <Canvas.RenderTransform>
      <ScaleTransform x:Name="Model0Scale" CenterX="58.188"
        CenterY="58.188" ScaleX="0.7" ScaleY="0.7"/>
    </Canvas.RenderTransform>

    <Polygon Canvas.ZIndex="-12545653" Fill="#FFCC0000"
      Stroke="#000000" StrokeThickness="1"
      Points="97.566,74.278 43.052,40.971 33.836,99.937 "/>
    <Polygon Canvas.ZIndex="-11948057" Fill="#FFCC0000"
      Stroke="#000000" StrokeThickness="1"
      Points="43.052,40.971 97.566,74.278 58.188,37.662 "/>
    <Polygon Canvas.ZIndex="-11309683" Fill="#FFCC0000"
      Stroke="#000000" StrokeThickness="1"
      Points="33.836,99.937 43.052,40.971 58.188,37.662 "/>
    <Polygon Canvas.ZIndex="-10481036" Fill="#FFCC0000"
      Stroke="#000000" StrokeThickness="1"
      Points="97.566,74.278 33.836,99.937 58.188,37.662 "/>
  </Canvas>
  <Ellipse x:Name="Ring0" Opacity="0" Stroke="#FFB47C0D"
    StrokeThickness="4" Width="116.376" Height="116.376" />
</Canvas>

Because these storyboards can operate independently, everything happens as you would expect regardless of how quickly the user moves the mouse. Typically, a MouseLeave animation for one sample will be taking place at the same time as the MouseEnter animation for the newly selected sample. However, you don't have to worry about it, as Silverlight will automatically handle running multiple active storyboards in parallel.

As a slight aside, you may be wondering how the corners of each of the four Polygons for the tetrahedron sample were calculated, not to mention the often far larger number of Polygons for each of the other samples in the application. I'm a big believer in the lazy approach of never doing anything yourself that a computer can do quicker. So all I did was use a modified version of Polyhedra during the production process. This version performed one frame of the central animation for each of the samples in turn—the frame with the fully closed shape. I placed each of the resulting Polygon groups in a set of Canvases that were equally spaced around a circle and streamed the whole lot to a file for use by the main program.

Laying objects around a circle involves stepping the angle up in equal increments of 2*PI/NumSamples and then specifying the center of each sample using a coordinate of x= Radius*Cos(Angle), y=Radius*Sin(Angle). Also, because the canvas positioning objects use Left and Top properties, I needed to offset the center by half the width and half the height, respectively.

Tips for Working with XAML

As you can see, it's possible to achieve quite a rich UI with XAML and almost no additional code. Even for a large file like Page.xaml, initialization is surprisingly quick. But before I move on, here are a few tips based on my experiences with the current implementation (March 2008 Beta) of Silverlight 2.0.

As I've already mentioned, currently triggers can only start a storyboard automatically (via the Begin method) for the Loaded event. For other events, a small amount of plumbing code is required in the form of an event handler like this:

public void MouseEnterHandler(
  object o, EventArgs e) {
  this.MouseEnterStoryBoard.Begin();
}

If you adopt a naming convention for your storyboards, such as the object name followed by the event name, you can cut down greatly on the number of separate event handlers that you need to code. For example, this method is used in Polyhedra as a shared event handler for all the samples in the circle:

public void MouseEnterHandler(
  object o, EventArgs e) {
  this.triggerStoryboard(o,"MouseEnter");
}
private bool triggerStoryboard(
  object o, string eventType) {
  Canvas el = o as Canvas;

  string name= el.GetValue(NameProperty) as String;
  Storyboard sb = el.FindName(name + eventType) as Storyboard;
  if (sb != null)
    sb.Begin();
    return (sb != null);
}

With the Silverlight 2.0 Beta, the main initialization (calling InitComponents) has moved from a Loaded event handler to the constructor for the codebehind object. This is more elegant, but beware that not everything is possible in the constructor. For example, it's not possible to call Begin or Pause on a storyboard here, so this will still need to be done in an event handler.

As I discovered from Andy Beaulieu's asteroid-blasting Silverlight Rocks! sample (www.andybeaulieu.com/Home/tabid/67/EntryID/73/default.aspx), a good way to achieve a code-based animation is to use a storyboard set to a short time period and have a Completed event handler that does a frame of animation and then restarts the storyboard:

public Page() { // Constructor for "code-behind"
  // Required to initialize variables
  InitializeComponent();
  this.animationTimer.Completed +=
    new EventHandler(animationTimer_Completed);
}

void animationTimer_Completed(object sender, EventArgs e) {
  [ Do a frame of animation ]
  this.animationTimer.Begin();
}

The September Refresh of the Silverlight 2.0 Alpha changed the requirements for storyboards so that an animation must now have a target even if it's not used:

<Canvas.Resources>
  <Storyboard x:Name="animationTimer">
    <DoubleAnimation Duration="00:00:00.01"
      Storyboard.TargetName="bogusTimerTarget"
      Storyboard.TargetProperty="Width" />
  </Storyboard>
</Canvas.Resources>
<Canvas Name="bogusTimerTarget">
</Canvas>

Don't try to have lots of separate Silverlight controls on the same HTML page. My first implementation of Polyhedra used a separate control for each sample in the circle, and it was a real memory hog. This may, in some cases, mean moving content from HTML to XAML to cut down on the number of controls you are using.

One of the advantages of XAML is that it offloads a lot of the routine stuff in the UI, allowing you to concentrate on the creative problem domain code. In this example, the problem domain involves the folding of templates to form the 3D shapes, which leads us to the next section.

How to Fold a Polyhedron

I suspect that Charles Petzold's book, mentioned earlier, is an adaptation of a far older, pre-object-oriented book by Niklaus Wirth called Algorithms+Data Structures=Programs. Even after all these years, it remains one of the most influential books that I have ever read, and, despite all the changes in languages and paradigms that have taken place since then, it is still highly relevant. The basic tenet of the book is an approach to development that consists of identifying what data structures can best model your problem, then identifying what algorithms can process or modify these data structures. When tackling a non-standard program, this is an approach that I often adopt.

I played around with various approaches before settling on the one finally used in Polyhedra. I wanted to see how little information I could practically start with to achieve the folding animation. It turns out that, since all the sides are the same length, it's almost possible to do everything just from the knowledge of which faces are connected to which in a given polyhedron. This suggests a graph data structure. Before reaching the final XAML output, a set of algorithms are used to process the graph through two separate tree data structures. All of this will be explained later.

Although Windows® Presentation Foundation (WPF) can support 3D in XAML, Silverlight only supports 2D by default because cross-browser compatibility is far easier to achieve when you don't have to worry about what Graphics Processing Unit (GPU) hardware is in the machine. Of course, when you get down to it, 3D on a computer screen is an illusion. Regardless of what manipulations take place, in the end a set of ordinary 2D polygons is being displayed on the monitor surface. If you are willing to do the 3D manipulations in your own code to calculate the coordinates of these polygons, a non-hardware-accelerated 3D rendering is possible. Provided you restrict yourself to a few dozen polygons, the performance is acceptable. (Subjectively, frame performance has improved moving from the Alpha to the Beta.)

When a shape is selected, a two-dimensional unfolded template called a Net (nothing to do with .NET) is built. This is done in two phases, as illustrated in Figure 4. First a graph is built in memory with one GraphNode corresponding to each face in the shape (see Graph.cs). This graph is built by streaming in a set of connectivity information (indicating which face is connected to which faces) from one of the .shp resources that are embedded in the application assembly. For example, here are the contents of cube.shp:

Figure 4 Building the Shape, from Graph to XAML
Figure 4 Building the Shape, from Graph to XAML(Click the image for a smaller view)
Figure 4 Building the Shape, from Graph to XAML
Figure 4 Building the Shape, from Graph to XAML (Click the image for a larger view)

1:3,4,5,2
2:1,5,6,3
3:2,6,4,1
4:3,6,5,1
5:1,4,6,2
6:2,5,4,3

Any GraphNode in the graph can then be chosen as the first node from which to start building the Net (see Net.cs in the code download for details). A two-dimensional FlatFace is then laid out with the same number of sides as the GraphNode has neighbors. Any neighboring GraphNode is then chosen to repeat the process, laying out the next FlatFace so it shares a side with the current one. Each GraphNode is only visited once, and the process continues until all GraphNodes have been visited, resulting in a tree structure of FlatFaces.

In each frame of the 3D animation, a three-dimensional polyhedron (possibly partially closed) is built from the Net (see Polyhedron.cs later in this article for the details of this operation). This involves copying the tree of FlatFaces into a tree of Faces replacing 2D corner points with 3D points. (Figure 5 shows the two coordinate systems. Because I want to start in the horizontal plane, Y should start as zero. So I map from 2D to 3D by setting X=X, Y=0, and Z=Y.)

Figure 5 Transforming Points from 2D to 3D
Figure 5 Transforming Points from 2D to 3D(Click the image for a smaller view)
Figure 5 Transforming Points from 2D to 3D
Figure 5 Transforming Points from 2D to 3D (Click the image for a larger view)

Finally, recursion is used to visit each join in the tree of Faces to perform a fold. The amount of the fold is a fraction of the angle (called a dihedral angle) that the faces will have to each other when the shape is fully closed. It is, in theory, possible to calculate the dihedral angles directly using just the connectivity information in the .shp file.

This is fairly easy for special cases such as any three faces meeting at a corner or any number of faces of the same type meeting at a corner. However the general case is somewhat more difficult, so I decided to store these angles in separate .dihedrals resources. For example, here's an excerpt from cube.dihedrals:

1,3:1.5707965056551
1,4:1.57079661280793
1,5:1.57079614793469
1,2:1.57079604078186
2,1:1.57079604078186
2,5:1.57079628466395
2,6:1.57079661280793
2,3:1.57079636892585
3,2:1.57079636892585
3,6:1.57079614793469
3,4:1.57079628466395
...

The minute differences in these numbers, which should all be PI/2 in this case, are merely an artifact of the way they were calculated.

The final process involves a series of transformations to project a view of the polyhedron onto the monitor as a set of polygons. You may have noticed that, during the folding animation, the polyhedron is also rotated about a vertical axis. To achieve this, I translate the pivot point to the origin, perform the rotation, and then translate to the viewpoint. Then I do a perspective transformation to ensure that far away objects (on the Z axis) appear smaller. (I'll explain a bit more about how to do transformations in the next section.)

Finally, a set of XAML polygons is generated corresponding to the projection of the 3D shape onto the monitor's surface. Silverlight is forced to draw these polygons in back-to-front order by setting the XAML ZIndex property for each Polygon using the Z coordinate of the center of the 3D polygon (scaled to squeeze a float into an int). This is a fairly simplistic way of achieving 3D that will only work if the polygons behave themselves by, for example, not intersecting—something I can get away with in this sample. Smarter forms of 3D involve mechanisms such as depth buffers. As these require GPU access, they are out of scope for the Silverlight .NET sandbox.

As a last touch, XAML allows Polygons to be partially transparent (using the Opacity property), which gives the artistic effect of a partially see-through polyhedron. And that, as they say, is all there is to it.

Emulating DirectX Math

I'm going to touch lightly on the math used to achieve the animation. If you want to go into more detail, there are many sites on the Web that will help, including wikipedia.com, euclideanspace.com, mathworld.wolfram.com, gamasutra.com, and gamedev.net.

I have already introduced some 2D transformations, such as RotationTransform and ScaleTransform, in the XAML samples shown earlier. In these, Silverlight does all the work for you. If you use XAML in a WPF environment, you also have access to 3D transformations, but these are not available in Silverlight. WPF relies upon DirectX to do the low-level work for it. This makes sense, as DirectX has a nice set of classes for doing the math required for 3D transforms. To do some of the same transformations in Silverlight, I've produced a DirectX Math emulation of many of these classes so that they can be used in the Silverlight sandbox.

If you have any experience coding in Direct3D®, you'll be quite at home with the Microsoft implementations of Vector2, Vector3, Matrix, and the like (see Figure 6). If not, I'll give a very brief and very non-rigorous overview and some examples.

Figure 6 Emulated DirectX Math Classes
Figure 6 Emulated DirectX Math Classes(Click the image for a smaller view)
Figure 6 Emulated DirectX Math Classes
Figure 6 Emulated DirectX Math Classes (Click the image for a larger view)

Locating a point in 2D requires two coordinates: X and Y. In 3D it requires three coordinates: X, Y, and Z. Although, strictly speaking, points are not vectors, 2D points can be stored in a Vector2 object and 3D points in a Vector3 object. (Vectors are actually entities with a magnitude and a direction, which you can illustrate by having an arrow that starts at the origin and ends at the point. In a sense, you can think of all vectors as starting from the origin.)

Shapes in three dimensions can be identified as a set of faces. Faces, in turn, are identified as a set of points corresponding to corners. Nearly everything you might want to do involves transforming these points to another set of points by applying one or more operations such as rotation, translation (or moving), or scaling. Nearly every operation that you might want to do can be represented by a 4-by-4 grid of numbers called a matrix. (Matrices in general can actually have any number of rows and columns, but we're only interested in the special ones used to manipulate 3D points with what are called homogeneous coordinates.) These operations are referred to as linear transformations because they don't (overly) distort the shape of the object they are applied to.

Very usefully, there is a way of combining two such matrices into one resulting matrix that has the same effect. This operation is called matrix multiplication and can be repeated several times. This means you can chain a whole set of transformations together into a single matrix with the same effect. This is far more efficient than applying each transformation separately. There's no need to worry about how matrix multiplication is implemented or how it does its transformation magic. Just use it as a tool.

To transform a source point into a target point, there is also a multiplication operation defined between a matrix and a vector object. To avoid confusion (with matrix-matrix multiplication), this is implemented in Vector3 using the TransformCoordinate method.

A couple of little code snippets should make things more clear. First, the following code in polyhedron.cs is used to fold two faces along their common side:

Vector3 axis = axisTo - axisFrom;
Matrix foldTransform =
  Matrix.Translation(-axisFrom) *
  Matrix.RotationAxis(axis, proportion * _dihedralAngle) *
  Matrix.Translation(axisFrom);
Vector3[] p= Vector3.TransformCoordinate(
  face.Points,foldTransform);

Here the common side is defined as a line from axisFrom to axisTo. Rotations are always defined about a line through the origin, so, before I rotate, I need to move one point of the side to the origin and then move it back afterward. To do this, I'm chaining together three linear transformations to build the matrix that I use to transform the face coordinate: move to the origin, rotate, and then move back again.

There's one other little trick here: the vector representing the axis of rotation also has to start from the origin, so I subtract axisFrom, moving (axisFrom,axisTo) to (origin, axisTo-AxisFrom).

If your head is not too full of math now, let's try one more example (based on projector.cs) that is used to transform the polygons of the polyhedron into what you would expect to see from the screen viewpoint:

Matrix projection =
  Matrix.Translation(-pivot) *
  Matrix.RotationY(yaw) *
  Matrix.RotationX(angle) *
  Matrix.Translation(viewPoint) *
  Geometry.Perspective(5);
Vector3[] p= Vector3.TransformCoordinate(face.Points,foldTransform);

Remember that the shape is being rotated in the horizontal plane while it is being folded. To do this, a point on the pivot axis is moved to the origin, then a rotation takes place about the vertical (Y) axis through the yaw angle. (If you've flown an aircraft in reality or simulation, you will have come across the other two possible independent rotations called pitch and roll.)

This time, instead of moving back to the start position, a move takes place to the viewpoint. Finally, a perspective transformation takes place to shrink the stuff that is far away on the Z axis. There's a lot going on here, yet it can all be combined into a single 4-by-4 matrix, which is pretty impressive.

More to Explore

Although not used in Polyhedra, quaternions are also available in the emulation library. They are a good way to combine a set of 3D rotations into a single rotation. They are also great for smoothly changing from one orientation to another. They were discovered by Sir William Rowan Hamilton (from my hometown of Dublin, Ireland) while walking along the street, and he promptly scribbled the equation on the side of the Broome Bridge so he wouldn't forget it (go to www.maths.tcd.ie/pub/HistMath/People/Hamilton/Quaternions.html). I'd like to claim that all our graffiti is equally erudite, but I don't suppose you'd believe me.

Quaternions also avoid a problem called gimbal lock. This played an important part in gyroscope navigation during the Apollo moon missions and was featured in a movie about the Apollo 13 mission.

There's plenty of other stuff that I haven't gone into, such as vector dot products and cross products, but hopefully I've given you a flavor of what's possible with the DirectX Math emulation. I originally developed this emulation to do transformations on a hosted server running ASP.NET where I couldn't install DirectX. You may find other applications for it in this type of environment.

Polyhedron should give you a taste of how Silverlight provides a useful, extensible tool for Web UIs that really grab the user's attention and get information across in fast and memorable ways. You can build on your existing .NET experience in all sorts of creative ways, and, unlike JavaScript, you don't have to be obsessive about the number of lines of code being executed on the client.

Declan Brennan is old enough to remember the first microprocessor, and he still hasn't gotten used to having his own personal genie in a bottle. He can't believe his luck at being in a world limited not by technology but only by imagination. Learn more about Declan at declan.brennan.name.


Interview++: Bjarne Stroustrup on the Evolution of Languages

Interview++

Bjarne Stroustrup on the Evolution of Languages

Howard Dierking
Contents
Thoughts on Language
Language Trends
Methodology and Best Practices
Looking toward the Future
Books and Phones

Every once in a while, an evolutionary leap rapidly advances and reshapes the entire field of engineering. Such a leap occurred in software development with the introduction of the C++ programming language. This leap was not inherent in the language itself: object-oriented languages such as Simula67 and Smalltalk existed prior to C++. But because C++ was built on top of the C programming language (and could compile existing C programs), it was able to bring the abstractions of object-oriented thinking into the mainstream.

C++ has inspired a great deal of thought surrounding software design and development, from design patterns to meta-programming. And because of its portability among hardware platforms and its lower-level expressiveness, C++ will assuredly be essential in a world of faster, smaller hardware.

I recently had the pleasure of talking to Bjarne Stroustrup, the creator of C++, about a host of topics, from his thoughts on languages to general industry trends to his own personal reading list. Many of the questions asked were suggested by readers via my blog, so thanks to everyone who contributed questions. And of course, thanks to Bjarne.

Thoughts on Language

Howard Dierking Why do programming languages connect with people on such a deep level—such that communities of language zealots form?

Bjarne Stroustrup You should ask a psychologist, a sociologist, or maybe even an economist, rather than a computer scientist! My guess is that the languages we use to express our ideas become a part of ourselves so that if you know only one language, proponents of other languages could appear personally threatening. In that case, the solution seems to be to know more languages well. I don't think you can be a professional in the software field knowing only one programming language. There may also be an economic reason: while fundamental understanding transcends programming language boundaries, many practical skills don't. So if I know only language X and its tool sets, and you argue for language Y and its toolsets, you are threatening my livelihood. Again, the solution seems to be to know several languages and toolsets (and to be solid on fundamentals). Unfortunately, my suggested solutions do not take into account that most people have very little time left after doing all that they feel necessary just to manage. That's no excuse for zealotry, though.

HD What should be the role of the IDE in software development? How should the IDE support a language?

BS I'm not a heavy IDE user. I appreciate a responsive IDE editor with understanding of my language, but I also want to be able to work without an IDE. I might have a different opinion if a good IDE was universally available—in reality a part of the language or vice versa (see

Figure 1 The IDE Designer as a Language(Click the image for a smaller view)
Figure 1 The IDE Designer as a Language
Figure 1 The IDE Designer as a Language (Click the image for a larger view)

HD Do you see noise or readability as a problem in today's general purpose languages? If so, what is the solution?

BS A simpler syntax would be nice, but I suspect that what most people grumble about when they talk about readability is not so much the actual text as the complexity of what is being expressed. Too many people expect to walk up to any program written in any language and—with only a bit of help from an online support facility—understand all the constructs used to express the program and all the logic of the program itself. Compare that to the way we look at natural languages and use them. Would you expect to understand a Shakespearean sonnet without background information? How about Beowulf in the original Old English? Maybe we expect too much from our programming languages. Any language that can express all that is needed for a wide range of application areas could be considered unnecessarily complex for any given application, but it must cope with an essentially unbounded set of applications. Domain-specific languages can help in specific cases, but now we have to deal with the complexities of many languages and their interactions.

HD How should a general-purpose language support application design ideas such as component programming and service programming?

BS A general-purpose language should support the writing of libraries that express general and application-specific notions, support tool building, and provide the glue needed to connect different parts of an application. For that, the language needs flexibility, an expressive type system, good basic performance, and long-term stability.

HD Is multiple dispatch a good thing?

BS Yes. A conventional single-dispatch object-oriented programming language (such as Simula, C++, Smalltalk, Java, and C#) cannot elegantly express simple operations, such as multiplying numbers or finding the intersection of two shapes, where the exact types are not known until run time. The resulting code (relying on double dispatch, the visitor pattern, and so on) is slow and not as maintainable as we'd like. Languages that support multiple dispatch at run time (like Dylan and CLOS) do better and languages (such as C++) that support it at compile time can sometimes help a bit. Last year, together with some of my students, I published a research paper on how to cleanly add multiple dispatch to C++. The resulting code for multiple dispatch use-cases is shorter, simpler, uses less memory, and runs faster than all workarounds we have seen (see

Figure 2 Multiple Dispatch via Multiple Inheritance(Click the image for a smaller view)
Figure 2 Multiple Dispatch via Multiple Inheritance
Figure 2 Multiple Dispatch via Multiple Inheritance (Click the image for a larger view)

HD What are your thoughts on covariance/contravariance in C++ today? Do you expect the current behavior to change in future language versions?

BS Not really. In that area, I think C++ does what it should. Are you thinking about examples such as converting a vector<Apple> to a vector<Fruit>? That's terribly unsafe unless the vector<Fruit> is immutable (or you could add an Orange to it). I don't think implicit run-time type checking is a good approach to such problems.

HD What are your thoughts on making message passing a key language feature, as opposed to method calling?

BS I like explicit message passing, but I have only used it a long time ago and in the context of distributed systems. Realistically, for that to work on a large scale, we'd have to do some serious work on language and tool support for message passing. I don't think that has been done, but I could be wrong. Lots of problems associated with sharing and locking go away if you rely on messages and message queues. I'd love to see a standard C++ library for that, and I just might get it (in a couple of years).

HD Is there an inherent conflict for general-purpose languages in simultaneously supporting a broad, diverse audience and encouraging improvements in code expressiveness and design elegance? What is the role of the language in supporting the latter?

BS I guess there is in the sense that you can approach elegance through specialization. Also, if you have a small audience (user community), you can cater to its tastes (only) and it is possible to depart from conventional notations and concepts if you somehow control your user community (saying something like "you need to read up on type theory to use this"). A general-purpose language is constrained not only by the need to support a wide variety of uses, but also by the need to be teachable to large groups of people with a variety of assumptions and educational backgrounds (the basics have to be usable by a high-school student with a poor teacher).

So I think a general-purpose language can encourage elegance as long as elegant code can be expressed in it. In the case of C++, very elegant code can be written. By being part of a general-purpose language and widely available, such examples can get into the hands of millions and into articles and textbooks that can be read by millions. No specialized language—however elegant—has those options.

HD Have we placed too much emphasis on architecture?

BS No, at least not the way I would define architecture. On the contrary, there is too little emphasis on architecture and too much poor coding with little understanding of structural principles. I suspect a major problem with architecture is that many programmers have only a vague idea of what makes good code good. Being able to recognize it when you see it is not enough. Having rules for what not to do is not enough. We need articulated prescriptive rules.

HD Has the open source community helped, hurt, or made no difference in software quality, design, and professionalism?

BS That's a really hard question. I have seen cases where it helped (increased the quality of code and the degree of professionalism of people involved), cases where it hurt (taught really bad habits and attitudes), and many cases where I couldn't tell.

I have no way of guessing what the effects on the community at large have been or what would have happened had the degree of open source work been greater or lesser. The community is simply too large for me to guess.

Language Trends

HD Do you see a serious shift towards dynamic languages in the near future?

BS Not really. I think people are comparing apples and oranges too often. I don't think we have a choice between static and dynamic languages in general and furthermore I don't think languages cleanly fit into those two categories: most if not all dynamic languages have aspects that are statically determined, and all the major static languages can do things that require run-time determination of the meaning of values. There are fashions, of course, and I can't guess about those, but I think that many real-world language choices are rationally made based on the requirements of an application, an application area, and/or the skills of the available developers. For example, I'd not try to implement a Java runtime in (say) Ruby or express a heavily interactive simulation language as C++ (as opposed to its implementation).

HD Would you see it as a good thing to ultimately remove all existence of void*/variant/object/etc? (Figure 3 shows a good example.)

Figure 3 Void** in COM QueryInterface Method (Close Figure 3)
STDMETHODIMP
CoCalc::QueryInterface(REFIID riid, void** ppv)
{
   if (riid == IID_IFirst || riid == IID_IUnknown)
    *ppv = static_cast<IFirst*>(this);
  else if (riid == IID_ISecond)
    *ppv = static_cast<ISecond>(this);
  else
  {
    *ppv = 0;
    return E_NOINTERFACE;
  }
  AddRef();
  return S_OK;
}

BS I guess that in principle it would be a good idea to get rid of all the catchall options, but in reality we could achieve that only by having a full classification of everything we wanted to express so that we could always be more specific. For example, I have never seen a real function that could do something to every object; if you do something, you always make some assumption about what you are manipulating (a pure forwarding function is the closest I can think of as a counter example). The current work on concepts in the C++ world (constraints/requirements on generic algorithms) will be a help here, but I don't see us being able to do without some truly general mechanism for expressing "something, but I really don't know what" for dealing with unanticipated needs.

HD With the trend shifting back to loosely typed languages, should we start considering Hungarian notation again?

BS I'm not sure that there is such a trend, though there probably is an increase in the fraction of the total work that is suitable for loosely typed languages. In other words, the use of statically typed languages could still be growing (I think so) while the use of loosely typed languages is growing faster. And no, don't use Hungarian. Hungarian is an awful idea. The source code should reflect the meaning of a program, not simulate a type system. If you really, really feel the need for Hungarian, you are probably using a language that is unsuitable for your application.

HD In 2000 you gave a presentation called "C++: A New Language for a New Millennium" and introduced the concept of a Wrap<T>. This is basically aspect-oriented programming (AOP). What do you think of AOP (generally) and its formalization of the Wrap<T> pattern (pointcuts, advice, and so on)?

BS I haven't spent enough time on AOP to give a solid answer. I like composition (especially non-intrusive) to be supported in a language (as opposed to both "not supported" and "supported by a tool"). I worry about extra-linguistic tools and about non-standard tool chains. C++ templates have been amazingly successful at non-intrusive composition—just look at the Standard Template Library (STL) and some of the uses in embedded systems programming. It is a real strength to be able to combine ideas without forcing them into a rigid or predesigned hierarchy. However, it has also been hard to manage for some developers and some maintainers. It is also notationally heavy. C++0x contains attempts to address that without flexibility or performance hits.

HD Certain characteristics of the C++ language have the potential to create some nasty unintended consequences (such as macros). What are some other unintended consequences that you would like to see disambiguated in C++ or modern languages in general?

BS Actually, I knew macros were nasty when I started using C; but like most people, I underestimated their nastiness and pervasive negative influences. The pervasive use of macros in C is probably the major reason we didn't have excellent C++ development environments a decade ago. There are also too many, too confusing ways of initializing objects in C++; I hope to address that with a uniform mechanism in C++0x. I mentioned templates in my answer to the previous question. They are the major success in later C++ (post 1985), but their success has strained the language—again, parts of C++0x are intended to deal with that.

Many minor and not-so-minor problems we have discovered over the years cannot be addressed in C++ for compatibility reasons. For example, the declarator syntax is an unnecessary complication—just about any linear notation would be better. Similarly, many defaults are wrong: constructors should not be conversions by default, names should not by default be accessible from other source files, and so on. Not controlling the linker has been a constant source of problems; in particular, implementers seem to delight in providing similar features in incompatible forms.

There have also been positive surprises. The most spectacular has been the pervasive use of destructors in techniques relating to resource management and error handling (using exceptions). I knew destructors were a good idea—after all, you have to reverse the effect of a constructor—but I didn't realize quite how central they would be to good use of C++.

HD You commented on your Web site, "I think we should look for elegance in the applications built, rather than in the languages themselves." Is the move toward domain specific languages (DSLs) a convergence of these?

BS Yes, almost certainly. It is often an attempt in that direction. Sometimes, it even works.

HD What are your thoughts on DSLs in general? What do you envision as the relationship between DSLs and general-purpose languages?

BS I worry about the number of languages being designed, implemented, introduced with great fanfare, and then fading away without significant impact. During this—typically many-years long phase of development—a new language consumes significant resources with essentially no returns. I wrote a paper on this phenomenon called "A Rationale for Semantically Enhanced Library Languages" (research.att.com/~bs/SELLrationale.pdf). I argue for using libraries, possibly supported by tools, and a general-purpose language.

I think that a DSL should be a last resort, not a first. If at all possible, the DSL should be firmly rooted in a general-purpose language and standard tool chains. A DSL needs a general-purpose language (or at least a systems programming language) for its implementation and the implementation of its run-time primitives. I think it would be best if a DSL is consciously and firmly paired with at least one general-purpose language so that it is easy to add new facilities through the use of libraries written in that general-purpose language. Obviously, a professional should master several languages, but I do wonder if the sum of complexity of a variety of DSLs couldn't become so high that it became a problem. Also, many (if not most) DSLs seem to "want to" become general-purpose languages.

HD You mentioned that many constructs in the C++ were intentionally left ambiguous due to varied definitions on different hardware. Do you see advancements in interoperability that could disambiguate some of these constructs?

BS "Ambiguous" is the wrong word. Far too many things have been left undefined or implementation-defined. I suspect that if I could redefine C++ from scratch it would have no undefined behaviors and far fewer implementation-defined ones. However, I don't have a time machine and we just cannot break hundreds of millions of lines of code by picking a set of resolutions today.

Methodology and Best Practices

HD What process methodology do you tend to use and teach?

BS Identify key application concepts, identify useful libraries, build new libraries in support of the application concept, try out ideas early, integrate early, test early and often, use documentation and tutorial material as design tools, and grow larger programs from smaller ones (iterating along the way). It should be obvious that I'm focusing on relatively small projects just now.

HD Do you see an intersection or affinity between a language and a development methodology?

BS I think so, as far as library design is seen as a design/development technique. Focusing on increasingly higher-level (closer to the application) facilities through library building places requirements on the language. I wouldn't overstate this point, but I don't think you could have a single development methodology for (say) COBOL, C, Java, C++, and Python and expect to gain more than the minimal support from each language.

HD What are some of your personal rules of thumb when creating software?

BS Focus on key concepts; focus on their interfaces; focus on the management of resources (memory, files, locks, and so on); focus on error handling. The design of good invariants for classes and Resource Acquisition Is Initialization (RAII) are key techniques.

HD There's a lot of buzz around being agile. What does "agile" mean to you? Does C++ support agile?

BS I don't use the word; it's much too vague. Of course C++ supports agile—whatever that means.

Looking toward the Future

HD How can a language evolve to support advanced features such as templates, dynamic events, and self-writing code, and at the same time remain accessible to new entrants?

BS I don't know. I don't think there is a general answer. New features can be important as far as they support more effective techniques in the context of a language. However, stability is essential: one of the reasons for the continuing strength of C and C++ is the care taken by the standards committees for old (often decades old) code to remain valid and for the integration of new features to be smooth. This is not easy, to say the least, and the introduction of new features doesn't always succeed. Too often, the concerns of novices are ignored by the standards committee members. Some of the key features planned for C++0x, such as uniform initialization, the auto keyword (for deducing a variable type from its initializer), and concepts such as checking of template argument requirements should serve to make the language easier to use by nonexperts.

HD Do you see language metadata as a key foundation for future programming languages?

BS No. I personally feel most uncomfortable with nontrivial uses of metadata.

HD Do you see a fundamental shift in the future for concurrency if CPUs begin ramping up the number of cores? How might the concurrency challenge be addressed in C++0x?

BS C++0x provides the basics: a machine model suitable for multithreading, a set of low-level primitives for library building, and a threads and locks library API. I would love to see more (and probably will over the next few years), especially a simpler, higher-level concurrency model based on thread pools, futures, and message queues. We need to find automatic or near-automatic ways of spreading a computation over many processors and localizing activities on those processors. There is lots of work in this area—much in C++—but not yet a dominant model. Examples include STAPL from Texas A&M University and TBB from Intel.

HD What are your thoughts on general-purpose computing on graphics processing units (GPGPU)?

BS Tantalizing, but I don't have the practical experience to comment beyond the obvious that it takes a lot of skill to utilize a special-purpose processor.

HD Do you foresee High-Performance Computing (HPC) ultimately becoming transparent to programmers? Moreover, would this goal be one for a language or for a library or framework?

BS Somewhat transparent, only; I don't think concurrency could or should be completely transparent. For starters, error handling can be very different depending on the availability of processors, shared (or not) memory, distribution (or not), and latency. Approximating transparency where that is appropriate is and will remain a goal for new languages, new language features, and (my favorite) new libraries. The latter is only feasible if the underlying language provides the basic guarantees needed as a machine model and a set of very low-level primitives. C++0x will do that.

HD How is the design of C++0x coming along?

BS We are near the end—finally! At least I hope so—nothing is certain until the votes are in and towards the end things can get quite tense and emotional. The current plan has us voting out the complete new standard for public comment in June, giving us a new formal standard 12 to 18 months later. Obviously, I could write a book on this one topic: how do you make a standard, what's supposed to be the guiding principles, and exactly what's in it? I almost have; see my HOPL paper "Evolving a language in and for the real world: C++ 1991-2006" (available at research.att.com/~bs/hopl-almost-final.pdf) and anything with C++0x in the title on my home pages. If you are a glutton for punishment, you can look for "WG21" on the Web and find all the ISO C++ standards committee's papers (including all the proposals). If nothing else, it will convince you that it is a lot of work. Improving a widely used programming language is hard, especially one that is at the implementation layer of many, many tools, languages, and applications. You can also find a few videos by me and others online and on my C++ page.

Books and Phones

HD What are you currently reading?

BS On technical material, I'm back to Hennessy and Patterson for a refresher on machine architecture, and I'm trying to collect papers and articles that help people write good code (for a course I'm teaching). Finding such articles is much harder than I expected; academic papers tend to be very specialized and nonacademic papers are often high on promises but short on evidence (suggestions welcome). Then, of course, there is a steady stream of documents relating to the C++ standardization. I was about to refresh my Knuth, but someone has absconded with my volume III, so that'll have to wait. For pleasure, I'm rereading a bit of O'Brian's Aubrey and Maturin series. I'm trying to refresh my understanding of science, so currently just coming off a Richard Dawkins binge. I also recently finished Rodger's Command of the Ocean: A Naval History of Britain, 1649-1815 (hence the visit to O'Brian).

HD You said "I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone." Do you have a smartphone, and has this gotten any easier?

BS I'm not a great fan of phones. I prefer face-to-face communication and the written word if I cannot get that. Even the fanciest phone is not quite up to well-phrased e-mail. I use a slim phone that fits comfortably in my pocket and I don't use anywhere near all of its features. I'd give up just about any feature for sound quality and reliability. To be fair, user interfaces tend to be much better today than when I made that remark.

Howard Dierking is the Editor-in-Chief of MSDN Magazine.


Office Development: Manage Metadata with Document Information Panels

Office Development

Manage Metadata with Document Information Panels

Ashish Ghoda
This article discusses:
  • Document Information Panels
  • Maintaining metadata
  • Using InfoPath forms
This article uses the following technologies:
2007Office System, .NET Framework 3.5
Code download available at: DocumentInformationPanel2008_04.exe (205KB)
Contents
Metadata Management with Document Information Panels
Document Properties Management
Custom Document Properties Management
The InfoPath Form Template
Show or Hide Document Information Panel
Insert Checklist Quick Parts
Binding
Integrating Document Information Panel with the Document
Version Issues
Wrapping It Up

T
hese days, every organization faces tough information management challenges. With critical information stored in disparate locations in a wide variety of formats, making the data easy to find and consume can be tricky. In addition to simply enabling discoverability and consumption of data, companies must also have policies in place for data retention and government compliance.

To meet these emerging needs and implement information-driven applications, documents must be appropriately attributed with metadata and that metadata should be made available across all systems in the enterprise. In this article I'll demonstrate how to enable effective metadata and content management using the Document Information Panel, a form-based component within Microsoft® Word 2007, Excel® 2007, and PowerPoint® 2007 that allows users to manage document-level metadata.

Metadata Management with Document Information Panels

The Microsoft Office platform always allowed you to maintain both standard and custom properties for documents. Document Information Panels, however, let you implement additional functionalities such as metadata-based search and automation of information-driven business processes. With Document Information Panels, users can implement as document-specific metadata on the following four types of document properties: core or standard properties, custom document properties, content types for Microsoft Office SharePoint® Server (MOSS) 2007, and extended and advanced custom properties. Let's look at each.

Core Properties Users can manage core document properties common to all Office documents—Author, Title, Subject, Keywords, Category, Status, and Comments—using the standard Document Properties view of the Document Information Panel.

Custom Document Properties To add further business context to Office documents, the 2007 Microsoft Office system allows you to attach a trusted custom InfoPath® form templates as a custom Document Information Panel. Figure 1 shows an InfoPath form template that I've called ChecklistMetadata—I'll walk through this with you later. It is attached as a custom Document Information Panel to a Word 2007 document.

Figure 1 InfoPath Form Template Attached to a Word 2007 Document
Figure 1 InfoPath Form Template Attached to a Word 2007 Document(Click the image for a smaller view)
Figure 1 InfoPath Form Template Attached to a Word 2007 Document
Figure 1 InfoPath Form Template Attached to a Word 2007 Document (Click the image for a larger view)

Content Types for SharePoint Server 2007 Documents stored in MOSS 2007 document libraries contain custom document properties that are mapped to content types defined for the associated SharePoint site. These documents will automatically display the content type's fields as custom Document Information Panels. Users can update the content type field values in the offline/online mode and they will automatically be saved back to the document library when the user is connected.

For more information on the custom Document Information Panel for MOSS 2007-based documents, you can go to go.microsoft.com/fwlink/?LinkId=109825.

Extended and Advanced Custom Properties Users can access and manage additional document-specific custom properties by accessing the document properties window through the Document Information Panel's Advanced Properties option, as shown in Figure 2. From this dialog, users can populate Extended Properties using the Summary tab and create new custom properties using the Custom tab. For even richer metadata management capabilities, the Document Information Panel can be configured to allow for the display and management of document metadata via InfoPath forms.

Figure 2 Advanced Properties Menu
Figure 2 Advanced Properties Menu(Click the image for a smaller view)
Figure 2 Advanced Properties Menu
Figure 2 Advanced Properties Menu (Click the image for a larger view)

The Developer tab in the 2007 Office system lets you customize documents using code, controls, customized XML, and templates—it is also where the Document Information Panel can be enabled. Figure 3 shows the highlighted Document Panel option for a Word 2007 document. To display and configure the Document Information Panel within an Office 2007 document, click on the Document Panel option.

Figure 3 Document Panel Option for a Word 2007 Document
Figure 3 Document Panel Option for a Word 2007 Document(Click the image for a smaller view)
Figure 3 Document Panel Option for a Word 2007 Document
Figure 3 Document Panel Option for a Word 2007 Document (Click the image for a larger view)

If you want to attach a custom Document Information Panel, specify a custom template using the network path or URL of the InfoPath form template. Note that you do not need to specify a custom template at this point if you just want to view the standard document properties.

Next, select the Standard Properties or Custom Properties option (only if a custom InfoPath form template is attached) to use as "Display by Default" for the Document Information Panel. When multiple views are available in the InfoPath form template, the default view will be displayed as the custom Document Information Panel with access to other views.

One of the key requirements for efficient metadata management is to have a consistent metadata schema across applications. The Document Information Panel facilitates this consistency and reuse—the same InfoPath 2007 template can also be attached as a custom Document Information Panel to Word, Excel, and PowerPoint 2007 documents.

As you probably know, the 2007 Office system documents are based on the Open XML format. As a result, they consist of a .zip file package containing XML files that hold content and metadata related to the document. As such, document properties are stored as XML inside of the package. Since all this information is stored as XML, it is quite simple to access it programmatically in order to automate metadata-driven business processes. I'll explain how that works in the following sections.

Document Properties Management

Standard document properties can be maintained through the Document Properties view of the Document Information Panel. To see where these properties are actually stored in the OpenXML package, open the .rels file in the _rels folder of the unzipped Office document. As you can see in Figure 4, this file shows that standard document properties (core properties) are stored in the core.xml file within the docProps folder. The core.xml file contains all of the standard document properties that are populated from the Document Properties view in the Document Information Panel.

Figure 4 A Relationship File (Close Figure 4)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships
   xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
  <Relationship Id="rId3"
    Type="http://schemas.openxmlformats.org/officeDocument/2006/
    relationships/extended-properties" Target="docProps/app.xml"/>
  <Relationship Id="rId2"
    Type="http://schemas.openxmlformats.org/package/2006/
    relationships/metadata/core-properties" Target="docProps/core.xml"/>
  <Relationship Id="rId1"
    Type="http://schemas.openxmlformats.org/officeDocument/2006/
    relationships/officeDocument" Target="word/document.xml"/>
  <Relationship Id="rId4"
    Type="http://schemas.openxmlformats.org/officeDocument/2006/
    relationships/custom-properties" Target="docProps/custom.xml"/>
</Relationships>

The code in Figure 5 demonstrates how to get a standard document property value and set it to a new value using the Word object model's DocumentProperties object. The BuiltInDocumentProperties property of the Word document object returns a DocumentProperties object that is a collection of standard DocumentProperty objects. The Item property is then used to retrieve a specific DocumentProperty object by specifying a property name (in this case, the category property). The Value property of the DocumentProperty object is used to set or get the value of the property.

Figure 5 Get a Standard Document Property (Close Figure 5)
Dim StandardProperties As Microsoft.Office.Core.DocumentProperties
Dim StandardProperty As Microsoft.Office.Core.DocumentProperty

Dim Category As String

'Assign Current Document Standard Properties
StandardProperties = _
  DirectCast(Globals.ThisDocument.BuiltInDocumentProperties, _
  Microsoft.Office.Core.DocumentProperties)

'Assign "category" property
StandardProperty = StandardProperties.Item("category")

'Get "category" property value
Category = StandardProperty.Value

'Set "category" property value to "Modified Checklist"
StandardProperty.Value = "Modified Checklist"

With the Word object model, you can also use the CustomXMLParts object to access these properties as XML files (see Figure 6). The CustomXMLParts object returns a collection of CustomXMLPart objects which contain the XML files that describe the document. The first XML file in the collection is the core.xml file, which contains the standard document properties. Once you have obtained a reference to the XMLDocument object, you can use XPath query methods such as SelectSingleNode to manipulate nodes such as the category property node.

Figure 6 Access Properties as XML (Close Figure 6)
Dim OfficeCustomXMLParts As Microsoft.Office.Core.CustomXMLParts
Dim Category As String

'Assign CustomXMLParts containing standard/core
'document properties XML Part
OfficeCustomXMLParts = DirectCast(Globals.ThisDocument.CustomXMLParts, _
  Microsoft.Office.Core.CustomXMLParts)

'Find the "category" property node and get the value of "category"
 'standard property

Category = OfficeCustomXMLParts.Item(1).SelectSingleNode _
 ("//*[local-name(.)='category']").Text

'Find the "category" property node and set the value of "category"
' standard property
OfficeCustomXMLParts.Item(1).SelectSingleNode _
 ("//*[local-name(.)='category']").Text = "modified category"

Custom Document Properties Management

Custom document properties can be maintained via an InfoPath form template that is attached as a custom Document Information Panel. The InfoPath form data is stored inside of the document as a custom XML document part. Note that relationships for all custom XML Parts included in the document are defined in the document.xml.rels file within the Word/_rels folder. The custom XML files themselves are found in the customXml folder. Each custom XML document part has its own relationship definition file, which is stored in the customXml/_rels subfolder. As shown below, the item1.xml part has a corresponding item1.xml.rels file, which points to the itemProps1.xml file and specifies that this XML part holds custom properties describing item1.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Relationships
  xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
  <Relationship Id="rId1"
    Type="http://schemas.openxmlformats.org/officeDocument/2006/
    relationships/customXmlProps" Target="itemProps1.xml" />
</Relationships>

The itemProps1.xml file then indicates through schema reference that the custom XML Schema is an InfoPath schema. This enables Word to open item1.xml as an InfoPath form:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<ds:datastoreItem ds:itemID="{A9851B97-75EE-4C97-ACC6-D9EB7DA1D3DC}"
  xmlns:ds="http://schemas.openxmlformats.org/officeDocument/
  2006/customXml">
  <ds:schemaRefs>
    <ds:schemaRef ds:uri="http://schemas.microsoft.com/office/
      infopath/2003/myXSD/2007-09-01T16:03:56" />
    <ds:schemaRef ds:uri="http://schemas.microsoft.com/
      office/infopath/2003" />
  </ds:schemaRefs>
</ds:datastoreItem>

In order to specify that custom XML properties should be attached as a custom Document Information Panel, simply indicate this relationship through a schema reference:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<ds:datastoreItem ds:itemID="{B97FA79B-5941-4441-9A4B-3E001C0278D6}"
   xmlns:ds="http://schemas.openxmlformats.org/officeDocument/2006/
  customXml">
  <ds:schemaRefs>
    <ds:schemaRef ds:uri="http://schemas.microsoft.com/office/2006/
    customDocumentInformationPanel" />
  </ds:schemaRefs>
</ds:datastoreItem>

Now if I open this item, it indicates that InfoPath is used as a custom XML property editor and my ChecklistMetadata.xsn file is attached as an InfoPath form template to the custom Document Information Panel. To access the attached InfoPath form XML file, I can use the CustomXMLParts object as I have used to access standard document properties.

The code below demonstrates use of the CustomXMLParts object and the SelectSingleNode method to retrieve the InfoPath form field ProjectName node of the CL—Business—Information Technology category checklist:

Dim InfoPathXML As XmlDocument = New XmlDocument()
Dim projectname As String

'Load InfoPath Form XML
InfoPathXML.LoadXml(Globals.ThisDocument.CustomXMLParts.Item(4).XML)

'Get the "Project Name" field value of the
'   CL - Business - Information Technology checklist metadata
projectname = InfoPathXML.SelectSingleNode _
 ("//*[local-name(.)='ProjectName']").InnerText

Additional custom properties can be maintained through the Advanced Properties view within the Document Information Panel. You can see where these properties are stored within the document package by looking at the .rels file within the root _rels folder. As you'll see, the relationship file indicates that custom document properties are stored in a file named custom.xml within the docProps folder.

For demonstration purposes I have added Client as a custom property to the document using the Advanced Properties dialog. As shown in the following code, the Client property is stored as follows in the docProps/custom.xml file:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<Properties xmlns="http://schemas.openxmlformats.org/
  officeDocument/2006/
  custom-properties"
  xmlns:vt="http://schemas.openxmlformats.org/officeDocument/2006/
  docPropsVTypes">
  <property fmtid="{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"
    pid="2" name="Client">
    <vt:lpwstr>
      Client Value
    </vt:lpwstr>
  </property>
</Properties>

The code in Figure 7 demonstrates how to programmatically get Client custom property value and set it to a new value using the Word object model. As shown, the CustomDocumentProperties property of the Word Document object returns a collection of custom DocumentProperty objects. The Item property can then be used to retrieve a specific DocumentProperty object by specifying property name (in this case, Client). The Value property of the DocumentProperty object is used to set or get the value of a custom document property.

Figure 7 Get Client Custom Property (Close Figure 7)
Dim CustomProperties As Microsoft.Office.Core.DocumentProperties
Dim CustomProperty As Microsoft.Office.Core.DocumentProperty
Dim client As String

'Assign Current Document Standard Properties
CustomProperties = DirectCast(Globals.ThisDocument.CustomDocumentProperties, Microsoft.Office.Core.DocumentProperties)

'Assign "client" property
CustomProperty = CustomProperties.Item("Client")

'Get "client" property value
client = CustomProperty.Value

'Set "client" property value to "Microsoft"
CustomProperty.Value = "Microsoft"

The following code demonstrates how to add a new custom property, Checklistversion, and set the value to 1:

Dim CustomProperties As Microsoft.Office.Core.DocumentProperties

'Assign Current Document Standard Properties
CustomProperties = _
  DirectCast(Globals.ThisDocument.CustomDocumentProperties, _
  Microsoft.Office.Core.DocumentProperties)

'Add "ChecklistVersion" as a custom property and set value to "1.1"
CustomProperties.Add("ChecklistVersion", False, _
  Microsoft.Office.Core.MsoDocProperties.msoPropertyTypeNumber, "1.1").

As you can see, the Add method of the CustomDocumentProperties object is used both to add the new custom document property and to set its value to 1.1. And while the value is added as a string, the msoPropertyTypeNumber specifies that the string should be considered to be a number.

So far, it is clear that the Document Information Panel allows users and system administrators to manage and maintain metadata more effectively by storing it within an OpenXML document package, but as a separate document part within that package. However, efficient metadata-driven business process implementation is only possible when the metadata values can be used to automate business processes, control the content environment, and help users to automate the generation of repeatable content.

Next, I'll develop a Checklist management application based in Word 2007 to demonstrate how you can automate business processes by associating standard document properties and InfoPath form-based custom fields with other Word 2007 components like Content Controls, Quick Parts/Building Blocks, and customized fluent Ribbon controls.

To automate checklist management using Word 2007, I first must develop and publish an InfoPath form template, ChecklistMetadata.xsn, representing checklist metadata to attach as a Custom Document Information Panel to the Word 2007 document. Then I created a Word 2007 Document Project in Visual Studio® and attach the ChecklistMetadata.xsn form template as Custom Document Information Panel. Next, I created a Custom Ribbon control that will be used to show and hide the Document Information Panel.

I then developed a Word 2007 template, ChecklistsQuickPartsTemplate.dotm, containing different checklists as categorized custom Quick Parts and distribute to the user's application data folder to integrate with Word 2007 documents. Then I can implement a Custom Ribbon control to insert checklists Quick Parts programmatically as building blocks to the document from the attached ChecklistsQuickPartsTemplate.dotm template. Insert the checklist Quick Parts based on the checklist category selected in the custom Document Information Panel

Following that, I programmatically bound in standard document properties and InfoPath form fields (displayed using a custom Document Information Panel) to the document itself using Word 2007 content controls. Now I'll show you how users can manually add standard and extended document properties as Quick Parts to the document.

The InfoPath Form Template

Every checklist can have common and checklist-specific metadata. The InfoPath form template I am going to develop defines a sample generic and checklist-specific metadata.

First, I define the following checklist categories in the InfoPath form template at design time. They are shown in Figure 8. For illustration purposes, I have defined only two types of main categories: Business and Personal. I have also defined and developed metadata sections for the two categories: Business—Information Technology and Personal—Hospital categories.

Figure 8 Sample Checklist Categories
Figure 8 Sample Checklist Categories(Click the image for a smaller view)
Figure 8 Sample Checklist Categories
Figure 8 Sample Checklist Categories (Click the image for a larger view)

The metadata added for the Business—Information Technology category includes Project Name, Purpose, and Team Members (all as multi-line textboxes, as you see in Figure 9). Project Number and Technology Used (as single-line textboxes), and Start and End Date as date pickers. Similarly, Figure 10 shows how the metadata is displayed for the Personal—Hospital category.

Figure 9 Business—Information Technology Metadata
Figure 9 Business—Information Technology Metadata(Click the image for a smaller view)
Figure 9 Business—Information Technology Metadata
Figure 9 Business—Information Technology Metadata (Click the image for a larger view)
Figure 10 Personal—Hospital Metadata
Figure 10 Personal—Hospital Metadata(Click the image for a smaller view)
Figure 10 Personal—Hospital Metadata
Figure 10 Personal—Hospital Metadata (Click the image for a larger view)

Based on the selection of the metadata category, the related metadata section will be displayed. This functionality is enabled utilizing InfoPath's conditional formatting functionality for each category section. For security reasons, the InfoPath form template must be published or installed to a local intranet or a trusted site before it can be attached to a document as a custom Document Information Panel.

Before publishing the InfoPath form template, set up proper security settings using Tools | Form Options | Security and Trust option (Figure 11). Once the template is published to a trusted location, attach it as custom Document Information Panel.

Figure 11 Security and Trust Settings
Figure 11 Security and Trust Settings(Click the image for a smaller view)
Figure 11 Security and Trust Settings
Figure 11 Security and Trust Settings (Click the image for a larger view)

To maintain a high level of security, digitally sign the InfoPath form template using a certificate provided by an official third-party certificate provider and publish it to a trusted location such as a MOSS document library or Forms Server.

While you are in the development phase, it can be less cumbersome to sign your InfoPath form template using a self-signed certificate and publish it to the local machine. More information on this process is available at go.microsoft.com/fwlink/?LinkId=109827.

After you have created the InfoPath form template, create a new Word 2007 Document project in Visual Studio 2008. At design time, attach the ChecklistMetadata.xsn file as a custom Document Information Panel to the document, and select custom view as the default view. The InfoPath form view, Checklist Metadata, should be displayed as a custom Document Information Panel in the document.

You can confirm that the InfoPath form template is working by changing checklist category to CL—Business—Information Technology and CL—Personal—Hospital. You should see appropriate metadata section displayed within the Document Information Panel.

Show or Hide Document Information Panel

In order to gain more real estate on the monitor while you are working on the document, it is sometimes preferable to hide the Document Information Panel. The Office object model allows programmatic control over the visibility of the Document Information Panel, and I'll show you how to show or hide it by creating a customized ribbon control.

Add a ribbon to the project by selecting the Ribbon (Visual Designer) option from the new item dialog. Name the ribbon CustomizeChecklistRibbon. Add a button control named ShowHideDIP, which you will use to show or hide the Document Information Panel (see Figure 12).

Figure 12 Show or Hide Document Information Panel
Figure 12 Show or Hide Document Information Panel(Click the image for a smaller view)
Figure 12 Show or Hide Document Information Panel
Figure 12 Show or Hide Document Information Panel (Click the image for a larger view)

The following code demonstrates how to show or hide the Document Information Panel programmatically. Add this code inside click event handler of the ShowHideDIP button:

'Check if Document Information Panel is visible
If Globals.ThisDocument.Application _
  .DisplayDocumentInformationPanel = True Then
  'Hide Document Information Panel
  Globals.ThisDocument.Application. _
  DisplayDocumentInformationPanel = False
Else
  'Show Document Information Panel
  Globals.ThisDocument.Application. _
  DisplayDocumentInformationPanel = True
End If

The DisplayDocumentInformationPanel property of the Word Application object gets or sets a Boolean value used to control the display of the Document Information Panel.

Insert Checklist Quick Parts

Metadata-based content reusability is one of the key requirements for effective task automation. The Word 2007 Quick Parts feature allows users to identify reusable content sections for a document, categorize those content sections, and add them as Quick Parts to either the default Microsoft building blocks template or a custom Quick Parts template. Once the content is added as a Quick Part and copied to the AppData folder on a user's machine, the user has access to the reusable content for any Word 2007 document.

To insert checklist Quick Parts specific to the selected category in the custom Document Information Panel, retrieve the ChCategory InfoPath form XML field from the click event of the "Insert Checklist" ribbon control (Figure 3) by calling this method:

Function GetChecklistCategoryInfoPathField() As String
  Dim InfoPathXML As XmlDocument = New XmlDocument()
  Dim ChecklistCategory As String
    InfoPathXML.LoadXml(Globals.ThisDocument.CustomXMLParts.Item(4).XML)
    ChecklistCategory = InfoPathXML.SelectSingleNode( _
      "//*[local-name(.)='ChCategory']").InnerText
  Return ChecklistCategory
End Function

The Checklist InfoPath XML, attached as a custom Document Information Panel, is the fourth file in the CustomXMLParts collection and is retrieved using the Item property of the CustomXMLParts object. I load the InfoPath XML into a new XmlDocument object and query the document for the selected ChCategory node.

The method shown in Figure 13 filters the checklist Quick Parts from the Checklist Quick Parts template based on the selected checklist category. All remaining checklist Quick Parts are then inserted into the document with each Quick Part starting on a new page. (For more on identifying and inserting checklists as categorized Quick Parts to the custom Checklist Quick Parts Template, see go.microsoft.com/fwlink/?LinkId=109830.)

Figure 13 InsertChecklistQuickPartsAsBuildingBlock (Close Figure 13)
Sub InsertChecklistQuickPartsAsBuildingBlock(ByVal _
  ChecklistCategory As String)

  'Declaration
  Dim oAttachedTemplate As Microsoft.Office.Interop.Word.Template
  Dim oBuildingBlock As Microsoft.Office.Interop.Word.BuildingBlock
  Dim selection As Microsoft.Office.Interop.Word.Selection
  Dim TotalQuickParts As Integer
  Dim iIndex As Integer

  'Define Attached Template - ChecklistsQuickPartsTemplate.dotm
  oAttachedTemplate = Globals.ThisDocument.AttachedTemplate

  'Get Total Number of quick parts available in the selected checklist
  ' category in the InfoPath form
  TotalQuickParts = oAttachedTemplate.BuildingBlockTypes _
    .Item(Word.WdBuildingBlockTypes.wdTypeQuickParts) _
    .Categories.Item("CL - " + ChecklistCategory) _
    .BuildingBlocks.Count

  selection = Globals.ThisDocument.Application.Selection

  'Insert all quick parts available in the defined category
  'Each quick part starts with a new page
  For iIndex = 1 To TotalQuickParts

    'Add Building Block Content Control
    selection.Range.ContentControls.Add(Word.WdContentControlType _
      .wdContentControlBuildingBlockGallery)

    'Define Quick Part as Building Block
    oBuildingBlock = oAttachedTemplate.BuildingBlockTypes _
      .Item(Word.WdBuildingBlockTypes.wdTypeQuickParts) _
      .Categories.Item("CL - " + ChecklistCategory) _
      .BuildingBlocks.Item(iIndex)

    'Insert Quick Part
    oBuildingBlock.Insert(selection.Range)

    selection.EndKey(Word.WdUnits.wdStory)
    selection.TypeParagraph()

    'No new page required if it is a last Quick Part
    If Not iIndex = TotalQuickParts Then
      selection.InsertBreak(Type:=Word.WdBreakType.wdPageBreak)
    End If

  Next

End Sub

After the code retrieves the template associated with the document (ChecklistQuickPartsTemplate.dotm, in this example), it proceeds to filter the template's BuildingBlock objects by the building block type (WdBuildingBlockTypes.wdTypeQuickParts) and by the user-selected category. Finally, it iterates through the filtered Quick Parts and inserts each checklist Quick Part into the document as a Content Control. You'll notice that the Selection object is used to define the insertion point in the active Word 2007 document.

Next, the code must insert a Page Break into the document before inserting the next checklist quick part. The InsertBreak method of the Selection object inserts a page, column, or section break into the Word document. The WdBreakType.wdPageBreak enumeration specifies that the break is a page break.

Binding

Since InfoPath form fields and document properties are defined and stored as custom XML files, you can bind the information available in the Document Information Panel to Content Controls within the document. The following code shows how to bind a Content Control to the ProjectName InfoPath form field populated within the custom Document Information Panel:

'Sample code to bind first content control to InfoPath
'Form Field ProjectName
Globals.ThisDocument.ContentControls(1).XMLMapping.SetMapping _
  ("//*[local-name(.)='ProjectName']", , _
  Globals.ThisDocument.CustomXMLParts.Item(4))

The XMLMapping object represents a mapping between a Word 2007 content control and an XML node—the SetMapping method of the XMLMapping object establishes this mapping. I have identified the node ProjectName of the custom XML file using an XPath expression. The Checklist InfoPath form XML file is retrieved utilizing the document object's CustomXMLParts collection.

The following demonstrates how to bind a Content Control to the category standard document property:

'Sample code to bind first content control to a standard document
'property - category
Globals.ThisDocument.ContentControls(2).XMLMapping.SetMapping _
  ("//*[local-name(.)='category']", ,  _
  Globals.ThisDocument.CustomXMLParts.Item(1))

The only difference between this code and the code to map the content control with the InfoPath form field is the change in the XPath expression and the change in the index used to retrieve the document part from the CustomXMLParts collection.

Figure 14 shows how the Content Control binds to the InfoPath form field related to ProjectName checklist metadata and then it binds to the Category document standard property. As you can see, mapped property values are automatically populated in Content Controls.

Figure 14 Content Control Bound to InfoPath Form Field
Figure 14 Content Control Bound to InfoPath Form Field(Click the image for a smaller view)
Figure 14 Content Control Bound to InfoPath Form Field
Figure 14 Content Control Bound to InfoPath Form Field (Click the image for a larger view)

Integrating Document Information Panel with the Document

It is possible to integrate standard and extended document properties as Quick Parts within the document. As you know, the standard document properties can be managed in the Standard Properties view of the Document Information Panel and extended document properties can be accessed using the Advanced Document Properties Management window through the Document Information Panel.

As shown in Figure 15, navigate to the Insert tab and click on the Quick Parts option. The Document Property option enables users to insert standard and extended document properties. In the figure, the Author standard document property is added as a Quick Part to the document. The inserted Quick Part is tightly integrated with the document property. In my example, if you change the value of the Author property within the Document Information Panel or within the document Quick Part, the modified value will automatically be reflected and the document property will be changed.

Figure 15 Author Standard Document Property Added as a Quick Part
Figure 15 Author Standard Document Property Added as a Quick Part(Click the image for a smaller view)
Figure 15 Author Standard Document Property Added as a Quick Part
Figure 15 Author Standard Document Property Added as a Quick Part (Click the image for a larger view)

Version Issues

The 2007 Office system product suites allow opening and saving documents in the compatibility mode so that users with earlier versions of Office (Office 97-Office 2003) can open and work on documents. Obviously the features that are not supported in the previous versions of the Office products will not be supported in my solution here. Let's see what that means.

The Document Information Panel displays the standard document properties and custom document properties. All functionalities related to the standard and custom Document Information Panel views will work if a document is stored in compatibility mode and only when you are using a 2007 Office system application to work on the document.

However, once you use an earlier version of Office to open and save the document, the custom Document Information Panel will be detached from the document along with its related functions. The corresponding information will not be available to the users even if at a later time the document is opened again using Office 2007.

The good news is standard document properties will be preserved as-is, even if the 2007 Office system document saved in compatibility mode is opened and saved using an earlier version of Office. If the document is opened using Office 97-Office 2003, the user can access the standard document properties through the Document Properties Management dialog. If the user opens a document using the 2007 Office system, the standard document properties can be accessed through the Document Information Panel. All of the changes made to the standard properties will be retained in cross-version scenarios.

If the option "Always show Document Information Panel on document open and initial save" is checked to make the Document Information Panel always visible, that information will not be retained once the document is saved in the compatibility mode and is opened and saved using an earlier version of Office. In this case, the Document Information Panel will not be visible by default upon opening the document using Office, after first being saved in the earlier version.

If any document properties are bound to content controls within the document, upon saving a Word 2007 document in compatible mode, all Content Controls will be converted to the static text and will cease to be mapped to the document properties.

Wrapping It Up

The effective use of the standard and custom Document Information Panels can facilitate efficient metadata management and allow building efficient content management applications in conjunction with the Quick Parts and Content Controls. The value of using these technologies is standardized, structured metadata across the 2007 versions of Word, Excel, and PowerPoint. The more widely you employ this feature, the easier it will be to search your content and drive businesses processes using it.

If you have questions regarding this Document Information Panel feature, feel free to e-mail me at AskAshish@technologyopinion.com. And remember, all code for this project is available on the MSDN® Magazine Web site.

Ashish Ghoda is a technical manager at a big-four accounting firm. He's a Microsoft Certified Professional and Microsoft Certified Application Developer in .NET. He also manages the Web site technologyopinion.com and writes development articles. You can contact him at AskAshish@technologyopinion.com.


Editor's Note: A Service-Oriented Editor's Note

Editor's Note
A Service-Oriented Editor's Note
Howard Dierking

The term "service" has come to mean many different things to different people. We have seen software characterized as a service or as something that requires services in addition to the main product. Within our applications, we have domain services and externally facing services. We call our user interface "presentation services" and our data access code "data services." And one of the more recent architectural trends is to plan for and implement service-oriented architectures.

While services and distributed applications are related, they are not equivalent. Most services, particularly those built on a framework like ASP.NET or Windows Communication Foundation (WCF), naturally lend themselves to distribution across the network. Conversely, distributed applications are best understood and maintained when they are composed from services. That said, distributed applications are a technical solution for solving problems such as scale, whereas services are more semantic in nature and exist as a way to decompose and manage the complexity of systems.

The question of services is something that I continue to grapple with in my own journey as a developer as it seems to require a paradigm shift in the way that I translate business domain concepts into technology artifacts. I have spent a great deal of energy thinking about and developing my skills in topics like object-oriented heuristics and design patterns. An idea that is generally accepted within this school of thought is that of the rich domain model. This is a class model that is very closely bound to the language of business, is independently testable without needing a user interface or database, and that encapsulates all of the operations that are required to address the problems of its business domain. To this day, I start all of my development work with two projects: a domain model class library and a domain model tests class library.

Initially, to enable services in my work, I simply decorated my domain classes with serialization attributes and sent them out over the wire via an ASP.NET or WCF service. Then I realized that this created unintentional side effects, as the public data interface of my domain objects made an implicit assumption about the logic behind the contract—and this became apparent only when testing based on the generated service proxies. As a result, I changed my approach to use a data container object, also known as a data transfer object (DTO), for passing the data across my service boundaries. This worked pretty well for a while as the semantics of my domain layer and the way that I created applications didn't really have to change all that much.

Then came Windows Workflow Foundation—and about the same time, I also started dabbling in Office Business Application (OBA) development. Add data warehousing along with a couple of other technologies to the equation and that brings us to today—and gives you a framework for how I'm trying to define services in the context of my development efforts. Now I am dealing with questions like how logic is factored between my workflow instances and my domain classes and how I can unit test my workflow services in a test-driven fashion.

I mention my own journey here because I imagine that many of you have had similar experiences in understanding how services fit into your applications and processes. If you have insights on how you have dealt with the challenge of service orientation in your work, please write and tell us about it!

Finally, I want to welcome Mike Calligaro and the Windows Mobile team to MSDN Magazine and introduce their new column "Going Places," focusing on all things mobile. As we continue to service-enable systems, those systems can reach far beyond the traditional network boundary, and Mike and his team will show us how to best leverage those capabilities on all sorts of platforms.

Thanks to the following Microsoft technical experts for their help with this issue: Tim Cahill, Barak Cohen, Alex DeJarnatt, Steve Fox, Matt Gibbs, Scott Hunter, Albert Kooiman, Steve Maine, Faisal Mohamood, Laurence Moroney, Mark Rideout, Stefan Schackow, Don Smith, Ed Triou, and Robert Wlodarczyk.


Toolbox: Logging Web App Errors, Learning LINQ, and More

Toolbox
Logging Web App Errors, Learning LINQ, and More
Scott Mitchell

Contents
Log and Review Web Application Errors
Blogs of Note
The Bookshelf

Log and Review Web Application Errors

In a perfect world there would be no errors in production. Database and Web servers would never go down, and code would be thoroughly tested and designed to gracefully handle any situation or input that might otherwise have caused an error. But we program in reality, where deadlines and never-ending feature requests are a constant and where database and Web servers can and do go offline. When errors occur in a production environment, it is essential that the details be logged and the development team promptly notified.

There are two common solutions for logging errors in an ASP.NET application: the ASP.NET health monitoring system and Atif Aziz's Error Logging Modules and Handlers (ELMAH). Both systems enable page developers to programmatically log the details of an error to a specified log source. Moreover, both offer mechanisms for automatically logging unhandled exceptions.

The health monitoring system from Microsoft is a robust, low-level framework that was introduced with ASP.NET 2.0. In addition to monitoring errors, it is capable of recording a variety of other metrics—including application lifecycle events such as application startup and shutdown, request-level events, and security-related events that include authorization violations and successful and failed login attempts.

When a monitored event occurs, it can be logged to any number of specified logging sources. The logging functionality in the health monitoring system is built atop the provider model, enabling developers to plug in a custom logging provider or to use any of the built-in providers. Among the built-in providers, there is one that records event details to the Windows® Event Log, another that writes the event to a SQL Server database, and one that sends the details to an e-mail address.

To begin using health monitor, simply add the appropriate configuration markup to the application's web.config file. The health monitoring configuration information must include the events to monitor, the logging providers, and how each monitored event is to be logged.

In addition to this basic information, the configuration markup can also include options such as whether the logging should be buffered, the minimum number of times an event must occur before logging commences, and the maximum number of times a given event should be recorded before future instances are ignored.

The other common logging option, ELMAH, is a set of open source HTTP handlers and modules. So while the ASP.NET health monitoring system can monitor a variety of events, ELMAH was designed to only log application errors. When an error occurs, ELMAH logs the details to a specified logging provider and, optionally, sends an e-mail to a configurable set of recipients.

There are a number of built-in and community-created ELMAH logging providers, including some that log to an XML file, to a SQL Server® database, to an Oracle database, and to the SQLite database (a lightweight, open source database engine).

ELMAH has some interesting display capabilities and supports all versions of ASP.NET. The ASP.NET health monitoring system does not provide any functionality for reviewing the logged events, but ELMAH includes built-in HTTP handlers that can display the error log from a Web page or as an RSS feed. And while health monitoring only works with ASP.NET 2.0 and beyond, ELMAH works with ASP.NET 1.0 and higher.

I first wrote about ELMAH in the August 2006 installment of Toolbox (see msdn.microsoft.com/msdnmag/issues/06/08/Toolbox). Since then, Atif has updated ELMAH to include a number of new features and performance enhancements such as exception filtering.

As I mentioned, by default, ELMAH logs all errors to the specified logging provider, however, now with exception filtering you can indicate under what circumstances an error should be dismissed (and therefore not logged). Furthermore, the latest version of ELMAH also gives you the ability to download the entire error log as a .csv file and provides improved performance for large error logs.

At their core, health monitoring and ELMAH provide the same functionality for logging errors in an ASP.NET application. Some developers prefer the health monitoring system because of its ability to monitor many different types of events. Others prefer ELMAH because of its built-in error log display capabilities and its support for ASP.NET version 1.x applications. In the long run, which error logging platform you choose is immaterial. What is essential, however, is that you have a rigorous error-logging system in place to record every single gory detail of those inevitable application errors.

Price: Free.

Health monitoring: go.microsoft.com/fwlink/?LinkId=109807

ELMAH: go.microsoft.com/fwlink/?LinkId=109809

Error Log in ELMAH
Error Log in ELMAH (Click the image for a smaller view)
Error Log in ELMAH
Error Log in ELMAH (Click the image for a larger view)

Blogs of Note

I recently was talking with a colleague and mentioned a post from Jeff Atwood's blog, Coding Horror. My colleague had never heard of the blog and asked me what it was about. The best I could offer was that it covered "computers and programming and stuff."

It is difficult to describe Coding Horror, as it spans a wide and deep variety of topics. There are very technical posts (usually about ASP.NET and Web-related technologies) alongside suggestions for becoming a better programmer and making better project management decisions. Other posts evaluate computer hardware, comparing brands of motherboards, mice, or monitors. There are posts about usability and design flaws in consumer gadgets and clever fare such as dieting and exercise for geeks, the future of blogging, the problem with EULAs, and the history of DONKEY.BAS.

Jeff definitely cuts to the core of some important industry issues. Check out his views on the state of higher education for software students today. As Jeff points out, you can study the theory of Turing machines or you can learn where to click in Visual Studio®, or anything in between. So what's the best approach if you plan to build software? I'm not going to tell you. You are going to have to read Jeff's blog to find out.

The blog is popular—more than 75,000 people subscribe to its RSS feed—so there are tons of comments. Plus Jeff offers many links to other prominent bloggers so you really do get a good sampling of opinions out there. The only downside is the lack of tools for exploring this wealth of content. There is a search box but no breakdown of posts by category.

www.CodingHorror.com

Jeff Atwood's Blog, Coding Horror
Jeff Atwood's Blog, Coding Horror (Click the image for a smaller view)
Jeff Atwood's Blog, Coding Horror
Jeff Atwood's Blog, Coding Horror (Click the image for a larger view)

The Bookshelf

One of the most important innovations in the Microsoft® .NET Framework 3.5 is LINQ. In short, LINQ enables developers to write C# and Visual Basic code that closely mirrors T-SQL's syntax, yet can be used to directly query any supported data source, including enumerable collections, XML documents, and SQL Server databases. For example, the following C# outputs the even numbers in the array defined here:

int[] numbers = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
var evens = from p in numbers
            where p % 2 == 0
            select p;

foreach (var e in evens)
    Console.WriteLine(e);

Due to the new language syntax and the breadth of LINQ's features and complexity, I have preferred learning about LINQ through books rather than by taking the time to hunt and peck for online articles and tutorials. And the first LINQ book I picked up was Pro LINQ in C# 2008 by Joseph C. Rattz Jr.

This book starts with a quick and intriguing introduction to LINQ, illustrating how it can be used to query collections, XML files, and SQL Server databases. This is followed by a brief explanation of the new language features that made LINQ possible: lambda expressions, anonymous types, object initializers, extension methods, automatic properties, and so forth.

The bulk of the book is divided into four sections covering LINQ to Objects, LINQ to XML, LINQ to DataSets, and LINQ to SQL. Each section offers a thorough examination of the LINQ syntax and the various classes and methods that enable LINQ to query these different data stores. The LINQ to Objects section serves as a technical reference to the standard query operators. These operators are enumerated and are also accompanied by a short description and programming example. The subsequent sections, however, offer a richer discussion of LINQ, including helpful tips, common pitfalls, and more involved real-world examples.

The last section, LINQ to SQL, is probably the most interesting and useful chapter for the majority of developers. Not surprisingly, one-third of the volume is devoted to LINQ to SQL and includes step-by-step instructions for using the Object Relational Designer and the SQLMetal.exe command-line tool to create the underlying SQL entity classes. The author also presents examples for using these classes along with the DataContext class to query, update, insert, and delete data. What's more, there are also chapters that address more complex database issues, such as concurrency control and transactions.

Price: $44.99.

www.apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. You can reach him via e-mail at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out: Measure Early and Often for Performance, Part 1

CLR Inside Out
Measure Early and Often for Performance, Part 1
Vance Morrison

Code download available at: CLRInsideOut2008_04.exe (205KB)
Contents
Have a Plan
Profile Data Processing Example
Measure Early and Measure Often
The Logistics of Measuring
Validating Your Performance Results
Validating Micro-Benchmark Data with a Debugger
Wrapping Up

A
s a Performance Architect on the Microsoft® .NET Framework Common Language Runtime team, it is my job to help people best utilize the runtime to write high-performance applications. The truth of the matter is that there is no mystery to this, .NET or otherwise—you just have to design applications for performance from the start. Too many applications are written with almost no thought given to performance at all. Often that's not a problem because most programs do relatively little computation, and they are much faster than the humans with which they interact. Unfortunately, when the need for high performance does present itself, we simply don't have the knowledge, skills, and tools to do a good job.

Here I'll discuss what you need to write high-performance applications. While the concepts are universal, I'll focus here on programs written for .NET. Because .NET abstracts the underlying machine more than a typical C++ compiler, and because .NET provides powerful but expensive features including reflection, custom attributes, regular expressions, and so forth, it is much easier to unwittingly inject expensive operations into a performance-critical code path. To help you avoid that expense, I'll show how to quantify the expense of various .NET features so you know when it's appropriate to use them.

Have a Plan

As I mentioned, most programs are written without much thought given to performance, but every project should have a performance plan. You must consider various user scenarios and articulate what excellent, good, and bad performance actually mean. Then, based on data volume, algorithmic complexity, and any previous experience building similar applications, you must decide if you can easily meet whatever performance goals you've defined. For many GUI applications, the performance goals are modest, so it's easy to achieve at least good performance without any special design. If this is the case, your performance plan is done.

If you don't know if you can easily meet your performance goals, you'll need to begin writing a plan by listing the areas likely to be bottlenecks. Typical problem areas include startup time, bulk data operations, and graphics animations.

Profile Data Processing Example

An example will make this more concrete. I'm currently designing the .NET infrastructure for processing profile data. I need to present a list of events (page faults, disk I/O, context switches, and so on) generated by the OS in a meaningful way. The data files involved tend to be large; small profiles are in the neighborhood of 10MB and file sizes well over 1GB are not unusual.

While working on forming my performance plan, I concluded that the display of the data would not be problematic if I computed only the parts of the dataset that were needed to paint the display; in other words, if display was "lazy." Unfortunately, it takes extra work to make GUI objects like tree controls, list controls, and textboxes lazy. This is why most text editors have unacceptable performance when file sizes get too large (say, for instance, 100MB). If I had designed the GUI without thinking about performance, the result would have almost certainly been unacceptable.

Laziness, however, does not help for operations that need to use all the data in the file (when computing summaries, for example). Given the dataset size, the data dispatch and processing methods are "hot" code paths that must be designed carefully. Most of the rest of the program is unlikely to be performance-critical and needs no special attention.

This experience is typical. Even in high-performance scenarios, 95 percent of the application does not need any performance planning, but you need to carefully identify the last 5 percent that does. Also, as in my case, it is usually pretty easy to determine which parts of the program are likely to be that 5 percent that matters.

Measure Early and Measure Often

The next step in high-performance design is to measure—before writing a line of code, you need to know whether your performance goals are even possible, and if so, what constraints they place on the design. In my case, I need to know the costs of basic operations being considered in my design, such as raw file I/O and database access. To proceed, I need some numbers. This is the most critical time in the design of the project.

Sadly, most performance is lost very early in the development process. By the time you have chosen the data structures for the heart of your program, the application's performance profile has been set in stone. Choosing your algorithms further limits performance. Selecting interface contracts between various sub-components constrains performance still further. It is critical that you understand the costs of each of these early design decisions and make wise ones.

Design is an iterative process. It is best to start with the cleanest, simplest, most obvious choice and work up a sketch of the design (I actually recommend a prototype of the hot code) and evaluate the performance of that. You should also think about what the design would look like if you were to make performance the only factor, and then estimate how fast that application would be. Now the fun engineering begins! You start tinkering with the design and thinking about alternatives between these two extremes, looking for designs that give you the best result.

Again, my experience with the profile data processor is instructive. Like most projects, my choice of data representation was critical. Should the data be in-memory? Should it be streamed over in a file? Should it be in a database? The standard solution is that any large dataset should be stored in a database; however, databases are optimized for relatively slow change, not for having large volumes of data changing frequently. My application would be dumping many gigabytes of data into the database routinely. Could the database handle this? With just a bit of measurement and analysis of database operations, it was easy to confirm that databases did not have the performance profile I needed.

After some more measurements on how much memory an application can use before inducing excessive page faulting, I ruled out the in-memory solution as well. That left streaming data from a file for my basic data representation.

There were still many other design decisions to be made, however. The basic form of the profile data is a list of heterogeneous events. But what should events look like? Are they strings (which are nicely uniform)? Are they C# structs or objects?

If they are objects, the obvious solution is to make one allocation per event, which is a lot of allocations. Is that acceptable? How exactly does dispatch work as I iterate over the events? Is it a callback model or an iteration model? Does dispatch work through interfaces, delegates, or reflection? There were dozens of design decisions to be made, and they all would have impact on the ultimate performance of the program, so I needed to take measurements to understand the tradeoffs.

The Logistics of Measuring

Clearly you will be doing a lot of measuring during design. So exactly how do you do that? There are many profiling tools that can help, but one general-purpose technique that's also the simplest and most available is micro-benchmarking. The technique is simple: when you want to know how much a particular operation costs, you simply set up an example of its use and directly measure how much time the operation takes.

The .NET Framework has a high-resolution timer called System.Diagnostics.Stopwatch that was designed specifically for this purpose. The resolution varies with your hardware, but it typically has a resolution of less than 1 microsecond which is more than adequate. Since this comes with the .NET Framework, you already have the functionality you need.

While Stopwatch is a great start, a good benchmark harness should do more. Small operations should be placed in loops to make the interval long enough to measure accurately. The benchmark should be run once before taking a measurement to ensure that any just-in-time (JIT) compilation and other one-time initialization has completed (unless, of course, the goal is to measure that initialization). Since measurements are noisy, the benchmark should be run several times and statistics should be gathered to determine the stability of the measurement. It should also be easy to run many benchmarks (design variations) in bulk and get a report that displays all the results for comparison.

I have written a benchmark harness called MeasureIt.exe which builds upon the Stopwatch class and addresses these goals. It is available with the code download for this column on the MSDN® Magazine Web site. After unpacking, simply type the following in order to run it:

MeasureIt

Within seconds it will run a set of more than 50 standard benchmarks and display the results as a Web page. An excerpt of the data is shown in Figure 1. In these results, each measurement performs an operation 10,000 times (the operation is cloned 10 times in a loop executed 1000 times). Each measurement is then performed 10 times and standard statistics (min, max, median, mean, standard deviation) are computed.

Figure 1 Measuring Various Operations with MeasureIt.exe (Close Figure 1)

Operation Measured | Median | Mean | StdDev | Min | Max | Samples
MethodCalls: EmptyStaticFunction() [count=1000 scale=10.0] | 1.000 | 1.005 | 0.084 | 0.922 | 1.136 | 10
MethodCalls: aClass.Interface() [count=1000 scale=10.0] | 1.699 | 1.769 | 0.090 | 1.696 | 1.943 | 10
ObjectOps: new Class() [count=1000 scale=10.0] | 6.248 | 8.040 | 3.556 | 5.087 | 16.296 | 10
Arrays: aIntArray[i] = 1 [count=1000 scale=10.0] | 0.616 | 0.638 | 0.071 | 0.612 | 0.850 | 10
Delegates: aInstanceDelegate() [count=1000 scale=10.0] | 1.233 | 1.244 | 0.088 | 1.160 | 1.398 | 10
PInvoke: FullTrustCall() [count=1000] | 7.452 | 6.946 | 0.804 | 5.878 | 7.913 | 10
Locks: Monitor lock [count=1000] | 11.487 | 12.129 | 0.901 | 11.322 | 13.843 | 10

To make the time measurements more meaningful, they are normalized so that the median time for calling (and returning) from an empty static function is one unit. It is not uncommon for benchmarks to have widely varying times, which is why all the statistical information is important. Note the large difference between the minimum (71.299) and maximum (953.864) times for the FinalizableClass benchmark. This variation needs to be explained before you can trust the data from that benchmark. In that particular case, it is the result of the runtime periodically executing slower code paths to allocate bookkeeping data structures in bulk. Already, having these statistics available is proving useful in validating the data.

This table is a gold mine of useful performance data, detailing the costs of most of the primitive operations used by .NET-targeted code. I will go into detail in my next installment of this column, but here I want to explain an important feature of MeasureIt: it comes with its own source code. To unpack MeasureIt's source code and launch Visual Studio® to browse it (if Visual Studio is available), type this:

MeasureIt /edit

Having the source means that you can quickly understand exactly what the benchmark is measuring. It also means that you can easily add a new benchmark to the suite.

Again, my experience with the profile data processor is instructive. At one point on the design, I could do a certain common operation with either C# events, delegates, virtual methods, or interfaces. To make a decision, I needed to understand the performance tradeoff among these choices. Within minutes I had written the micro-benchmark to measure the performance of each of the alternatives. Figure 2 displays the relevant rows and you can see that there is no substantial difference between the alternatives. This knowledge allows me to choose the most natural alternative, knowing I was not sacrificing performance to do so.

Figure 2 Measuring .NET Events, Delegates, Interfaces, and Virtual Methods (Close Figure 2)

Operation Measured | Median | Mean | StdDev | Min | Max | Samples
MethodCalls: aClass.Interface() [count=1000 scale=10.0] | 1.651 | 1.660 | 0.084 | 1.579 | 1.814 | 10
MethodCalls: aClass.VirtualMethod() [count=1000 scale=10.0] | 1.228 | 1.175 | 0.077 | 1.083 | 1.277 | 10
Delegates: aInstanceDelegate() [count=1000 scale=10.0] | 1.151 | 1.159 | 0.085 | 1.075 | 1.314 | 10
Events: Fire Events [count=1000 scale=10.0] | 1.228 | 1.195 | 0.070 | 1.088 | 1.291 | 10

Validating Your Performance Results

The MeasureIt application makes collecting data for a broad variety of benchmarks very easy. Unfortunately, MeasureIt does not address an important aspect of using benchmark data: validation. It is extremely easy to measure something other than what you thought you were measuring. The result is data that is simply wrong, and worse than useless. The old adage "if it sounds too good (or bad) to be true, it probably is" definitely applies to performance data. It is imperative that you validate data that you use in any important design decision.

Validating Micro-Benchmark Data with a Debugger

What does it mean to validate performance results? It means collecting other information that also will predict the performance result and seeing if the two methodologies agree. For very small micro-benchmarks, inspecting machine instructions and making an estimate based on the number of instructions executed is an excellent check. In a debugger like Visual Studio, it should be as easy as setting a breakpoint in your benchmark code and switching to the disassembly window (Debug -> Windows ->Disassembly). Unfortunately, the default options for Visual Studio are designed to simplify debugging, not to do performance investigations, so you need to change two options to make this work.

First, go to Tools | Options... | Debugging | General and clear the Suppress JIT Optimization checkbox. This box is checked by default, which means that even when debugging code that should be optimized, the debugger tells the runtime not to do so. The debugger does this so that optimizations don't interfere with the inspection of local variables, but it also means that you are not looking at the code that is actually run. I always uncheck this option because I strongly believe that debuggers should strive to only inspect, and not to change the program being debugged. Note that unsetting this option has no effect on code that was compiled for debugging since the runtime would not have optimized that code anyway.

Next, clear the Enable Just My Code checkbox from Tools | Options | Debugging | General dialog. The Just My Code feature instructs the debugger not to show you code that you did not write. Generally, this feature removes the clutter of call frames that are often not of interest to the application developer. However, this feature assumes that any code that is optimized can't be yours (it assumes your code is compiled using the debug configuration or suppressed JIT Optimizations is turned on). If you allow JIT optimizations but don't turn off Just My Code, you will find that you never hit any breakpoints because the debugger does not believe your code is yours.

Once you have unchecked these options, they remain unchecked for ALL projects. Generally this works out well, but it does mean that you don't get the Just My Code feature. You may find yourself switching Just My Code on and off as you go from debugging to performance evaluation and back.

As an example of using a debugger to validate performance results, you can investigate an anomaly in the data shown in the excerpt in Figure 3. This data shows that calls to an interface method of a C# structure is many times faster than a call to a static method. This certainly seems odd, given that you would expect a static method call to be the most efficient type of call. To investigate this, you set a breakpoint in this benchmark and run the application. Switch to the disassembly window (Debug -> Windows -> Disassembly) and see that the whole benchmark consists of just the following code:

Figure 3 Using a Debugger to Validate Performance Results (Close Figure 3)

Operation Measured | Median | Mean | StdDev | Min | Max | Samples
MethodCalls: EmptyStaticFunction() [count=1000 scale=10.0] | 1.000 | 0.964 | 0.102 | 0.857 | 1.196 | 10
MethodCalls: aStructWithInterface.Interface() [count=1000 scale=10.0] | 0.031 | 0.029 | 0.012 | 0.021 | 0.039 | 10

aStructWithInterface.InterfaceMethod();
00000000  ret

What this shows is that the benchmark (which is 10 calls to an interface method) has been inlined away to be nothing. The ret instruction is actually the end of the delegate body that defines the whole benchmark. Well, it is not surprising that doing nothing is faster than doing method calls, so this shows the reason for the anomaly.

The only mystery is why static methods don't get inlined, too. This is because for static methods, I specifically went out of my way to suppress inlining with the MethodImplOptions.NoInlining attribute. I intentionally "forgot" to put this on this interface call benchmark to demonstrate that the JIT compiler can make certain interface calls as efficient as non-virtual calls (there is a comment mentioning this above the benchmark).

Wrapping Up

To reiterate, it is very easy to measure something other than what you intended, especially when measuring small things that are subject to JIT compiler optimizations. It is also very easy to accidentally measure non-optimized code, or measure the cost of JIT compilation of a method rather than the method itself. The MeasureIt /usersGuide command will bring up a user's guide that discusses many of the pitfalls you might encounter when creating benchmarks. I strongly recommend that you read these details when you are ready to write your own benchmarks.

The point that I want to stress is this concept of validation. If you can't explain your data, you should not use it for making design decisions. If you have unusual data, ideally you should collect more data, debug the benchmarks, or collaborate with others who have more expertise until you can explain your data. You should be highly suspicious of unexplainable data, and should not use it in making any important decisions.

This discussion is about the basics of writing high-performance applications. Like any other attribute of software, good performance needs to be designed into the product from the beginning. To do this, you need measurements that quantify the tradeoffs of making various design decisions. This means doing performance experiments. MeasureIt makes it easy to generate good-quality micro-benchmarks quickly, and as such it should become an indispensable part of your design process. MeasureIt is also useful out of the box because it comes with a set of benchmarks that cover most of the primitive operations in the .NET Framework.

You can also easily add your own benchmarks for the part of the .NET Framework that most interests you. With this data you can form a model of application costs and thus make reasonable (rough) guesses about the performance of design alternatives even before you have written application code.

There is a lot more to say about the performance of applications in .NET. There are potential pitfalls associated with building micro-benchmarks, so please do read the MeasureIt users guide before writing any. I have also deferred discussion about situations where disk I/O, memory, or lock contention is the important bottleneck. I have not even discussed how to use various profiling tools to validate and monitor the performance health of your application after it has been designed.

There is a lot to know, and the sheer volume of information often discourages developers from doing any such testing at all. However, since most performance is lost in the design of an application, if you do nothing else, you should think about performance at this initial stage. I hope this column will encourage you to make performance an explicit part of the design on your next .NET software project.

Send your questions and comments to clrinout@microsoft.com.

Vance Morrison is the Compiler Architect for the CLR team at Microsoft, where he has been involved in the design of .NET since its inception. He drove the design for the .NET Intermediate Language (IL) and was lead for the just-in-time (JIT) compiler team.


Basic Instincts: My Namespace Extensions with My Extensibility

Basic Instincts
My Namespace Extensions with My Extensibility
Matthew DeVore

Code download available at: BasicInstincts2008_04.exe (163KB)
Contents
Ways to Expand the My Namespace
Expanding My with a Singleton
Miscellaneous My Extension Scenarios
Coding the Extension
Packaging the Extension into a Template
My Extensions as Templates
The .customdata File
Try It!
Publish the Extension in a Visual Studio Installer File

T
he My namespace feature was introduced in Visual Basic® 2005 to provide shortcut methods and APIs for common coding tasks. Since then, users have been able to write My namespace extensions to make it easy to access their own code libraries. My Extensibility, new in Visual Basic 2008, makes extending the My namespace even easier.

With the new My Extensibility feature, My namespace extensions can be activated or deactivated through the Project Properties Designer, or when an associated reference is added or removed from a project. This capability makes extending the Visual Basic development environment simple through deploying APIs for common coding tasks.

In this installment, I will focus on how to integrate your extension with the My Extensibility feature. If you find yourself wanting more information on how to write the actual code extensions to the My namespace, please read "Simplify Common Tasks by Customizing the My Namespace" in the July 2005 issue of MSDN® Magazine (msdn.microsoft.com/msdnmag/issues/05/07/My).

Ways to Expand the My Namespace

There are a few code models you can use to expand My. Let's start with the easiest one, the one you already know how to use. In fact, adding anything to the My namespace is very much like adding something to any other namespace. Here is the simplest example:

Namespace My.HandyStuffForMy
    <Global.Microsoft.VisualBasic.HideModuleName()> _
    Friend Module HandyStuffForMyModule
        Sub Foo()
            ...
        End Sub

        Property Bar()
            ...
        End Property
    End Module
End Namespace

This will cause a sub called Foo and a property called Bar to appear in a namespace called HandyStuffForMy within the My namespace. There are a couple of things here that you should notice: the HideModuleName attribute and the Friend access modifier on the Module. The HideModuleName attribute will prevent the Module name from showing up in IntelliSense®. Because you don't need to type Module names in order to refer to their members, showing the Module would only clutter up IntelliSense (see Figure 1).

Figure 1 Sub Namespace in IntelliSense
Figure 1 Sub Namespace in IntelliSense

If you would like to add these members directly to My without putting them in a sub namespace (see Figure 2), change Namespace My.HandyStuffForMy to just Namespace My, like the following:

Namespace My
    <Global.Microsoft.VisualBasic.HideModuleName()> _
    Friend Module HandyStuffForMyModule
        Sub Foo()
            ...
        End Sub

        Property Bar()
            ...
        End Property
    End Module
End Namespace

Figure 2 No Sub Namespace
Figure 2 No Sub Namespace

Additionally, if you use a namespace that already exists in My, such as Resources, these members will be added to that namespace. This gives you a lot of flexibility for how your extensions are accessed. The second thing to be aware of is the access modifier on the module. The Friend access modifier will prevent other assemblies from accessing these members. If you use Public access, the elements may conflict with symbols declared in these other project's My namespace.

I'll set up the sample My Extension in a moment using the strategy I just outlined. However, not everything in My is a namespace. There are also Properties that return global instances of certain classes, also known as singletons.

Expanding My with a Singleton

My.Computer, My.Settings, and My.Application are singleton instances of classes called My.MyComputer, My.MySettings, and My.MyApplication, respectively. We can easily add members to these classes using the partial class feature. For instance, the following code adds a sub called Shutdown to the My.Computer object:

Namespace My
    Partial Class MyComputer
        Sub Shutdown()
            ...
        End Sub
    End Class
End Namespace

This code can be used for expanding My.Application and My.Settings. Partial classes let you add members to a class that is defined somewhere else as long as the source code for that class exists in the current project. This includes classes like MyComputer for Visual Basic projects, as the compiler injects the classes automatically before the code is compiled.

What if you want to allow your added elements to be accessed via a singleton, just like My.Computer, My.Settings, and My.Application, but you want to create your own? This involves adding a ReadOnly Property to the My namespace using the strategy I introduced at the beginning of this column and then having that Property return the global instance as supplied by the ThreadSafeObjectProvider(Of T) class. Comparing this to the previously mentioned new namespace strategy, this would cause a new instance of the class to be generated on a per-thread basis. Figure 3 shows how you would add a global object called My.CustomMyObject with members named Foo and Bar. Figure 4 shows the resulting IntelliSense experience in the IDE.

Figure 3 Adding My.CustomMyObject (Close Figure 3)
Namespace My
  <Global.System.ComponentModel.EditorBrowsable _
  (Global.System.ComponentModel.EditorBrowsableState.Never)> _
  Partial Friend Class MyCustomMyObject
    Public Sub Foo()
            ...
    End Sub

    Public Property Bar()
            ...
        End Property
  End Class

  <Global.Microsoft.VisualBasic.HideModuleName()> _
  Friend Module CustomMyObjectModule
    Private instance As ThreadSafeObjectProvider(Of MyCustomMyObject)

    ReadOnly Property CustomMyObject() As MyCustomMyObject
      Get
        Return instance.GetInstance()
      End Get
    End Property
  End Module
End Namespace
Figure 4 My.CustomMyObject in IntelliSense
Figure 4 My.CustomMyObject in IntelliSense

Miscellaneous My Extension Scenarios

Extending My.User is probably the most difficult scenario. It is defined outside of your project, in the Visual Basic runtime DLL, which means you cannot extend it with partial classes. In order to extend My.User, a project-level compilation constant must be defined, which means that extending My.User requires that the developer using the extension execute this manual step.

In any case, if you want code examples and guidance for extending My.User, please refer to the aforementioned MSDN Magazine article, which also contains information on specialized scenarios, such as expanding My.Application.Log, expanding My.Computer.Network, and customizing the way My.Settings stores application configuration information.

What about objects such as My.WebServices and My.Forms? The classes of these objects are nested types defined in Modules, which means they cannot be used with the Partial Class feature, and they are NotInheritable, which means they cannot be extended like My.User. There is no easy way to add members to them. In this case, you must find a better way for your extension to fit into the My namespace.

Coding the Extension

Now that I've explained extensions to the My namespace, let's see how you can easily implement and share these extensions in our projects. I just showed how My is a namespace like any other namespace, and you can add top-level properties to it by simply adding a module and specifying a namespace of My. Let's now create a set of validation functions that can check if a string matches the format of U.S. and Canadian postal codes, phone numbers, e-mail addresses, and a few other important formats.

Let's begin by starting Visual Studio® and creating a new console application. Name the project MyValidation. Next, add a new Module to the solution called MyValidation.vb and replace the code in the MyValidation.vb file with the code shown in Figure 5 (which has been excerpted here for space).

Figure 5 Validation Code Excerpt (Close Figure 5)
Imports System.Text.RegularExpressions
Imports System
Imports System.Linq
Imports Microsoft.VisualBasic

Namespace My.Validation
  <Global.Microsoft.VisualBasic.HideModuleName()> _
  Public Module MyValidationMod
    Public Function IsEmpty(ByVal value As Object) As Boolean
      If (value Is Nothing) OrElse _
         (value Is System.DBNull.Value) OrElse _
         (value.ToString = String.Empty) OrElse _
         (TypeOf value Is Date AndAlso CDate(value) = Date.MinValue) _
      Then
        Return True
      Else
        Return False
      End If
    End Function

    Public Function IsAlphaNumeric(ByVal value As String, _
     Optional ByVal emptyOK As Boolean = False, Optional ByVal _
     whitespaceOK As Boolean = False) As Boolean
      If IsEmpty(value) Then Return emptyOK

      Dim expr As String
      If whitespaceOK Then
        expr = "^[a-zA-Z0-9\s]+$"
      Else
        expr = "^[a-zA-Z0-9]+$"
      End If

      Return Regex.IsMatch(value, expr)
    End Function

...

    Public Function IsCanadianProvince(ByVal st As String) As Boolean
      Dim allProv = "|AB|BC|MB|NB|NL|NT|NS|NU|ON|PE|QC|SK|YT"
      Return st.Length = 2 AndAlso allProv.IndexOf("|" & st) <> -1
    End Function

    Public Function IsUSAState(ByRef st As String) As Boolean
      Dim allStates = "|AL|AK|AZ|AR|CA|CO|CT|DE|FL|GA|HI|ID|IL|IN|IA|" & _
        "KS|KY|LA|ME|MD|MA|MI|MN|MS|MO|MT|NE|NV|NH|NJ|NM|NY|NC|ND|OH|" & _
        "OK|OR|PA|RI|SC|SD|TN|TX|UT|VT|VA|WA|WV|WI|WY"
      Return st.Length = 2 AndAlso allStates.IndexOf("|" & st) <> -1
    End Function

    Public Function IsUSAPostalCode(ByVal zip As String) As Boolean
      If String.IsNullOrEmpty(zip) Then Return False

      Return Regex.IsMatch(zip, "^\d{5}(-\d{4})?$")
    End Function

    Public Function IsCanadianPostalCode(ByVal zip As String) As Boolean
      If String.IsNullOrEmpty(zip) Then Return False

      Return Regex.IsMatch(zip, "^[A-Z]\d[A-Z] \d[A-Z]\d$")
    End Function

    Public Function IsNorthAmericanPhone(ByRef phone As String) As Boolean
      If String.IsNullOrEmpty(phone) Then Return False

      Dim expr As String = "^(?:\([2-9]\d{2}\)\ ?|[2-9]\d{2}" & _
                           "(?:\-?|\ ?))[2-9]\d{2}[- ]?\d{4}$"

      Return Regex.IsMatch(phone, expr)
    End Function

    Public Function IsEmail(ByRef email As String) As Boolean
      If String.IsNullOrEmpty(email) Then Return False

      Dim localPartCharset = "[0-9a-zA-Z!#$%*/?|\^{}`~&'+\-=_]"
      Dim domainPartCharset = "[0-9a-zA-Z\-]"
      Dim expr = String.Format("^{0}+(\.{0}+)*@{1}+(\.{1}+)*", _
        localPartCharset, domainPartCharset)

      Return Regex.IsMatch(email, expr)
    End Function
  End Module
End Namespace

Be careful when creating a My Extension template that your code can work in any project configuration. You need to pay particular attention to the possible values of Option Explicit, Option Strict, Option Compare, and (new in Visual Basic 2008) Option Infer. You can either write code that will work in the least-flexible configuration (Option Explicit On, Option Strict On, Option Infer Off, and Option Compare unspecified), or specify each option explicitly at the top of each code file in your extension, which is the recommended approach.

Also, you need to be careful to account for different project imports to prevent conflict with symbols defined in the project. You do this by qualifying any symbol not defined in your template with the Global or the My keyword. For instance, instead of Text.Encoding, use Global.System.Text.Encoding. If you only use Text.Encoding, your template will not work in Windows® Forms Applications, which imports two namespaces that contain a namespace called Text. If you omit Global and only use System.Text.Encoding, then your template may not work in a project that defines a namespace called System.

It's always a good idea to thoroughly test your code before distributing it to others, and My extensions are no different. Let's use the automatically created module (Module1.vb) to write a test for the My Extension. It becomes a hassle to re-export the extension when you find a bug, so it is best to make the extension as functional as possible and make sure it's properly tested before you export it for the first time.

Open Module1.vb by double-clicking it in the Solution Explorer and place the test code, which has been excerpted in Figure 6, into the file, replacing the code that was there before. For the full code, see the code download on the MSDN Magazine Web site. Now you are ready to test the My Extension, so press F5 to execute the project and attach the debugger. You should see the following output:

Figure 6 Test Code Excerpt (Close Figure 6)
Module Module1
  Private testNumber As Integer
  Private testGroup As String

  Sub StartTestGroup(ByVal name As String)
    testGroup = name
    testNumber = 0
  End Sub

  Sub Test(ByVal result As Boolean, ByVal expected As Boolean)
    testNumber += 1

    If result <> expected Then
      Console.WriteLine("Test #{0} of group {1} FAILED.",  _
        testNumber, testGroup)
    End If
  End Sub

  Sub Main()

    StartTestGroup("IsAlphaNumeric")
    Test(My.Validation.IsAlphaNumeric("123HH2", True, True), True)

    ...

    StartTestGroup("IsCanadianPostalCode")
    Test(My.Validation.IsCanadianPostalCode("H0H 0H0"), True)

    ...

    StartTestGroup("IsCanadianProvince")
    Test(My.Validation.IsCanadianProvince("ON"), True)

    ...

    StartTestGroup("IsEmail")
    Test(My.Validation.IsEmail("a@b.com"), True)

    ...

    Console.WriteLine _
      ("Tests are finished. No FAILED messages indicates success.")
    Console.WriteLine("Press any key to continue.")

    Console.ReadKey(True)
  End Sub
End Module

Tests are finished. No FAILED messages indicates success.
Press any key to continue.

Packaging the Extension into a Template

Now that the extension is written and properly tested, it's time to package it up into a My Extension template. With the project open, click the File menu, then select Export template.... If you don't see this option on the File menu, right-click on the toolbar and select Customize, then select the Commands tab, choose the File category, then drag the Export Template command onto the File menu. Once you export the template, the Export Template Wizard appears. Select Item template (see Figure 7). My Extensions are always item templates—never project templates.

Figure 7 Choose Template Type
Figure 7 Choose Template Type(Click the image for a smaller view)
Figure 7 Choose Template Type
Figure 7 Choose Template Type (Click the image for a larger view)

You can select the project from which the template will be created; in this solution there is only one project, so just click Next. In the Items to Export listbox, select just the MyValidation.vb file and click Next. On the next step, you can select which references you would like to include with this item. You have to select all the references required for the My Extension to operate once it has been added to a project. For this example, select System.Core, then click Next again.

In the last step, fill in the Template description textbox; for instance, "Extends the My namespace with a set of string validation methods." Finally, uncheck the checkbox "Automatically import the template into Visual Studio." This setting ensures that the template won't be imported right away. You're going to need to modify a few things in the template before it's ready to use as a My Extension, so just click Finish.

My Extensions as Templates

In a My Extension template, as in any Visual Studio template, all of the template parameters are available. These are special strings that can be inserted into the My Extension code automatically when it is added to a project. For instance, $clrversion$ will be replaced with the current version of the CLR. In the case of My Extension templates, $safeitemname$ will be replaced with the value of the <DefaultName> element in the .vstemplate file, minus the file extension. This will be equal to the name of the .vb file containing the template code.

Anything else that you can do with a regular template can also be done with a My Extension template. For instance, within the .vstemplate file, by setting the value of the <Hidden> element to false, you can include the template in the Add New Item Dialog when adding items to projects. You can also force the addition of assembly references with the <References> section, and you can add a family of code files as is done by the Windows Form template to include the form's design code as well as the user code.

For more information on configuring the template through the .vstemplate file, see the "VSTemplate Element" reference that is available in the MSDN Library at go.microsoft.com/fwlink/?LinkId=110905. For more information on template parameters, see the "Visual Studio Template Parameters" documentation at go.microsoft.com/fwlink/?LinkId=110907.

There are a few key differences between typical item templates and My Extensibility item templates. In order to enable the Item Template so it can be managed from the My Extensions page of the Visual Basic Project Designer, you must take special steps to configure it properly.

First you need to add a <CustomDataSignature> element to the MyTemplate.vstemplate file located in the exported template .zip file. This indicates that the Item Template is to be treated as a My Extension. You also need to prevent the items in the template from being opened automatically when the template is added to the project by adding a couple of attributes onto the <ProjectItem> element. This is because My Extension template code should be invisible to the consumer. In addition, you must add the <Hidden> element into the MyTemplate.vstemplate to indicate that the template should not be displayed in the list of installed templates when the user elects to add items to his project through the Add New Item dialog. Finally, you need to add a .customdata file to the package in order to provide information that does not apply to typical templates, namely a unique ID, a version number, and a trigger assembly.

To do this, you will have to unpack the .zip file, perform the alterations, and then repack the necessary files. So, first unzip the template's contents to an empty directory so you can edit them. Visual Studio will place the file in %MyDocs%\Visual Studio 2008\My Exported Templates, where %MyDocs% is your My Documents folder.

Open the MyTemplate.vstemplate file in a text or XML editor and make the edits that you see highlighted in red in Figure 8. Remember that XML, unlike Visual Basic, is case sensitive.

Figure 8 My Extensibility Item Template (Close Figure 8)
<VSTemplate Version="2.0.0"
   xmlns=http://schemas.microsoft.com/developer/vstemplate/2005
   Type="Item">
  <TemplateData>
    <DefaultName>MyValidation.vb</DefaultName>
    <Name>MyValidation</Name>
    <Description>Extends the My namespace with a set of string
       validation methods.</Description>
    <ProjectType>VisualBasic</ProjectType>
    <SortOrder>10</SortOrder>
    <Icon>TemplateIcon.ico</Icon>

    <!-- Configures the template to not be shown as an option
      when the user selects to add an item template to his project -->
    <Hidden>true</Hidden>

    <!-- Indicates it is to be treated as a My Extension. -->
    <CustomDataSignature>Microsoft.VisualBasic.MyExtension
    </CustomDataSignature>

  </TemplateData>
  <TemplateContent>
    <References>
      <Reference> <Assembly>System.Core</Assembly> </Reference>
    </References>
    <ProjectItem SubType="Code" TargetFileName="$fileinputname$.vb"
      ReplaceParameters="true" OpenInEditor="false"
      OpenInWebBrowser="false" OpenInHelpBrowser="false">MyValidation.vb
    </ProjectItem>
  </TemplateContent>
</VSTemplate>

Create a new file to be packed into the .zip file called MyValidation.customdata. Note that any file that ends with .customdata will work. In Notepad or any text editor, add the following markup:

<VBMyExtensionTemplate
    ID="MyValidationMyNamespaceExtension"
    Version="1.0.0.0" />

The AssemblyFullName attribute, which is omitted in this example, indicates which assembly will trigger the My Extension. In other words, you can specify here that whenever this assembly is added to the project, the user will be invited to also add the My Extension. Whenever this assembly is removed from the project, and the My Extension is currently active, the user will be invited to remove the My Extension. To specify a trigger assembly, such as System.Windows.Forms, you would insert this text into the .customdata file:

<VBMyExtensionTemplate
    ID="MyValidationMyNamespaceExtension"
    Version="1.0.0.0"
    AssemblyFullName="System.Windows.Forms" />

Now you are ready to copy the files back into the template .zip file. You may be tempted to edit the template's icon, but the icon of My Extension templates are not shown to the user, so I recommend leaving it as is. Copy the four files back into the .zip file using Windows Explorer. Then copy the .zip file to %MyDocs%\Visual Studio 2008\Templates\ItemTemplates\, where %MyDocs% is your My Documents folder.

The .customdata File

The handling of My Extensions within a project is a largely hidden and automated process. For Visual Studio to handle these files smoothly with little input from the user, the Project file, which has a .vbproj extension, has some new attributes. You can see them if you open them in a text editor the .vbproj file of the project to which you added the MyValidation extension. You will see something like this:

<ItemGroup>
  ...
  <Compile Include="My Project\MyExtensions\MyValidation.vb">
    <VBMyExtensionTemplateID>
      MyValidationMyNamespaceExtension
    </VBMyExtensionTemplateID>
    <VBMyExtensionTemplateVersion>
      1.0.0.0
    </VBMyExtensionTemplateVersion>
  </Compile>
  ...
</ItemGroup>

But if you take a look at the entry for a typical code file, such as Module1.vb, you will only see this:

<ItemGroup>
  <Compile Include="Module1.vb" />
</ItemGroup>

As you can see, there are a few differences. First, the MyValidation.vb file is stored in the My Project folder, which means it is not visible unless the user presses the Show All Items button in the Solution Explorer pane. Additionally, the MyValidation.vb file has two extra pieces of metadata, which correspond to the values you added in the MyValidation.customdata file.

By storing this information here, Visual Studio can allow the user to add and remove the extension through the My Extensions Project Designer tab rather than adding or removing the file directly. Additionally, if the reference to the trigger assembly is removed, Visual Studio can warn the user that the My Extension may stop working and prompt you to remove the template. This information also enables Visual Studio to prevent two copies of the Extension from existing in the same project.

Try It!

The My Extension is now ready for integrated use in the My Extensibility feature. To use it, first start a new instance of Visual Studio and create a new project. Then, open the Project Designer by right-clicking on the Project and selecting Properties or double-clicking on My Project in the Solution Explorer. On the My Extensions tab, click Add Extension and select the template you just created, and then click OK. The extension is now loaded into the project. If you type My. in your code (notice the dot), you will now see Validation as one of the namespaces available through IntelliSense.

Publish the Extension in a Visual Studio Installer File

In order to make it much easier to distribute your extension to other developers, you can create an installer file. A Visual Studio Installer file is a .zip file renamed to have an extension of .vsi. In the file are the Visual Studio add-on and a .vscontent file that describes it. Let's expand on the sample and create a .vsi file for the My Extension. First, create a text file named MyValidation.vscontent with the text in Figure 9 in it to describe the content.

Figure 9 MyValidation Installer (Close Figure 9)
<VSContent xmlns="http://schemas.microsoft.com/developer/vscontent/2005">
    <Content>
        <FileName>MyValidation.zip</FileName>
        <DisplayName>My.Validation Extension</DisplayName>
        <Description>Extends the My namespace with a set of string
           validation methods.</Description>
        <FileContentType>VSTemplate</FileContentType>
        <ContentVersion>1.0</ContentVersion>
        <Attributes>
            <Attribute name="TemplateType" value="Item"/>
        </Attributes>
    </Content>
</VSContent>

When you create a different My Extension, the only elements likely to change are the FileName, DisplayName, Description, and ContentVersion values. For more information on creating a Visual Studio Installer file, as well as in-depth explanations of these elements, see the MSDN Library topic at go.microsoft.com/fwlink/?LinkId=110908.

Put this MyValidation.vscontent and the MyValidation.zip file into another .zip file called MyValidationInstall.zip. Then rename MyValidationInstall.zip to MyValidationInstall.vsi. Now all you have to do is double-click on it in order to install the template into Visual Studio automatically. Note that you will be prompted to overwrite the template if you already have one with the same name. If you've been following along that's exactly what will happen, so overwriting the file should be fine. Now you can distribute this .vsi file in order to easily install your My Extension on other developers' machines.

Send your questions and comments to instinct@microsoft.com.

Matthew DeVore has one-and-a-half years of experience on the Visual Basic Language QA team, working on the latest evolution of the language he first learned to program in, QBasic. In Visual Basic 2008, Matt tested Anonymous Types, My Extensibility, and, to a lesser extent, Lambda Expressions.


Cutting Edge: ListView Tips and Tricks

Cutting Edge
ListView Tips and Tricks
Dino Esposito

Contents
Building a Hierarchical Menu
Building a Hierarchical View
Improve the User Experience with Extenders
DataPager and ListView
Multiple Item Templates
Summing It Up

I
n last month's issue, you were introduced to the ListView control, a new addition to the ASP.NET 3.5 control toolbox. To recap, the ListView is an enhanced version of the DataList control that provides more control over generated markup, support for paging, and full integration with the data source-based binding model.

In this column, I'll move beyond the basics of ListView templates and data binding to implement features that are fairly common in real-world pages but require some extra coding. You'll see how to use nested ListView controls to create hierarchical views of data and how to extend the eventing model of the ListView by deriving a custom ListView class.

In particular, I'll refine the eventing model so you can use a different template for different groups of bound data items. For example, you'll be able to use a different template for all data items in a data set that match some given criteria. This will go far beyond simply styling certain items differently; you can do that easily in any view control by simply handling the ItemDataBound event.

Usually menus are implemented as a sequence of <li> tags that are styled using CSS. Rendering a flat menu doesn't pose any particular binding problems, but what happens when you need one or more sub-menus? In this case, you can either use the built-in Menu control or create a more custom rendering strategy by using a ListView. Incidentally, also note that by default the Menu control uses a table-based output unlike the more CSS-friendly output that you can obtain with a ListView. (To obtain a CSS-friendly output for a Menu control, you need to install and configure the CSS Control Adapter Toolkit, which you can download from www.asp.net.)

Building a Hierarchical Menu

Many Web apps offer a vertical menu on the left or right side of the page. This menu enables the user to navigate to pages two or more nesting levels deep. The ASP.NET Menu control is definitely a viable option here. However, I tend to use the Menu control only when I have a hierarchical data source (typically, an XML file) to feed the menu and when I need to create fly-out sub-menus.

For a static multi-level list of items, I opt for a repeater-like control to output markup created by a UI design team. In ASP.NET 3.5, the repeater-like control of choice is the ListView control.

Consider a menu like the one in Figure 1. It's shown in the CoffeeNCream free HTML template you can download from oswd.org. In the sample page, I simply incorporated the HTML markup in an ASP.NET Master Page.

Figure 1 A Standard Menu
Figure 1 A Standard Menu(Click the image for a smaller view)
Figure 1 A Standard Menu
Figure 1 A Standard Menu (Click the image for a larger view)

The HTML source code of an item of the right-side menu looks like the following:

<h1>Something</h1>
<ul>
    <li><a href="#">pellentesque</a></li>
    <li><a href="#">sociis natoque</a></li>
    <li><a href="#">semper</a></li>
    <li><a href="#">convallis</a></li>
</ul>

As you can see, it contains a top-level string followed by a list of links. You use a first ListView to create the H1 elements and then a nested ListView (or a similar data-bound control) to render the list of links. As a first step, you need to grab data to populate the menu. Ideally, you would use a collection of the following pseudotype objects to generate each item:

class MenuItem {
  public string Title;
  public Collection<Link> Links;
}

class Link  {
  public string Url;
  public string Text;
}

A reasonable method for populating a MenuItem collection is to render information from an XML file. Here's a possible schema for the document:

<Data>
  <RightMenuItems>
     <MenuItem>
       <Title>Something</Title>
       <Link url="..." text="pellentesque" />
         :
     </MenuItem>
  </RightMenuItems>
</Data>

The following demonstrates how to use LINQ to XML to load and process the content:

var doc = XDocument.Load(Server.MapPath("dataMap.xml"));
var menu = (from e in doc.Descendants("RightMenuItems")
            select e).First();
var menuLinks = from mi in menu.Descendants("MenuItem")
                select new
                {
                   Title = mi.Value,
                   Links = (...)
                };

After loading the document, you select the first node named RightMenuItems and then grab all of its MenuItem children. The content of each MenuItem node is loaded into a new anonymous type with two properties—Title and Links. How would you populate the Links collection? Here's some code:

Links = (from l in mi.Descendants("Link")
         select new {Url=l.Attribute("url").Value,
                     Text=l.Attribute("text").Value})

The next step is to bind this compound data to the user interface. As mentioned, you use an outer ListView to render the title and a second, nested ListView to render the list of child links (see Figure 2). Note that the innermost ListView must be bound to data using the Eval method—any other approach will not work:

Figure 2 Nested ListViews (Close Figure 2)
<asp:ListView runat="server" ID="RightMenuItems"
  ItemPlaceholderID="PlaceHolder2">
  <LayoutTemplate>
      <asp:PlaceHolder runat="server" ID="PlaceHolder2" />
  </LayoutTemplate>

  <ItemTemplate>
    <h1><%# Eval("Title") %></h1>

      <asp:ListView runat="server" ID="subMenu"
        ItemPlaceholderID="PlaceHolder3"
          DataSource='<%# Eval("Links") %>'>
          <LayoutTemplate>
            <ul>
              <asp:PlaceHolder runat="server" ID="PlaceHolder3" />
            </ul>
          </LayoutTemplate>
          <ItemTemplate>
            <li>
              <a href='<%# Eval("Url") %>'><%# Eval("Text") %></a>
            </li>
          </ItemTemplate>
      </asp:ListView>
  </ItemTemplate>
</asp:ListView>

<asp:ListView runat="server" ID="subMenu"
    ItemPlaceholderID="PlaceHolder3"
    DataSource='<%# Eval("Links") %>'>
    ...
</asp:ListView>

You start the data binding process by attaching data to the top-level ListView. When this happens, the body of the ListView is rendered entirely, including the nested ListView. You could, in theory, intercept the ItemDataBound event of the parent ListView, walk your way through the control tree, grab a reference to the child ListView, and bind it programmatically to data. If you do, you won't throw an exception, but the binding command on the inner ListView is lost because it fires too late to affect the rendering. A data-binding expression, on the other hand, is automatically evaluated during any data-binding event at just the right point in the control lifecycle. This ensures that the correct data is bound properly to the user interface.

Building a Hierarchical View

The same model for populating a hierarchical menu can be employed for building any hierarchical view of data. In this case, an alternate option would be to use a TreeView control to prepare a multi-level representation of the data. Data binding on a TreeView control, though, requires a hierarchical data source. Using nested ListView controls gives you more flexibility with regard to both the structure of the data source and the resulting user interface. Let's expand on these concepts.

Suppose you need to create a hierarchical grid of data where customers, orders, and order details are displayed according to existing table relationships. How would you retrieve the data and bind it to the control? Have a look at the code in Figure 3. You can use LINQ to SQL to easily load data into an object model that naturally lends itself to containing a hierarchy of data. Note that when you run a query in LINQ to SQL, you actually retrieve only the data you requested explicitly. Put another way, only the first level of the graph is fetched; any related objects are not automatically loaded at the same time.

Figure 3 Loading the Proper Data (Close Figure 3)
Public Class DataCache
{
    public IEnumerable GetCustomers()
    {
        NorthwindDataContext db = new NorthwindDataContext();
        DataLoadOptions opt = new DataLoadOptions();
        opt.LoadWith<Customer>(c => c.Orders);
        opt.LoadWith<Order>(o => o.Order_Details);
        db.LoadOptions = opt;

        var data = from c in db.Customers
                   select new { c.CompanyName, c.Orders };

        return data.ToList();
    }

    public int GetCustomersCount()
    {
        // Return the number of customers
        NorthwindDataContext db = new NorthwindDataContext();
        return db.Customers.Count();
    }

    public IEnumerable GetCustomers(int maxRows, int startRowIndex)
    {
        if (maxRows < 0)
            return GetCustomers();

        NorthwindDataContext db = new NorthwindDataContext();
        DataLoadOptions opt = new DataLoadOptions();
        opt.LoadWith<Customer>(c => c.Orders);
        opt.LoadWith<Order>(o => o.Order_Details);
        db.LoadOptions = opt;

        var data = (from c in db.Customers
                    select new {
                      c.CompanyName,
                      c.Orders
                    }).Skip(startRowIndex).Take(maxRows);
        return data.ToList();
    }
}
NorthwindDataContext db = new NorthwindDataContext();
DataLoadOptions opt = new DataLoadOptions();
opt.LoadWith<Customer>(c => c.Orders);
opt.LoadWith<Order>(o => o.Order_Details);
db.LoadOptions = opt;

The DataLoadOptions class can be used to modify the default behavior of the LINQ to SQL engine so that data referenced by a specific relationship is loaded immediately. The code in Figure 3 ensures that orders are loaded along with customers and that details are loaded along with orders.

The LoadWith method loads data according to the specified relation. The AssociateWith method can then allow for filtering related prefetched objects, as shown here:

opt.AssociateWith<Customer>(
     c => c.Orders.Where(o => o.OrderDate.Value.Year == 1997));

In this example, only orders issued in the year 1997 are prefetched when customer data is fetched. You use the AssociateWith method when you need to prefetch related data and also when you need to apply a filter. It is entirely up to you to ensure that no cyclical references exist between tables—for instance, when you load orders for a customer and then load the customer for an order, as shown here:

DataLoadOptions opt = new DataLoadOptions();
opt.LoadWith<Customer> (c => c.Orders);
opt.LoadWith<Order> (o => o.Customer);

Now that you have all the data ready, you can start thinking about binding. In this case, a two-level ListView control will do the trick quite well. You bind the top-level ListView to the collection of Customer objects and the innermost ListView to the Orders property of each bound Customer object. The code in Figure 4 shows the markup for a three-level hierarchical view where customers are displayed on the first level and are rendered by the ItemTemplate property of the outermost ListView. The embedded ListView is then bound to orders. And finally, the ItemTemplate of the embedded ListView contains a GridView to list the details of each order.

Figure 4 Three-Level Hierarchy (Close Figure 4)
<asp:ListView ID="ListView1" runat="server"
  DataSourceID="ObjectDataSource1"
  ItemPlaceholderID="lvItemPlaceHolder">

  <LayoutTemplate>
    <asp:PlaceHolder runat="server" ID="lvItemPlaceHolder" />
  </LayoutTemplate>

  <ItemTemplate>
    <asp:Panel runat="server" ID="panelCustomerInfo"
      cssclass="customerInfo">
      <%# Eval("CompanyName") %>
    </asp:Panel>
    <asp:panel runat="server" ID="panelCustomerDetails"
      cssclass="customerDetails">
      <asp:ListView runat="server"
        DataSource='<%# Eval("Orders") %>'
        ItemPlaceholderID="lvOrdersItemPlaceHolder">

        <LayoutTemplate>
          <ul>
            <asp:PlaceHolder runat="server"
              ID="lvOrdersItemPlaceHolder" />
          </ul>
        </LayoutTemplate>

        <ItemTemplate>
          <li>
            Order #<%# Eval("OrderID") %>
            <span class="orderDate">
              placed on <%#
              ((DateTime)Eval("OrderDate")).ToString
              ("ddd, dd MMM yyyy") %>
            </span>
            <span class="orderEmployee">
              managed by <b><%# Eval("Employee.LastName") %></b>
            </span>
            <asp:GridView runat="server"
              DataSource='<%# Eval("Order_Details") %>'
              SkinID="OrderDetailsGridSkin" >
            </asp:GridView>
          </li>
        </ItemTemplate>
      </asp:ListView>
    </asp:panel>
  </ItemTemplate>
</asp:ListView>

Improve the User Experience with Extenders

Frankly, the user interface you get from the code in Figure 4 is not very appealing. Because a hierarchical data view is being constructed, an expand/collapse panel would be a particularly apt solution for improving the user experience. The ASP.NET AJAX Control Toolkit provides a ready-made extender that, when applied to a Panel server control, can add a dropdown effect to the information associated with each customer and order.

Use the CollapsiblePanelExtender control to define a panel in the page control tree that will be expanded and collapsed via script. Needless to say, as a page developer you will not have to write any JavaScript. All of the script required for expanding and collapsing the panel is silently injected by the extender control. Let's have a look at the properties you might want to set on the extender:

<act:CollapsiblePanelExtender runat="server" ID="CollapsiblePanel1"
     TargetControlID="panelCustomerDetails"
     Collapsed="true"
     ScrollContents="true"
     SuppressPostback="true"
     ExpandedSize="250px"
     ImageControlID="Image1"
     ExpandedImage="~/images/collapse.jpg"
     CollapsedImage="~/images/expand.jpg"
     ExpandControlID="Image1"
     CollapseControlID="Image1">
</act:CollapsiblePanelExtender>

Some slight changes to the code in Figure 4 are needed to support the collapsible panel extender. In particular, you should edit the panel named panelCustomerInfo to add the button used to expand and collapse the child view. Here's one way to rewrite the panel's markup:

<asp:Panel ID="panelCustomerInfo" runat="server">
  <div class="customerInfo">
    <div style="float: left;"><%# Eval("CompanyName") %></div>
    <div style="float: right; vertical-align: middle;">
      <asp:ImageButton ID="Image1" runat="server"
               ImageUrl="~/images/expand.jpg"
               AlternateText="(Show Orders...)"/>
    </div>
  </div>
</asp:Panel>

The button is rendered using a right-justified image on the same row as the customer name. The TargetControlID property on the extender references the panel in the page that will be collapsed and expanded. This is the panel that physically contains orders and order details. As you see in Figure 4, it is the panel named panelCustomerDetails.

The ExpandControlID and CollapseControlID attributes indicate the ID of the elements that, when clicked, expand and collapse the target panel. If you plan to use different images to reflect the state of the panel, then you need to also specify the ID of an image control. This information belongs to the ImageControlID attribute. The ImageControlID is associated with two other properties, CollapsedImage and ExpandedImage—these hold the URL of the images.

The ExpandedSize property sets the maximum height in pixels allowed for the expanded panel. By default, any content that exceeds the height is just clipped out. However, if you set the ScrollContents property to true, a vertical scrollbar is added to let users scroll the entire contents.

Finally, the Collapsed Boolean property allows for you to set the initial state of the panel and SuppressPostback indicates whether the expansion of the panel should be a completely client-side operation. When SuppressPostback is true, no postback is used to expand or collapse the panel. This means that no updates are possible on the displayed data. For relatively static data that doesn't change frequently, this is definitely the best possible choice as it reduces page flickering and network traffic. However, if you need to display data more dynamically in the control, you can still minimize flickering by using an UpdatePanel control. Figure 5 shows the resulting user interface of a tri-level data view.

Figure 5 Data View with Three Levels
Figure 5 Data View with Three Levels(Click the image for a smaller view)
Figure 5 Data View with Three Levels
Figure 5 Data View with Three Levels (Click the image for a larger view)

DataPager and ListView

The ListView control offers paging facilities through the new DataPager control. The DataPager is a general-purpose paging control that can be used by any data-bound control that implements the IPageableItemContainer interface. As of ASP.NET 3.5, the ListView is the only control that supports this interface.

The DataPager control can display a built-in or template-based user interface. Whenever the user clicks to jump to a new page, the DataPager control invokes a method on the IPageableItemContainer interface. This method is expected to set internal variables in the paged control so that only a specific page of data is displayed during the next data-binding operation.

It turns out that selecting the right page of data remains the problem of the data-bound control—in this case, the ListView. Just as with other "view" controls in ASP.NET, the ListView control relies on external code for paging. If data is bound through the data source property, then user code should provide for the paged data. Otherwise, if data is bound through a data source control, you should configure the data source control properly to support paging.

Both the LinqDataSource and ObjectDataSource controls offer built-in paging capabilities. The LinqDataSource has the AutoPage property for enabling or disabling default paging. For hierarchical data, you also need to ensure that the LINQ data context has proper load options set. The programming interface of the LinqDataSource doesn't have properties for setting the LoadOptions property on the data context object. However, by handling the ContextCreated event, you can access the newly created data context and configure it as you like:

void LinqDataSource1_ContextCreated(
    object sender, LinqDataSourceStatusEventArgs e)
{
    // Get a reference to the data context
    DataContext db = e.Result as DataContext;

    if (db != null)
    {
       DataLoadOptions opt = new DataLoadOptions();
       opt.LoadWith<Customer>(c => c.Orders);
       opt.LoadWith<Order>(o => o.Employee);
       opt.LoadWith<Order>(o => o.Order_Details);
       db.LoadOptions = opt;
    }
}

As an alternative to this action, you can use the ObjectDataSource control for providing data and implementing any paging logic. Then, in the business object you can either use LINQ to SQL or plain ADO.NET for data access.

A snag I ran into while using DataPager and ListView together is worth mentioning, though. I initially had a content page with ListView and DataPager hosted in the same content placeholder. I referenced the ListView control in the DataPager using the PagedControlID property, as shown in the following. It worked just fine:

<asp:DataPager ID="DataPager1" runat="server"
     PagedControlID="ListView1"
     PageSize="5"
     EnableViewState="false">
  <Fields>
     <asp:NextPreviousPagerField
        ShowFirstPageButton="true"
        ShowLastPageButton="true" />
  </Fields>
</asp:DataPager>

Next, I moved the DataPager to another content area in the same master. And all of a sudden, the DataPager was unable to communicate with the ListView control. The problem lies with the algorithm used by the DataPager control to locate the paged control. This algorithm doesn't work if the two controls are hosted by different naming containers. To work around the issue, you need to identify the paged control using its full, unique ID (that includes naming container information). Unfortunately, you can't easily set this information declaratively.

You can't use ASP-style code blocks because they are treated like literals if used to set a property of a server control. You can't use a data binding expression <%# ... %> either because the expression is evaluated too late for the needs of the DataPager. The Load event is too late and would cause the DataPager to fire an exception. The simplest workaround is to set the PagedControlID property programmatically in the page's Init event, like so:

protected void Page_Init(object sender, EventArgs e)
{
   DataPager1.PagedControlID = ListView1.UniqueID;
}

Multiple Item Templates

Just like other template-based and data-bound controls, the ListView repeats the same item template for each bound data item. So what if you want to change it for a certain subset of items? Honestly, I have to admit that I never had the need to use more than one item template in years of ASP.NET programming. Several times I customized the appearance of a small group of items in DataGrid and GridView controls based on runtime conditions; however, that always entailed applying a different set of style attributes.

Only in very few cases did I programmatically add new controls (mostly Label controls or table cells) to the existing template. In data-bound controls that fire data binding events, this is not a really difficult task—at least not if you have intimate knowledge of the internal structure of the controls you are manipulating.

Although programmatic injection of controls is a solution that worked really well in practice, it never captured my heart. So I decided to try a different route when a customer asked me to modify a ListView-based menu in a Web page. In a menu similar to the one in Figure 1, I needed to render the items of one submenu horizontally rather than vertically.

The ListView control generates its markup by looping through the data source and applying the following algorithm. First, it checks whether an item separator is required. If so, it instantiates the template and creates the data item object. The data item object is the container of the item template and carries information about the index of the item in the view and the bound data source. When the item template is instantiated, the ItemCreated event fires. The next step is data binding. After this has completed, the ItemDataBound event is fired.

As you can see, there's no public event that you can handle that allows programmatically changing the template for each item. You can change the template in the Init or Load page event, but that would be for all bound items. If you handle ItemCreated and set the ItemTemplate property there, the change will affect the next item, but not the item being currently processed. You would need an ItemCreating event, but no such an event is fired by the ListView control. The solution, then, is to create your own ListView control, as in Figure 6.

Figure 6 Firing an ItemCreating Event (Close Figure 6)
namespace Samples.Controls
{
  public class ListViewItemCreatingEventArgs : EventArgs
  {
    private int _dataItemIndex;
    private int _displayIndex;

    public ListViewItemCreatingEventArgs(int dataItemIndex,
                                         int displayIndex) {
      _dataItemIndex = dataItemIndex;
      _displayIndex = displayIndex;
    }

    public int DisplayIndex {
      get { return _displayIndex; }
      set { _displayIndex = value; }
    }

    public int DataItemIndex {
      get { return _dataItemIndex; }
      set { _dataItemIndex = value; }
    }
  }

  public class ListView : System.Web.UI.WebControls.ListView
  {
    public event EventHandler<ListViewItemCreatingEventArgs>
                                               ItemCreating;

    protected override ListViewDataItem CreateDataItem(int
                           dataItemIndex, int displayIndex) {
      // Fire a NEW event: ItemCreating
      if (ItemCreating != null)
        ItemCreating(this, new ListViewItemCreatingEventArgs
                             (dataItemIndex, displayIndex));

      // Call the base method
      return base.CreateDataItem(_dataItemIndex, displayIndex);
    }
  }
}

By overriding the CreateDataItem method, you have a chance to run your code just before the item template is instantiated. The CreateDataItem method is declared protected and virtual in the ListView class. As you can see in Figure 6, the method override is quite simple. You first fire a custom ItemCreating event and then proceed by calling the base method.

The ItemCreating event passes a couple of integers back to the user code—the absolute index of the item in the data source and the page-specific index. For example, for a page size of 10, when the ListView is working on rendering the first item of the second page, dataItemIndex contains 11 items and displayIndex contains 1 item. In order to use the new ItemCreating event, simply declare the method and handler on your custom ListView control, as you can see in the following code:

<x:ListView runat="server" ID="ListView1"
   ItemPlaceholderID="itemPlaceholder"
   DataSourceID="ObjectDataSource1"
   OnItemCreating="ListView1_ItemCreating">
   <LayoutTemplate>
      <div>
         <asp:PlaceHolder runat="server" ID="itemPlaceholder" />
      </div>
   </LayoutTemplate>
</x:ListView>

In your code, you can handle the event like so:

void ListView1_ItemCreating(
     object sender, ListViewItemCreatingEventArgs e)
{
    string url = "standard.ascx";
    if (e.DisplayIndex % DataPager1.PageSize == 0)
        url = "firstItem.ascx";

    ListView1.ItemTemplate = Page.LoadTemplate(url);
}

Here, two different user controls are employed to render the data items. The specific user control is determined by the display index. All items share the same template except the first. Figure 7 shows the page in action.

Figure 7 Multiple Item Templates
Figure 7 Multiple Item Templates(Click the image for a smaller view)
Figure 7 Multiple Item Templates
Figure 7 Multiple Item Templates (Click the image for a larger view)

When you think about the complexity of common real-world pages, this solution appears too simple. More often than not, you need to use different templates based on the content to display. You need to further enhance the custom ListView control to change the item template within the data-binding process. Have a look at the code in Figure 8.

Figure 8 Choosing Template Based on Content (Close Figure 8)
namespace Samples.Controls
{
  public class ListViewItemCreatingEventArgs : EventArgs
  {
    private int _dataItemIndex;
    private int _displayIndex;

    public ListViewItemCreatingEventArgs(int dataItemIndex,
                                         int displayIndex) {
      _dataItemIndex = dataItemIndex;
      _displayIndex = displayIndex;
    }

    public int DisplayIndex {
      get { return _displayIndex; }
      set { _displayIndex = value; }
    }

    public int DataItemIndex {
      get { return _dataItemIndex; }
      set { _dataItemIndex = value; }
    }
  }

  public class ListView : System.Web.UI.WebControls.ListView
  {
    public event EventHandler<ListViewItemCreatingEventArgs>
     ItemCreating;

    private int _displayIndex;
    private bool _shouldInstantiate = false;

    protected override void InstantiateItemTemplate(Control container,
     int displayIndex) {
      if (_shouldInstantiate) {
        base.InstantiateItemTemplate(container, displayIndex);
        _shouldInstantiate = false;
      }
    }

    protected override ListViewDataItem CreateDataItem(int
     dataItemIndex, int displayIndex) {
      // Fire a NEW event: ItemCreating
      if (ItemCreating != null)
        ItemCreating(this, new
          ListViewItemCreatingEventArgs(dataItemIndex,
          displayIndex));

      // Cache for later
      _displayIndex = displayIndex;

      // Call the base method
      return base.CreateDataItem(_dataItemIndex, displayIndex);
    }

    protected override void OnItemCreated(ListViewItemEventArgs e) {
      base.OnItemCreated(e);

      // You can proceed with template instantiation now
      _shouldInstantiate = true;
      InstantiateItemTemplate(e.Item, _displayIndex);
    }
  }
}

The CreateDataItem method fires the ItemCreating event and caches the display index for later use. In addition, the InstantiateItemTemplate method is overridden to delay the actual template instantiation. A private boolean flag is used for that purpose. As mentioned, the ListView starts the data-binding process after instantiating the item template.

In the implementation shown in the code in Figure 8, however, no item template is really instantiated until the ItemCreated event is fired. When the ItemCreated event is raised, the data item object is bound to the ListView item container through the DataItem property. By handling the ItemCreated event in your code, you can decide which item template to use based on the bound data item, as you can see here:

protected override void OnItemCreated(ListViewItemEventArgs e)
{
   base.OnItemCreated(e);

   _shouldInstantiate = true;
   InstantiateItemTemplate(e.Item, _displayIndex);
}

In this case, the base method fires the ItemCreated event to the page. After that, the custom ListView control resets the Boolean flag and invokes the method to instantiate the item template. In the end, the item template is instantiated a bit later than in the built-in ListView control, but you can set the ItemTemplate property for each item programmatically in the ItemCreated event handler after looking at the content of the bound data item (see Figure 9). Figure 10 shows a sample page where a blue template is used for men and a pink template is used for women.

Figure 9 Setting the Item Template (Close Figure 9)
void ListView1_ItemCreated(object sender, ListViewItemEventArgs e)
{
    // Grab a reference to the data item
    ListViewDataItem currentItem = (e.Item as ListViewDataItem);
    Employee emp = (Employee) currentItem.DataItem;
    if (emp == null)
        return;

    // Apply your logic here
    string titleOfCourtesy = emp.TitleOfCourtesy.ToLower();
    string url = "forgentlemen.ascx";
    if (titleOfCourtesy == "ms." || titleOfCourtesy == "mrs.")
        url = "forladies.ascx";

    // Set the item template to use
    Samples.ListView ctl = (sender as Samples.Controls.ListView);
    ctl.ItemTemplate = Page.LoadTemplate(url);
}
Figure 10 A Standard Menu
Figure 10 A Standard Menu(Click the image for a smaller view)
Figure 10 A Standard Menu
Figure 10 A Standard Menu (Click the image for a larger view)

Summing It Up

At the end of the day, the new ASP.NET 3.5 ListView control is a revamped version of the DataList control that has existed since ASP.NET 1.0. The ListView allows for tighter control over the generated markup and fully supports data source objects.

In this column, you saw how to use nested ListView controls to build pageable and multi-level data views and how to modify the standard ListView's rendering process by deriving a custom control and overriding a few methods. The final result is a control that supports multiple item templates. No other ASP.NET data-bound control provides such a level of flexibility.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an IDesign architect and the author of Programming ASP.NET 3.5 Core Reference. Based in Italy, Dino is a frequent speaker at industry events worldwide. You can reach him at cutting@microsoft.com or join his blog at weblogs.asp.net/despos.


Foundations: Code Access Security in WCF, Part 1

Foundations
Code Access Security in WCF, Part 1
Juval Lowy

Code download available at: Foundations2008_04.exe (479KB)
Contents
CAS at a Glance
Client-Side CAS
Partially Trusted Clients
Client-Side Demands
Raw WCF Demands
Structured Demands with PartialTrustClientBase<T>
Analyzing the Demands of Invoke
Implementing Client-Side Structured Demands

I
ntroduced in the Microsoft® .NET Framework 1.0, code access security (CAS) is probably the single differentiating capability of .NET compared with unmanaged code. CAS is built into the very fabric of the .NET Framework, affecting every operation in managed code, something that unmanaged code can never achieve.

The first release of Windows® Communication Foundation (WCF) offered no support for CAS; the System.ServiceModel assembly did not allow any partially trusted callers, which disabled CAS support. The second release introduced rudimentary support for CAS in some of the HTTP bindings and only for a limited set of scenarios. This change enabled me to write a small framework that provides comprehensive support for CAS without compromising either the WCF programming model or CAS.

In this first installment of two columns on CAS, I will briefly discuss code access security in WCF, and then proceed to show my solution for enabling partially trusted clients of WCF services.

CAS at a Glance

The .NET Framework defines 24 different security permissions, governing almost any type of operation. There are permissions for file I/O, UI, reflection, security, network, data access, and so on. A permission type can be applied to a specific resource, such as permission to read from a specific file in the case of the file I/O permission or permission to display specific types of windows as with the UI permission. It may also be completely denied (such as no file I/O operations at all) or completely granted (such as unrestricted file I/O access).

Permissions are grouped into permission sets and every assembly is assigned a set. The .NET Framework defines some standard permission sets such as FullTrust (implies all permissions) or Execution (permission to only access the CPU). Administrators can use the .NET Configuration tool to define custom permission sets, and developers can define custom permission sets programmatically—or use a permission set file, or define a ClickOnce application manifest with the permission set required by their application.

The CLR assigns each assembly its permissions when the assembly is loaded. Assemblies are granted these permissions based on some form of evidence substantiating their identity. There are origin-based evidences that examine where the assembly is loaded from (for example, all code coming from the Global Assembly Cache, or GAC, is granted full trust), or content-based evidence that examines some aspect of the assembly itself, such as its strong name.

Each app domain is always assigned a permission set called the app domain security policy, and any assembly loaded into that app domain is restricted to that permission set; otherwise it encounters a security exception. New app domains are launched with the FullTrust permission set, and since all code originating from the local machine also gets FullTrust by default, most .NET-based applications just work out of the box, in effect not utilizing CAS at all. This renders the code (and the user, the data, the machine, and even the network) susceptible to a variety of maladies, from security attacks such as viruses or worms, to plain user mistakes.

Code that is executing with less than full trust is called partially trusted code. Whenever any piece of managed code tries to access a resource or perform any operation using the .NET Framework (including interoperability with unmanaged code), the .NET Framework verifies that the assembly containing that code has the required permissions to perform the operation. If that assembly lacks a demanded permission, the .NET Framework throws a security exception thus aborting the operation.

Since a trusted assembly can be lured by a malicious, less-trusted assembly into performing operations that the less-trusted assemblies does not have permissions to execute, it is insufficient to demand the permissions only of the assembly performing an operation. Therefore, the .NET Framework walks the entire stack of callers, verifying that every caller up the stack has the required permissions. This stack walk is called a security demand, and it is performed regardless of the executing assembly's permissions.

Your code can also assert a permission—that is, declare that every caller up the stack has the demanded permissions. A permission assertion has the effect of stopping a stack walk. Your code can only assert permissions it already has, and it additionally requires the special security assertion permission.

It is always a good idea when asserting one permission to demand another in its place. Developers can demand or assert permissions programmatically using dedicated permission classes or use a matching set of attributes. Developers can also actively refuse permissions at the assembly, class, or method level. Refusing permissions or permitting only the limited set of permissions required by the code to execute reduces the surface area for a luring attack. For more on code access security, see Chapter 12 in my book, Programming .NET Components 2nd Edition, where I devote more than 100 pages to this fundamental technology and its applications.

Client-Side CAS

In the .NET Framework 3.5, WCF only allows a limited set of scenarios to execute in partial trust. WCF allows only the HTTP bindings BasicHttpBinding, WSHttpBinding, and WebHttpBinding (with the exclusion of the WSDualHttpBinding) to be called under partial trust and only with no security at all or with transport security. Furthermore, in the case of the WSHttpBinding, aspects such as message security, reliable messaging, and transactions are disallowed. All partial trust-enabled bindings must use text encoding. The client cannot use additional WCF facilities such as diagnostics. To enable usage in a partially trusted environment, the System.ServiceModel assembly allows for partial trust callers by including the AllowPartiallyTrustedCallers attribute as part of the assembly definition:

[assembly: AllowPartiallyTrustedCallers]

In the first release of WCF, omitting this attribute precluded all partial-trust use. In the .NET Framework 3.5, enforcing the limited set of supported features is now the responsibility of the bindings. Each non-HTTP binding actively demands full trust of its callers, be it the client proxy or the service host. The allowed HTTP bindings themselves do not demand full trust, but instead demand permissions according to the context of use. On the client side, these bindings demand permission to execute (security permission with execution flag), and permission to connect to the service (Web permission with connect flag to the target URI).

Beyond these demands on the client there are additional limitations on configuration. For example, the config file cannot contain any reference to any certificate store (for client-side certificate credentials) since touching the certificate store will cause WCF to demand full trust.

Ideally, you would like to tap into the full power of WCF from distributed transactions to reliable calls to various security credential types to intranet (or even same-machine) application communication over TCP and inter-process communication (IPC) channels such as named pipes, and do all that without trading off CAS—that is, without resorting to full trust.

Partially Trusted Clients

To enable clients at any partial-trust level to use any WCF feature and binding, you need to block the demand of the bindings for full trust. The only way to do that is to have the proxy itself assert full trust. Asserting full trust can be done via the PermissionSetAttribute using the Assert flag of the SecurityAction enum and specifying the type-safe string "FullTrust" for the permission name:

[PermissionSet(SecurityAction.Assert,Name = "FullTrust")]

Though the permission name is specified using a string, there is a degree of safety added by the compiler in that it verifies the allowed values.

In addition, you must prevent the client from directly accessing any method of the base class of ClientBase<T> (which still demands full trust), so the proxy needs to hide the commonly used methods of Close and Dispose. Having the proxy class itself access methods or properties of ClientBase<T> (such as Channel or constructors) is fine since the proxy asserts full trust.

The problem is that in order to assert full trust, the proxy itself must be granted full trust, something the partially trusted client is not able to provide in the first place. Consequently, you need to factor out the proxy class to its own assembly, mark it as public, and grant that assembly full trust. You can accomplish this by using the .NET Framework 2.0 Configuration control panel applet—simply identify the proxy's assembly using some content-based evidence such as its strong name and grant the assembly full trust.

You can also install the proxy assembly in the client's GAC. Since all assemblies coming from the GAC are granted full trust, the proxy will also gain full trust. You also need to remember to declare the AllowPartiallyTrustedCallers attribute; this will allow partially trusted callers to call the assembly.

Finally, you need to add the definition of the contract used by the proxy to the proxy's assembly (and mark the contract as public as well). This is required because WCF demands full trust of all assemblies up the call chain, and if the contract comes from a partially trusted assembly, the demand will fail. Figure 1 shows examples of these contract and proxy definitions.

Figure 1 Asserting Full Trust by the Proxy (Close Figure 1)
[assembly: AllowPartiallyTrustedCallers]

[ServiceContract]
public interface IMyContract
{
  [OperationContract]
  void MyMethod();
}

[PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
public class MyContractClient :
  ClientBase<IMyContract>,IMyContract,IDisposable
{
  public MyContractClient() {}

  public MyContractClient(string endpointName) : base(endpointName) {}

  /* More constructors */

  public void MyMethod() {
    Channel.MyMethod();
  }
  public new void Close() {
    base.Close();
  }
  void IDisposable.Dispose() {
    Close();
  }
}

Client-Side Demands

Unfortunately, the technique showed in Figure 1 presents a potential security breach. By suppressing the security demands of WCF, any partially trusted client can now call any WCF service. Imagine a client that was not granted permissions connecting to a TCP socket or a Web site. While that client cannot use the network directly, it can bypass that limitation by calling out from the restricted client environment over WCF.

The solution for that problem is to use a dedicated subclass of ClientBase<T> that on the one hand will assert the blank WCF demand for full trust, and on the other hand will demand the appropriate specific security permissions according to what the client is trying to do. Such a proxy class is my PartialTrustClientBase<T> class, shown in Figure 2.

Figure 2 PartialTrustClientBase<T> Class (Close Figure 2)
public abstract class PartialTrustClientBase<T> :
  ClientBase<T>,IDisposable
  where T : class
{
  [PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
  public PartialTrustClientBase() {}

  [PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
  public PartialTrustClientBase(string endpointName) :
    base(endpointName) {}

  [PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
  public PartialTrustClientBase(Binding binding,
    EndpointAddress remoteAddress) :
    base(binding,remoteAddress) {}

  //Useful only for clients that want full-brunt raw demands from WCF
  protected new T Channel
  {
    [PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
    get {
      return base.Channel;
    }
  }

  [PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
  new public void Close() {
    base.Close();
  }
  void IDisposable.Dispose() {
    Close();
  }

  protected object Invoke(string operation,params object[] args) {
    if(IsAsyncCall(operation)) {
      DemandAsyncPermissions();
    }
    DemandSyncPermissions(operation);
    CodeAccessSecurityHelper.PermissionSetFromStandardSet(
      StandardPermissionSet.FullTrust).Assert();

    Type contract = typeof(T);
    MethodInfo methodInfo = contract.GetMethod(operation);
    return methodInfo.Invoke(Channel,args);
  }

  protected virtual void DemandAsyncPermissions() {
    CodeAccessSecurityHelper.DemandAsyncPermissions();
  }

  protected virtual void DemandSyncPermissions(string operationName) {
    this.DemandClientPermissions(operation);
  }

  bool IsAsyncCall(string operation) {
    if(operation.StartsWith("Begin")) {
      MethodInfo info = typeof(T).GetMethod(operation);
      object[] attributes = info.GetCustomAttributes(
        typeof(OperationContractAttribute),false);
      Debug.Assert(attributes.Length == 1);
      return (attributes[0] as
        OperationContractAttribute).AsyncPattern;
    }
    return false;
  }
}

PartialTrustClientBase<T> is used just like the regular proxy base class. You still need to grant full trust to the proxy class derived from it and allow partial-trust callers. However, unlike Figure 1, PartialTrustClientBase<T> does not assert full trust at the class level. Instead, it asserts full trust locally, just when required. In addition, PartialTrustClientBase<T> may also be used to demand other CAS permissions.

If you derive your proxy class from PartialTrustClientBase<T> and have the proxy assert full trust as shown here, then no demands will be made on the calling client:

[PermissionSet(SecurityAction.Assert,Name = "FullTrust")]
public class MyContractClient :
  PartialTrustClientBase<IMyContract>,IMyContract
{
  public MyContractClient() {}

  public MyContractClient(string endpointName) :
    base(endpointName) {}

  public void MyMethod() {
    Channel.MyMethod();
  }
}

The only difference between this code and Figure 1 is the new version is cleaner since the hiding of Close and Dispose is now done by PartialTrustClientBase<T>. This new proxy still suppresses all security demands by WCF and leaves the partially trusted client exposed to a luring attack or enables that partially trusted client to do more than intended.

Raw WCF Demands

The more security-conscious use of PartialTrustClientBase<T> is to not have the proxy assert full trust:

public class MyContractClient :
  PartialTrustClientBase<IMyContract>,IMyContract
{
  public MyContractClient() {}
  public MyContractClient(string endpointName) :
    base(endpointName) {}
  public void MyMethod() {
    Channel.MyMethod();
  }
}

To support such use, PartialTrustClientBase<T> surgically asserts full trust on its constructors and its Close method. In addition, PartialTrustClientBase<T> hides the Channel property of ClientBase<T> and asserts full trust on the get accessor. This is sufficient to suppress the WCF binding demand for full trust, since that demand is made when constructing the proxy, and when opening and closing its channel—but not when actually using it. The interesting effect of structuring the proxy this way is that now the client code will be subjected to the raw WCF security demands—that is, all the security demands required to marshal the call to the service!

For example, if the proxy is using the TCP binding, the proxy will first demand the client permission to execute (all managed code requires that permission). Second, the proxy will demand that the client have permission to connect to the required port on the service machine, and unrestricted DNS permissions (required to resolve the host's address). Third, there are some collateral permission demands unrelated to the use of TCP, pertaining to the context of the call.

If the client wishes to use Windows security and send the user's interactive identity, then the proxy will demand environment permission access to the USERNAME variable. If the client wishes to send alternative Windows credentials, then the proxy will demand security permission to control the principal. If the client wants to dispatch the call asynchronously or receive duplex callbacks, then the proxy will demand permissions to control the policy and the evidence (both are flags of the security permission, required when bouncing calls between threads). If the client wishes to use reliable messaging, then the proxy will demand policy control as well. If the client uses message security with user-name credentials, with service certificate negotiation and without validating the negotiated service certificate, then the proxy will also demand permission to control the policy and the evidence. If the client utilizes the WCF diagnostics and tracing facility, then the proxy will demand access to the COMPUTERNAME environment variable (to be able to trace it) and unmanaged code access (presumably to access the log files, so it should have been a file I/O permission instead).

Finally, the proxy will also demand of the client permission to execute unmanaged code. Unmanaged code access is a highly privileged security permission, granted only to the most trustworthy code. Granting unmanaged code access permission can amount to disabling CAS since unmanaged code is exempted from CAS. Classes and frameworks designed to work in a partial trust environment should never demand unmanaged code access (even if they use interop) and instead demand a more narrow permission describing the nature of the unmanaged operation performed. The TCP channel demands unmanaged code simply because its original design did not take into account partially trusted clients.

Then there are certain WCF capabilities that bluntly resort to full trust demands, even though there are perfectly matching permissions types. Any attempt to propagate a transaction requires full trust (instead using the distributed transaction permission) and any access of a certificate store requires full trust (instead of using the certificate store permission). Figure 3 shows the raw WCF security demands invoked by code such as my last revision of PartialTrustClientBase<T>, when the bindings are set to their default settings, for a few key scenarios such as transactions, reliability, diagnostics, asynchronous calls, certificate store access, and message security.

Figure 3 Raw WCF Client-Side Security Demands (Close Figure 3)

Scenario | Permissions
TCP | Security permission with execution and unmanaged code, unrestricted DNS, socket permission to connect to port on target host
IPC | Security permission with execution, unmanaged code, control policy, and control evidence
WS and WS Dual | Security permission with execution and unmanaged code, Web permission to connect to URI
Basic and Web | Security permission with execution, Web permission to connect to URI
MSMQ | Security permission with execution and unmanaged code
Asynchronous calls, Duplex over TCP | Security permission to control policy and evidence
RM over TCP | Security permission to control policy
Windows security with interactive user credentials | Environment permission to read USERNAME
Windows security with alternative credentials | Security permission to control principal, Environment permission to read USERNAME
Diagnostic tracing | Security permission with unmanaged code, Environment permission to read COMPUTERNAME
User-name credentials, Message security with service cert negotiation and without service cert validation | Security permission to control policy and evidence, Store permission to enumerate certificates
User-name credentials, TCP message security with service cert negotiation and without service certificate validation | Security permission to control policy and evidence
Any certificate store access, Transaction propagation | Full trust
User-name credentials, Message security without service certificate negotiation or with service certificate validation, Certificate credentials | Full trust

The only bindings that do not demand unmanaged code access are the basic and Web bindings. The WS binding defaults to message security by default, resulting in an unmanaged code access permission demand.

Structured Demands with PartialTrustClientBase<T>

As you can see from Figure 3, key aspects of WCF such as transactions, reliable messaging, unlimited message security interaction, and certificate store access all require full trust, thus rendering my prior version of PartialTrustClientBase<T> pointless in a partial-trust environment. Furthermore, demanding unmanaged code access by virtually all non-HTTP bindings (and the WS binding with message security) is unacceptable, negating the very idea of CAS for a partial-trust client.

To enable appropriate, secure, and correct use of WCF by a partially trusted client, PartialTrustClientBase<T> offers the Invoke method defined as:

protected object Invoke(string operation,params object[] args);

Invoke accepts the name of the operation to invoke and its parameters in the form of a comma-separated params array of objects. Rather than using the Channel property for invoking the call, Invoke uses reflection. Here is a proxy deriving from PartialTrustClientBase<T> using the Invoke method:

public class MyContractClient :
  PartialTrustClientBase<IMyContract>,IMyContract
{
  public MyContractClient() {}
  public MyContractClient(string endpointName) :
    base(endpointName) {}
  public void MyMethod() {
    Invoke("MyMethod");
  }
}

Invoke will first demand the appropriate CAS permissions according to the scenario of the calling client and the target service endpoint. If the client is granted those permissions—that is, no security exception was raised by the demands—Invoke will programmatically assert full trust and proceed to invoke the operation, satisfied that the client has the right permissions to call the service. I call such behavior a structured permission demand.

Invoke cannot use an attribute to declaratively assert full trust, since that will mask any more granular permission it demands. Instead, the implementation of Invoke in Figure 2 first checks whether the call is invoked asynchronously (using the AsyncPattern flag of the operation contract). If it is, Invoke demands the appropriate permissions using the DemandAsyncPermissions helper method. Invoke then demands the synchronous permissions using the DemandSyncPermissions helper method. For the call itself, Invoke programmatically asserts full trust using my CodeAccessSecurityHelper class shown in Figure 4.

Figure 4 CodeAccessSecurityHelper Class (Close Figure 4)
public enum StandardPermissionSet {
  Internet,
  LocalIntranet,
  FullTrust,
  Execution,
  SkipVerification
}

public static class CodeAccessSecurityHelper {
  public static PermissionSet PermissionSetFromStandardSet(
    StandardPermissionSet standardSet);

  public static void DemandClientPermissions<T>(this ClientBase<T> proxy)
    where T : class;

  public static void DemandAsyncPermissions();

  //More members

The PermissionSetFromStandardSet method takes my enum value representing one of the standard .NET permission sets and returns the matching instance of PermissionSet (see Figure 5). PermissionSet, as its name implies, is a collection of permissions, but it can also represent the super permission set full trust (which is not really a set of individual permissions; it is more like a single permission). The PermissionSet class supports the IStackWalk interface, which lets you install a stack walk modifier, such as a modifier that stops the demand for the permissions in the permission set by asserting that all callers up the stack have those permissions. The stack walk modifier is removed automatically when the method that installed it returns. You can also explicitly remove it with the static RevertAssert method of PermissionSet.

Figure 5 PermissionState and PermissionSet (Close Figure 5)
public enum StandardPermissionSet {
  Internet,
  LocalIntranet,
  FullTrust,
  Execution,
  SkipVerification
}

public static class CodeAccessSecurityHelper {
  public static PermissionSet PermissionSetFromStandardSet(
    StandardPermissionSet standardSet);

  public static void DemandClientPermissions<T>(this ClientBase<T> proxy)
    where T : class;

  public static void DemandAsyncPermissions();

  //More members

Both the DemandAsyncPermissions and DemandSyncPermissions helper methods of PartialTrustClientBase<T> use corresponding methods on CodeAccessSecurityHelper to perform their demands.

The chart in Figure 6 shows the structured demands raised by PartialTrustClientBase<T>.Invoke as a function of the binding used and other aspects of the scenario such as the use of transactions, reliability, certificate store access, diagnostics, callbacks, and asynchronous calls.

Figure 6 Structured Security Demands of PartialTrustClientBase<T> (Close Figure 6)

Scenario | Permissions
TCP | Security permission with execution, unrestricted DNS, Socket permission to connect to port on target host
IPC | Security permission with execution, control policy, and control evidence
WS, Basic | Security permission with execution and Web permission to connect to URI
WS-Dual | Security permission with execution and Web permission to connect to URI, and Web permission to accept callbacks to callback address, minimal ASP.NET hosting permission
MSMQ | Security permission with execution, MSMQ permission to send to queue
RM over TCP | Security permission to control policy
Duplex over TCP, Asynchronous calls (AsyncPattern), User-name credentials, Message security with service certificate negotiation and without service certificate validation | Security permission to control policy and control evidence
Windows security with interactive user credentials | Environment permission to read USERNAME
Windows security with alternative credentials | Security permission to control principal
Transaction propagation | Unrestricted distributed transaction permission
User-name credentials, Message security without service cert negotiation or with service cert validation, Certificate credentials | Store permission to enumerate stores, open store, and enumerate certificates
Diagnostic tracing | Environment permission to read COMPUTERNAME, File I/O permission to path discovery, append, write to log files

Analyzing the Demands of Invoke

I based the structured demands on a few elements. First, whenever possible, I tried to approximate the raw demands on the client raised by WCF. That said, I did smooth the rough edges of WCF since it was not designed for comprehensive partial-trust use. None of the bindings and scenarios listed in Figure 6 demand full trust or unmanaged code access. There are plenty of places in the .NET Framework that were designed for use in a partial-trust environment in a similar context, and I relied on the same demands as those. Finally, in the other cases, to compensate for suppressing the full trust demands, I used experience, familiarity with CAS, and common sense to map WCF activities to demands for dedicated permission types.

When the WS-Dual binding is used, Invoke demands Web permission to connect to the target endpoint, as with any other HTTP binding. However, to allow for hosting the callback object it also demands minimal ASP.NET hosting permission and Web permission to accept the calls to the callback address. When the MSMQ binding is used, Invoke demands MSMQ permission to send messages to the target queue.

With any attempt to propagate the client's transaction to a service, Invoke demands unrestricted distributed transaction permission. In this case, a transaction-aware binding is used, the transaction flow is enabled in the binding, the transaction flow at the operation level is allowed, and the client has an ambient transaction.

Any attempt by the proxy to access the certificate store triggers demands for permission to enumerate the certificate stores, to open a store, and to enumerate the certificates in a store. This will happen when the client uses certificate credentials, or when message security is used and the client needs to validate the negotiated service certificate, or when the client does not negotiate a certificate and instead just loads a certificate to use for securing the message.

When the client uses WCF diagnostics, Invoke demands environment permission to read the computer name and file I/O permission for the log and trace files used.

Implementing Client-Side Structured Demands

As mentioned already, the demands are carried out by CodeAccessSecurityHelper, whose partial implementation is shown in Figure 7. All permission types in the .NET Framework support the IPermission interface:

Figure 7 Implementing CodeAccessSecurityHelper (Partial) (Close Figure 7)
public static class CodeAccessSecurityHelper
{
  public static void DemandClientPermissions<T>(this ClientBase<T> proxy,
    string operationName) where T : class
  {
    DemandClientConnectionPermissions(proxy.Endpoint);
    DemandTransactionPermissions(proxy.Endpoint,operationName);
    DemandTracingPermissions();
    DemandClientSecurityPermissions(proxy);
    DemandEnvironmentPermissions(proxy);
    DemandClientStorePermissions(proxy.Endpoint);
  }

  internal static void DemandClientConnectionPermissions(
    ServiceEndpoint endpoint)
  {
    PermissionSet connectionSet = new PermissionSet(PermissionState.None);

    if(endpoint.Binding is NetTcpBinding)
    {
      connectionSet.AddPermission(new SocketPermission(
        NetworkAccess.Connect,TransportType.Tcp,
        endpoint.Address.Uri.Host,endpoint.Address.Uri.Port));

      connectionSet.AddPermission(new DnsPermission(
        PermissionState.Unrestricted));
    }

    /* Rest of the bindings */

    connectionSet.Demand();
  }

  internal static void DemandTransactionPermissions(
    ServiceEndpoint endpoint)
  {
    DemandTransactionPermissions(endpoint,null);
  }

  internal static void DemandTransactionPermissions(
    ServiceEndpoint endpoint, string operationName)
  {
    bool transactionFlow = false;
    bool flowOptionAllowed = false;

    if(endpoint.Binding is NetTcpBinding)
    {
      NetTcpBinding tcpBinding = endpoint.Binding as NetTcpBinding;
      transactionFlow = tcpBinding.TransactionFlow;
    }

    /* Checking other bindings */

    if(transactionFlow)
    {
      if(Transaction.Current != null)
      {
        //If operationName is null then at least one operation
        //needs to allow flow to trigger demand
        foreach(OperationDescription operation in
          endpoint.Contract.Operations)
        {
          string name = operationName ?? operation.Name;
          if(name != operation.Name)
          {
            continue;
          }
          foreach(IOperationBehavior behavior in operation.Behaviors)
          {
            if(behavior is TransactionFlowAttribute)
            {
              TransactionFlowAttribute attribute =
                behavior as TransactionFlowAttribute;
              if(attribute.Transactions !=
                TransactionFlowOption.NotAllowed)
              {
                flowOptionAllowed = true;
                break;
              }
            }
          }
          if(flowOptionAllowed)
          {
            break;
          }
        }
        if(flowOptionAllowed)
        {
          IPermission distributedTransactionPermission =
            new DistributedTransactionPermission(
            PermissionState.Unrestricted);
          distributedTransactionPermission.Demand();
        }
      }
    }
  }
  //Rest of the implementation
}

public interface IPermission : ...
{
   void Demand();
   //More members
}

Demanding any permission is done by instantiating the permission object and calling its implementation of the Demand method. When constructing a permission set, you can add individual permissions to it and then call Demand on the permission set to demand all permissions in the set.

The extension method DemandClientPermissions of CodeAccessSecurityHelper does the bulk of the demands on behalf of PartialTrustClientBase<T> (using an extension enables using it with any proxy class). It has a series of helper methods, all demanding permissions for their respective aspect. Figure 7 shows the code for DemandClientConnectionPermissions, which is used to demand connectivity permissions according to the binding. It examines the binding type used by the proxy and adds the appropriate permissions to a permission set object. It then calls Demand on the permission set.

The DemandTransactionPermissions method first verifies that the proxy is using a binding capable of transaction flow. If so, it verifies that the calling client actually has an ambient transaction to propagate (Transaction.Current is not null). If so, it scans the collection of operations for the endpoint's contract looking for the operation currently invoked. When the operation is found, DemandTransactionPermissions retrieves that operation's collection of operation behaviors. DemandTransactionPermissions examines each behavior, looking for the TransactionFlowAttribute. If the attribute is configured to allow transaction propagation, then DemandTransactionPermissions demands the distributed transaction permission. In a similar manner, DemandClientPermissions uses other helper methods to demand the appropriate permissions.

Finally, to enable partially trusted clients and callbacks, I defined the class PartialTrustDuplexClientBase<T,C>, and implemented it very much like PartialTrustClientBase<T>, one exception being that it adds the duplex support for clients receiving callbacks.

So that's a quick run through code access security in WCF, along with my solution for enabling partially trusted clients with appropriate levels of code-level permission. My next installment will demonstrate partially trusted services and partially trusted hosts. You will also get to see some advanced programming techniques for WCF and the .NET Framework.

Send your questions and comments to mmnet30@microsoft.com.

Juval Lowy is a software architect with IDesign providing WCF training and architecture consulting. He is also the Microsoft Regional Director for the Silicon Valley. His recent book is Programming WCF Services. Contact Juval at www.idesign.net.


Test Run: Testing SQL Stored Procedures Using LINQ

Test Run
Testing SQL Stored Procedures Using LINQ
Dr. James McCaffrey

Code download available at: TestRun2008_04.exe (156KB)
Contents
The System under Test
The Test Harness
Extending the Harness
Faster, More Compact, and Better Debugging

T
he need to test a program that accesses and manipulates a back-end SQL Server® database is very common. In many such cases, the application interacts with the back-end data through the use of SQL stored procedures. In this type of scenario, you can think of the stored procedures as auxiliary methods of the system under test; and they must therefore be tested just like any other module in the system.

Although there are several approaches you can take when testing SQL stored procedures, I have discovered that using LINQ dramatically simplifies test automation. In this month's column, I show you how to test SQL stored procedures using LINQ—specifically, using the LINQ to SQL provider. I'll assume you have intermediate level C# and SQL skills but no LINQ experience.

If you examine Figure 1, you'll see that I have written a small C# console application test harness that demonstrates the key activities you must perform when testing SQL stored procedures. I am testing a simple but representative stored procedure named usp_DeleteMovie, which resides in a database of movie-related data named dbMovies.

Figure 1 Testing a Stored Procedure Using LINQ
Figure 1 Testing a Stored Procedure Using LINQ(Click the image for a smaller view)
Figure 1 Testing a Stored Procedure Using LINQ
Figure 1 Testing a Stored Procedure Using LINQ (Click the image for a larger view)

For each test case, I initialize a test bed database, call usp_DeleteMovie with a particular movie ID, and then check to see if the resulting state of dbMovies is correct. What is not apparent in Figure 1 is that I am using LINQ, which makes it easier to create my test harness than it is when I use ADO.NET and T-SQL. In the following sections, I will describe the dummy dbMovies database and usp_DeleteMovie stored procedure I am testing. I'll also show you the code and explain the LINQ techniques used, and then I'll explain how you can modify and extend my examples in order to meet your own needs.

The System under Test

Before digging into the harness, let's look at the sample database of the system under test so that you'll understand the test automation. In a typical development environment, developers, testers, and database architects usually work with their own local copies of the back-end database. So, let's assume that you have access to a SQL script that creates the current version of the database under test. A realistic database script would be long and complex, but the sample database creation script in Figure 2 is simple in order to keep the main concepts clear.

Figure 2 Database Creation Script (Close Figure 2)
use master
go

if exists(select * from sys.sysdatabases where name='dbMovies')
  drop database dbMovies
go

create database dbMovies
go

use dbMovies
go

create table tblMain
(
  movID char(3) primary key,
  movTitle varchar(35) null,
  movRunTime int null
)
go

create table tblPrices
(
  movID char(3) primary key,
  movPrice money
)
go

create procedure usp_DeleteMovie
  @movID char(3)
as
  delete from tblMain where movID = @movID
  delete from tblPrices where movID = @movID
go

create procedure usp_GetMovieDataByPrice
  @movPrice money
as
  select m.movID, m.movTitle, p.movPrice, m.movRunTime
  from tblMain as m
  inner join tblPrices as p
    on m.movID = p.movID
  where p.movPrice = @movPrice
go

create procedure usp_GetMoviePrice
  @movID char(3),
  @price money out
as
  declare @ct int
  select @price = movPrice from tblPrices where movID = @movID
  select @ct = count(*) from tblPrices where movID = @movID
  return @ct
go

My database is named dbMovies and contains two tables, tblMain and tblPrices. The tblMain table has a movie ID column (which is the primary key), a title column, and a column that stores the duration of the movie (in minutes). The tblPrices table has a movie ID column and a column that stores the price of the movie.

My database has three stored procedures: usp_DeleteMovie, usp_GetMovieDataByPrice, and usp_GetMoviePrice. I precede the names of my stored procedures with usp_ to indicate that these are user-defined stored procedures as opposed to system stored procedures (sp_) or extended stored procedures (xp_). The points of note are that usp_DeleteMovie changes the state of the database but does not explicitly return a value; usp_GetMovieDataByPrice returns one or more SQL rowsets but does not change the state of dbMovies; and usp_GetMoviePrice returns two scalar values (one by a return value and one by an out-type parameter) but does not change database state.

These represent the three most common situations when testing SQL stored procedures: change state with no return value, return rowsets with no change of state, and return scalar values with no change of state. I will address all three scenarios shortly. Let's take a closer look at usp_DeleteMovie, the stored procedure being tested in Figure 1. The procedure accepts a movie ID as an input parameter and then deletes corresponding rows in tables tblMain and tblPrices, as you see here:

create procedure usp_DeleteMovie
  @movID char(3)
as
  delete from tblMain where movID = @movID
  delete from tblPrices where movID = @movID

From the perspective of test automation, you need to know how to prepare the dbMovies database associated with the stored procedure by setting it to a known state. Also, you'll want to know how to call the stored procedure and how to determine whether the stored procedure correctly modified the state of the database. As you'll see, performing these tasks with LINQ is much easier than with previous approaches.

The Test Harness

Let's walk through the code for the test harness that produced the output in Figure 1. The overall structure is shown in Figure 3, and the complete source code for all examples I discuss can be found in the code download that accompanies this column on the MSDN® Magazine Web site.

Figure 3 Test Harness Structure (Close Figure 3)
using System;
using System.Linq;
using System.Text;
using System.Data.Linq;
using System.Security.Cryptography;

namespace Run
{
  class Program
  {
    static void Main(string[] args)
    {
      try
      {
        Console.WriteLine("\nBegin test automation using LINQ");
        string connString =
          "server=(local);database=dbMovies;Trusted_Connection=true";
        Console.WriteLine
          ("\nStored procedure under test = usp_DeleteMovie");
        DbMovies db = new DbMovies(connString);

        string[] testCaseData = new string[] {
          "TC001:002:85-A0-D9-28-7B-AA-8D-5F-4C-43-B5-B6-A5-99-F4-63",
          "TC002:003:44-FA-D2-38-49-98-F0-BD-16-BD-9E-B6-0C-F1-73-35" };

        foreach (string testCase in testCaseData)
        {
          // parse inputs, expected state
          // call ResetDatabase()
          // call usp_DeleteMovie
          // call GetDatabaseState()
          // determine pass/fail
        }
        Console.WriteLine("\n====================================");
        Console.WriteLine("\nEnd test run");
      }
      catch (Exception ex)
      {
        Console.WriteLine("Fatal: " + ex.Message);
      }
    }

    static void ResetDatabase(string connString) { . . . }
    static string GetDatabaseState(string connString) { . . . }
  }
}

LINQ is part of the Microsoft® .NET Framework 3.5, which is included with Visual Studio® 2008. Using Visual Studio 2008, I created a C# console application targeting the .NET Framework 3.5. I deleted the "using System.Collections.Generic" statement generated by Visual Studio because my harness does not use any methods from that namespace. Notice that Visual Studio added a using statement to the System.Linq namespace by default. I added a reference to the System.Data.Linq assembly and then added the corresponding using statement.

LINQ is actually a term used to describe a collection of technologies including LINQ to SQL and LINQ to XML. LINQ to SQL was formerly called DLINQ, and you will still find many online references to that older term. To test stored procedures using LINQ, I use a combination of base LINQ functionality contained in System.Linq and LINQ to SQL functionality that is specified in the System.Data.Linq namespace. I also add a using statement that points to the System.Security.Cryptography namespace because I will be using a crypto-hash method to verify the database state. After an introductory WriteLine statement, I set up a connection string to my test bed database:

string connString =
  "server=(local);database=dbMovies;Trusted_Connection=true";

As you'll see shortly, it is useful to pass a connection string to auxiliary helper methods in the harness. Next comes a critical LINQ to SQL concept; I instantiate an object that represents my SQL database:

DbMovies db = new DbMovies(connString);

LINQ to SQL is not magic and does not automatically know about your database. Behind the scenes of my test harness I generated C# code that encapsulates all the information I need to access and manipulate the dbMovies database, including a DbMovies class. So where does this code come from?

There are three main approaches to get LINQ to SQL mapping code. You can use a command-line tool named sqlmetal.exe; you can use the Visual Studio 2008 Object Relational Designer (O/R Designer) GUI tool; or you can simply create the mapping code from scratch. I prefer to use the sqlmetal.exe tool because it is flexible and automation-friendly. To generate the C# wrapping code, I launched a Visual Studio 2008 command shell, navigated to the root directory of my test harness project, and entered this:

> sqlmetal.exe /server:(local) /database:dbMovies /sprocs
 /code:mapping.cs

In this case, I am using the default Windows® Authentication mode to connect to the dbMovies database, but sqlmetal.exe can accept user and password information if necessary. I specify the /sprocs command-line switch in order to have sqlmetal.exe generate code that wraps the stored procedures in dbMovies, and I use the /code switch to indicate that I want source code as output rather than an intermediate .dbml (database markup language) file such as that which would be generated by the O/R Designer tool.

After generating the mapping.cs file, I add the file to my Visual Studio project. The mapping.cs file contains all the wrapping code I need to access my SQL database including a DbMovies class with a constructor that accepts a connection string:

public partial class DbMovies : System.Data.Linq.DataContext
{
  public DbMovies(string connection) :
    base(connection, mappingSource)
  {
    //...
  }
  // wrappers to tables, stored procedures, etc.
}

Figure 4 illustrates a simplified relation between my test harness and my test bed database. After instantiating a DbMovies object in my test harness, I set up some test case data to exercise the usp_DeleteMovie stored procedure:

Figure 4 Relation between Database and Test Harness
Figure 4 Relation between Database and Test Harness(Click the image for a smaller view)
Figure 4 Relation between Database and Test Harness
Figure 4 Relation between Database and Test Harness (Click the image for a larger view)

string[] testCaseData = new string[] {
  "TC001:002:85-A0-D9-28-7B-AA-8D-5F-4C-43-B5-B6-A5-99-F4-63",
  "TC002:003:44-FA-D2-38-49-98-F0-BD-16-BD-9E-B6-0C-F1-73-35" };

Here I am only using two test cases; in a realistic testing scenario you might have hundreds or even thousands of test cases. For simplicity, I am embedding my test case data directly into my test harness. I could also have stored my data externally in the form of a flat text file, an XML file, a SQL table, or other data store. In general, external test case data is superior to internal test case data because external data is easier to modify and share.

My test case data has just three fields, delimited by colon characters. The first is a test case ID. The second represents a movie ID that will be used as an input to the usp_DeleteMovie stored procedure under test. The third is a string that represents a 16-byte MD5 crypto-hash value, which in turn represents the expected state of the database after the stored procedure under test has been called. I will explain this idea shortly. I use my test case data to control my main processing loop:

foreach (string testCase in testCaseData)
{
  // parse inputs, expected state
  // call ResetDatabase()
  // call usp_DeleteMovie
  // call GetDatabaseState()
  // determine pass/fail
}

Within the main processing loop, I use the String.Split method to parse my test case data string into individual string variables:

Console.WriteLine("\n====================================\n");
string[] data = testCase.Split(':');
string caseID = data[0];
string movieID = data[1];
string expectedState = data[2];
Console.WriteLine("Test case ID = " + caseID);
Console.WriteLine("\nID of movie to delete = " + movieID);

Now, before I can call the stored procedure under test I must set my database to a known state. This is a fundamental principle of testing SQL stored procedures (and databases in general). Many stored procedures change the state of their associated database by inserting, deleting, or updating data in one or more tables. In order to verify that a stored procedure operates correctly, you must examine the initial state of the database under test so that you can check for an expected state after the stored procedure has run. Setting a database to a known state is a task where LINQ really shines compared to previous technologies:

Console.WriteLine("Setting up test bed database");
ResetDatabase(connString);

To keep my Main method relatively clean, I farm out the work of setting up my test bed database to a helper method named ResetDatabase rather than place the code inline:

static void ResetDatabase(string connString)
{
  // delete all existing data in TblMain and TblPrices
  // populate TblMain and TblPrices with "rich" test data
}

I pass the connection string into my ResetDatabase method. First I delete all existing data in my tables, like so:

DbMovies dbLocal = new DbMovies(connString);
var allRowsInTblMain = from t in dbLocal.TblMain select t;
var allRowsInTblPrices = from t in dbLocal.TblPrices select t;
dbLocal.TblMain.DeleteAllOnSubmit(allRowsInTblMain);
dbLocal.TblPrices.DeleteAllOnSubmit(allRowsInTblPrices);
dbLocal.SubmitChanges();

I instantiate a DbMovies object. Next I use LINQ code to retrieve all existing data in both of my tables as LINQ to SQL entity collections. If you are new to LINQ, you may not have seen the implicit data type declaration mechanism using the var keyword. You can interpret var to mean "I want Visual Studio to determine the exact data type here based on the assignment code." In other words, as an alternative to using implicit typing such as this:

var allRowsInTblMain = from t in dbLocal.TblMain select t;

The following would have been an equally valid approach:

System.Linq.IQueryable<TblMain> allRowsInTblMain =
  from t in dbLocal.TblMain select t;

I really like implicit data typing in conjunction with LINQ. While writing the ResetDatabase code, I came across an issue that I want to alert you to. Originally I tried to use a single, static, global DbMovies object in my test harness rather than create a local object in my helper method ResetDatabase. However, using that approach I received the exception: "Cannot add an entity with a key that is already in use." It turns out that LINQ to SQL caches a lot of information for performance reasons. If you call a stored procedure, LINQ to SQL holds information about the entities involved, and then if you try to manipulate data related to those entities, you may get this error. A simple workaround is to avoid global objects when using LINQ to SQL.

After I have my entity collections, I pass them to the DeleteOnSubmit methods associated with each table. This effectively marks all rows in tblMain and tblPrices for deletion but does not actually perform any deletions. To execute the deletions, I issue a SubmitChanges statement, which will cause all pending changes, including my deletion requests, to be processed and propagated to the actual dbMovies database represented by the DbMovies object.

Now I am ready to populate my tables with rich data. When writing an application with a back-end database, you generally supply the database with normal data to use during the development process. But thorough stored procedure testing requires data specially designed to uncover potential problems. Here I hard-code some test bed data, which is not particularly rich in the testing sense but keeps the ideas clear:

string[] data = new string[] {
  "001:Active Assembly:101:11.11",
  "002:Bug Bashing Bit:102:22.22",
  "003:Computing in C#:103:33.33",
  "004:Data is Digital:104:22.22"
};

In a realistic scenario, I would add much more data and include null values, duplicate values, illegal values, boundary values, and so on. I could have read this rich test bed data from an external data store, but there is less reason to do so for this setup data than there is with the actual test case data. I parse my test bed data and then set up insert requests, as you see in Figure 5.

Figure 5 Create LINQ Insert Requests (Close Figure 5)
foreach (string entry in data)
{
  string[] tokens = entry.Split(':');
  TblMain tm = new TblMain();
  tm.MovID = tokens[0];
  tm.MovTitle = tokens[1];
  tm.MovRunTime = int.Parse(tokens[2]);
  TblPrices tp = new TblPrices();
  tp.MovID = tokens[0];
  tp.MovPrice = decimal.Parse(tokens[3]);
  dbLocal.TblMain.InsertOnSubmit(tm);
  dbLocal.TblPrices.InsertOnSubmit(tp);
}
dbLocal.SubmitChanges();

I instantiate TblMain and TblPrices objects. These classes are defined in the code generated by the sqlmetal.exe tool. Because LINQ is simply an extension of the .NET Framework, I get all the advantages associated with the .NET environment including IntelliSense® code-completion and design-time syntax checking. But let me mention a test design pitfall you should avoid when populating a test bed database with rich data. In a realistic scenario, the database under test will contain stored procedures that insert data. You should definitely not use such procedures to populate your test bed—you would effectively be using part of the system under test to verify the system under test. Now, back in the main processing loop in my test harness, after my database has been set to a known state I am now ready to call the stored procedure under test. Once again, LINQ to SQL greatly simplifies this task:

Console.WriteLine("Calling usp_DeleteMovie");
db.Usp_DeleteMovie(movieID);

The Usp_DeleteMovie C# method is a friendly stand-in for the underlying usp_DeleteMovie stored procedure. Trust me, this is much easier than calling a stored procedure using ADO.NET—particularly in cases where there are many parameters. Now I am ready to determine if the stored procedure under test correctly modified the associated database. Determining database state, like resetting database state to a known configuration, is another fundamental task when testing stored procedures:

Console.WriteLine("Checking resulting state of test bed database");
string actualState = GetDatabaseState(connString);
Console.WriteLine("Expected state = " + expectedState);
Console.WriteLine("Actual state   = " + actualState);

In order to fetch the expected and actual database states, I use another helper method named GetDatabaseState:

static string GetDatabaseState(string connString)
{
  // combine data from tblMain and tblPrices into an aggregate string
  // convert aggregate to an MD5 byte array
  // return string image of byte array
}

First, I use LINQ to easily retrieve all data in my tables:

DbMovies db = new DbMovies(connString);
var allRowsInTblMain = from t in db.TblMain select t;
var allRowsInTblPrices = from t in db.TblPrices select t;

And then I build up an aggregate string by concatenating the string representations of each column value:

string s = "";
foreach (var record in allRowsInTblMain)
  s += record.MovID + record.MovTitle + record.MovRunTime;
foreach (var record in allRowsInTblPrices)
  s += record.MovID + record.MovPrice;

Now, I could use this aggregate string as my actual state and compare the string against an expected string. However, a string representation of the values in a test bed database could be very large, and some SQL data types, such as binary, don't map well to string representations. A better approach in most situations is to compute and use a hash of the values in the database:

MD5CryptoServiceProvider md5 =
  new MD5CryptoServiceProvider();
byte[] ba = md5.ComputeHash(Encoding.Unicode.GetBytes(s));
string cryptoHash = BitConverter.ToString(ba);
return cryptoHash;

I compute an MD5 (Message Digest version 5) 16-byte value of the aggregate string. I could have just as easily used a different hashing algorithm such as SHA-1 (Secure Hash Algorithm version 1). Crypto-hashes like MD5 accept a byte array of any size and produce a fingerprint-like identifying value that is always a fixed size. For readability, I convert the resulting byte array to a string using the static System.BitConverter class. Now I can determine if the stored procedure under test modified the state of dbMovies as expected or not:

if (expectedState == actualState)
  Console.WriteLine("\nTest case result = Pass");
else
  Console.WriteLine("\nTest case result = **FAIL**");

Here I just display a test result to the shell. In a production environment you can save your results to external storage. Recall that my test case data contained an expected database state such as 85-A0-D9-28-7B-AA-8D-5F-4C-43-B5-B6-A5-99-F4-63. So just where did this expected value come from? Determining expected state is the most time-consuming task when testing stored procedures. You must manually determine the expected state of your test bed database after a stored procedure has been called, given an initial state, and then use a utility program to compute an MD5 (or similar) crypto-hash of all or part of the values in expected state.

A common approach used to determine expected state, but one which is incorrect, is to run your test harness without supplying an expected value, and then fetch the hash value of the resulting state of the database. This approach creates test case data that merely verifies that the stored procedure under test produces the same, possibly incorrect, state.

Extending the Harness

Let's take a look at testing stored procedures that return a SQL rowset but that do not affect the data in the associated database. Recall that the usp_GetMoveDataByPrice accepts a movie price and returns movie ID, title, runtime, and price data for all movies that have the specified price. To call usp_GetMovieDataByPrice, you can write code along the lines of:

DbMovies db = new DbMovies(connString);
decimal moviePrice = (decimal)22.22;
var results = dbc.Usp_GetMovieDataByPrice(moviePrice);

Very, very clean. Here I am using an implicit data type for object results. I could have also used an explicit data type for results and written:

System.Data.Linq.ISingleResult<Usp_GetMovieDataByPriceResult> results =
  db.Usp_GetMovieDataByPrice(moviePrice);

Now, to verify the correctness of the rowset returned into object results, I can first fetch an aggregate string composed of the column values of the returned rowset:

string s = "";
foreach (var r in results)
  s += r.MovID + r.MovTitle + r.MovRunTime + r.MovPrice;

And I can use this aggregate string as an actual return value, or I can compute a hash of the string as explained in the previous section. Notice that usp_GetMovieDataByPrice does not explicitly affect the state of dbMovies because it does not change the data in tblMain and tblPrice. This means that in your test harness you can omit the call to the ResetDatabase helper method if you wish.

Testing a stored procedure that returns a scalar value is generally the easiest scenario to test because there is a single expected result. For example, consider usp_GetMoviePrice, which is defined as:

create procedure usp_GetMoviePrice
  @movID char(3),
  @price money out
as
  declare @ct int
  select @price = movPrice from tblPrices where movID = @movID
  select @ct = count(*) from tblPrices where movID = @movID
  return @ct

This stored procedure effectively returns two scalar values; the price of a movie with a specified ID is stored into an out parameter and the number of movies found with the specified ID is an explicit return value. To call usp_GetMoviePrice using LINQ to SQL, you can write code like this:

DbMovies db = new DbMovies(connString);
string movieID = "002";
decimal? actualPrice = null;
int actualCount = dbc.Usp_GetMoviePrice(movieID, ref actualPrice);

The question mark character after the decimal keyword may also be new to you. This is an example of a nullable type. Because all SQL data values can be null, the wrapping code generated by sqlmetal.exe and the O/R Designer tools generate most data members as nullable. Figure 6 shows how you would typically call uspGetMoviePrice using ADO.NET. I think you'll agree that the LINQ approach is significantly simpler.

Faster, More Compact, and Better Debugging

Based on my initial experience with LINQ and LINQ to SQL, I estimate that using LINQ reduces the time and effort to create test automation for stored procedures by at least 50 percent. Because LINQ is a part of the .NET Framework, you get the benefits of working directly with any .NET Framework-compliant language like C#. Instead of typing something like this

SqlCommand sc = new SqlCommand("INSERT INTO tblFoo VALUES('" +
someValue + "', '" + otherValue + "')";

and then hoping you got all the quotes and commas in the right place, LINQ lets you write normal C#-style code and reports syntax errors immediately.

Because LINQ to SQL is essentially a well-designed wrapper around lower-level code (often classes like SqlDataReader), your test code is much shorter, much cleaner, and therefore easier to create, modify, and maintain. Because LINQ is a part of the .NET environment, you can use the Visual Studio debugger to track down errors in your test harness—something that is awkward or impossible with pre-LINQ approaches in many scenarios. Maybe most importantly, LINQ just feels right.

While researching this column, I found myself really enjoying the experimentation process. LINQ encourages investigatory activities because it makes your life as a developer/tester so easy. This characteristic of LINQ promotes more thorough testing, which will produce better software products.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft's Redmond, Washington, campus. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes, and he can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Service Station: Building a WCF Router, Part 1

Service Station
Building a WCF Router, Part 1
Michele Leroux Bustamante

Code download available at: ServiceStation2008_04.exe (238KB)
Contents
Default Addressing Semantics
Routing Architecture
The Router Contract
Forwarding Messages
Logical and Physical Addressing
MustUnderstand Headers

H
osting and consuming a Windows® Communication Foundation (WCF) service usually requires a few fundamental steps: implement the service, configure endpoints where the service can be reached, host the service, generate a Web Services Description Language (WSDL) file or enable metadata exchange so that clients can generate a proxy to call the service, write code to instantiate the proxy with its associated configuration, and start calling service operations. You rarely need to take a look under the hood, but even in the simplest case, both the client and service channels rely on compatible configuration to handle addressing semantics and message filtering to ensure the correct operation is invoked.

Sometimes it is useful to introduce an intermediary or router service between a client and a target service to receive messages that flow between them and perform additional activities such as logging, priority routing, online/offline routing, load balancing, or to introduce a security boundary. When such an intermediate service is introduced, it becomes necessary to tweak some addressing and message filtering behaviors to accommodate.

So let's take a closer look at how you work with intermediate services, which for simplicity I'll refer to collectively as routers. In this issue, I'll explain WCF addressing and message filtering concepts while focusing on the router scenario, and I'll also explain some options for routing configuration along with the appropriate settings. In Part 2 of this series, I'll show you how to leverage this foundation for more advanced, practical routing implementations.

Default Addressing Semantics

In the June 2007 Service Station column (msdn.microsoft.com/msdnmag/issues/07/06/ServiceStation), Aaron Skonnard explained how WCF handles logical and physical endpoint addressing, addressing headers, and message filtering. In this section, I will review some of those fundamental addressing features and how they impact routing scenarios—but you will also find Aaron's column useful for additional details behind these WCF features.

Typically, clients send messages directly to the target service using a proxy generated from the service description. In order for the client and service to be compatible, they share equivalent contracts and endpoint configurations. Considering the service contract and configuration shown in Figure 1, you can derive several important addressing requirements for the service.

Figure 1 Service Contract and Endpoint Configuration (Close Figure 1)
            Service Contract
[ServiceContract(Namespace =
"http://www.thatindigogirl.com/samples/2008/01")]
public interface IMessageManagerService
{
  [OperationContract]
  string SendMessage(string msg);

  [OperationContract]
  void SendOneWayMessage(string msg);
}
Endpoint Configuration
<system.serviceModel>
  <services>
    <service name="MessageManager.MessageManagerService"
      behaviorConfiguration="serviceBehavior">
      <endpoint
        address="http://localhost:8000/MessageManagerService"
        contract="MessageManager.IMessageManagerService"
        binding="basicHttpBinding" />
    </service>
  </services>
</system.serviceModel>

First, the expected Action addressing header for a request to the SendMessage operation is:

http://www.thatindigogirl.com/samples/2008/01/IMessageManagerService/SendMessage

Since OperationContractAttribute doesn't specify an Action, this value is derived from the service contract namespace, the contract name (which defaults to the interface name), and the operation name (which defaults to the method name).

Second, the expected Action header for a response from SendMessage is:

http://www.thatindigogirl.com/samples/2008/01/IMessageManagerService/SendMessageResponse

Since the OperationContractAttribute doesn't specify a ReplyAction, this value is derived in the same way as the Action property, with the suffix "Response" appended.

Finally, the expected To header for messages targeting the service endpoint is:

http://localhost:8000/MessageManagerService

This value is derived from the address attribute for the endpoint element, which is considered the logical address of the endpoint. While it can be otherwise specified, the physical address of the endpoint usually matches the logical address. This means clients usually send messages to a physical address that matches the To header.

Service metadata describes these requirements such that clients can generate a compatible proxy and configuration. The service contract generated for the client reflects the same Action and ReplyAction settings of the service, and the client binding configuration will reflect an endpoint with the appropriate logical and physical address. For example, the following client endpoint is compatible with the service in Figure 1:

<client>
  <endpoint
    address="http://localhost:8000/MessageManagerService"
    binding="basicHttpBinding"
    contract="localhost.IMessageManagerService"
    name="basicHttp" />
</client>

The client proxy uses the address property of the client endpoint element for both its logical and physical address. As a result, messages are sent to a physical address that matches the To header, as I indicated earlier. When the proxy calls the SendMessage operation, a message is sent with the To and Action headers that are shown in Figure 2.

Figure 2 Message Sent from Proxy (Close Figure 2)
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <To s:mustUnderstand="1"
      xmlns="http://schemas.microsoft.com/ws/2005/05/addressing/none">
      http://localhost:8000/MessageManagerService
    </To>
    <Action s:mustUnderstand="1"
      xmlns="http://schemas.microsoft.com/ws/2005/05/addressing/none">
      http://www.thatindigogirl.com/samples/2008/01
        /IMessageManagerService/SendMessage
    </Action>
  </s:Header>
  <s:Body>
    <SendMessage xmlns="http://www.thatindigogirl.com/samples/2008/01">
      <msg>test</msg>
    </SendMessage>
  </s:Body>
</s:Envelope>

The combination of the To and Action headers, respectively, indicate to the service model which channel dispatcher should process the message and which operation the dispatcher should invoke. By default, there must be an endpoint with a matching logical address to the To header and an operation that matches the Action header. Figure 3 illustrates this flow.

Figure 3 Typical Addressing without a Router
Figure 3 Typical Addressing without a Router(Click the image for a smaller view)
Figure 3 Typical Addressing without a Router
Figure 3 Typical Addressing without a Router (Click the image for a larger view)

In the following sections I will explain the implications of the logical and physical address, the To and Action headers, and the message filtering rules when a router is introduced into the scenario.

Routing Architecture

Although there are some variations on how you might design a router, most routers must be able to receive messages that can target any service, and they must be able to forward the original message to the appropriate target service. There are two fundamental approaches to designing a router: a pass-through router or a processing router.

A pass-through router is transparent to the client. The client's relationship is with downstream services, but messages just happen to flow through the router. The client must send the message to the router using a compatible transport protocol and message encoder, but any security, reliable sessions, application sessions, or other messaging protocols required by the service channel are satisfied by the message generated at the client. The router may look at message headers or even inject headers, but the original message elements are still forwarded to the service unaltered. Figure 4 illustrates this relationship.

Figure 4 Operation of a Pass-Through Router
Figure 4 Operation of a Pass-Through Router(Click the image for a smaller view)
Figure 4 Operation of a Pass-Through Router
Figure 4 Operation of a Pass-Through Router (Click the image for a larger view)

A processing router plays a more active role in processing messages for the application. Thus, the client's relationship is with the router as far as transport, encoding, and protocol compatibility go, though the client must still additionally be able to send messages that are compatible with downstream services. Messages flow through the router to services, and the message body—along with any required headers for the service—is left intact.

Security, reliable sessions, and headers or messages related to other communication protocols are usually handled by the router, and the router builds a new message with appropriate communication protocols for downstream services. Figure 5 illustrates compatibility for a processing router.

Figure 5 Operation of a Processing Router
Figure 5 Operation of a Processing Router(Click the image for a smaller view)
Figure 5 Operation of a Processing Router
Figure 5 Operation of a Processing Router (Click the image for a larger view)

Each routing configuration has practical implementations. It is also possible to implement a hybrid solution somewhere between the two.

The Router Contract

Routers receive messages intended for downstream services and are responsible for forwarding those messages to the appropriate service. They are also responsible for receiving responses from the service and returning those to the client. A typical router contract exposes a single operation that can process any message request or response. In the following example, that operation is called ProcessMessage:

[ServiceContract(Namespace =
"http://www.thatindigogirl.com/samples/2008/01")]
public interface IRouterService {
  [OperationContract(Action = "*", ReplyAction = "*")]
  Message ProcessMessage(Message requestMessage);
}

In a normal situation, the Action and ReplyAction properties of the OperationContractAttribute are derived as discussed earlier in this column—from the service contract namespace, the contract name, and the operation name. By default, when a message arrives, the channel dispatcher must find an operation to match the exact Action header. However, in a case when Action and ReplyAction are set to "*", the channel dispatcher will send any messages not mapped to a specific operation to that catch-all operation regardless of the Action header value. And in order to avoid ambiguity, only one operation can specify "*" for the Action or ReplyAction property.

A typical router supplies a single operation like ProcessMessage that can handle any message that reaches it. While the Action and ReplyAction will ensure the message is mapped to ProcessMessage by the channel dispatcher, the operation signature must also be capable of processing any message.

In order to address this, ProcessMessage receives and returns untyped messages in the form of the Message type. The router can access the header collection and the message body through this type, but no automatic serialization occurs beyond the common addressing headers, which are deserialized and made available through strongly typed properties.

Any further processing of the messages is up to the router implementation. A basic router will simply receive the untyped message and forward it as is to downstream services, awaiting a reply. Likewise, the reply is forwarded to the calling client in the same raw format.

Forwarding Messages

After the router receives a message and processes it according to its own requirements, it forwards the message to the appropriate downstream service for further processing. A simple router implementation for the contract discussed earlier is shown in Figure 6. ProcessMessage constructs a client channel (or proxy) with ChannelFactory<T> and uses this proxy to forward messages to a particular service endpoint, returning any responses.

Figure 6 A Simple Router Implementation (Close Figure 6)
[ServiceBehavior(
  InstanceContextMode = InstanceContextMode.Single,
  ConcurrencyMode = ConcurrencyMode.Multiple)]
public class RouterService : IRouterService {
  public Message ProcessMessage(Message requestMessage) {
    using (ChannelFactory<IRouterService> factory =
      new ChannelFactory<IRouterService>("serviceEndpoint")) {

      IRouterService proxy = factory.CreateChannel();

      using (proxy as IDisposable) {
        return proxy.ProcessMessage(requestMessage);
      }
    }
  }
}

Proxies are usually strongly typed to the target service contract, but in this case the proxy should be able to forward any message and receive any reply—something that the router contract facilitates. In this simple example, the router merely forwards the original message to the target service and returns any response. If the operation at the target service is one-way, no reply is sent.

Since the contract works with untyped messages, the same message is forwarded to the service, as shown in Figure 7. But you must realize that there is, however, one change made to the message that you may not expect: the To header is altered before the message is sent to the service.

Figure 7 Addressing Semantics through a Simple Router
Figure 7 Addressing Semantics through a Simple Router(Click the image for a smaller view)
Figure 7 Addressing Semantics through a Simple Router
Figure 7 Addressing Semantics through a Simple Router (Click the image for a larger view)

Recall that, by default, the proxy will use the logical address from its endpoint configuration to set the To header for outgoing messages—even if a raw Message instance is passed that already has a To header. While this may seem like a good thing—since all services require that the To header match the logical address of one of their service endpoints—it can cause other side effects. For one, if the updated To header is not signed and the service has security enabled, the message will be rejected.

Ideally, the client should send a message with a To header matching the target service, the router should accept that message despite the mismatch, and the router should forward this message to the service without altering the To header. This can be handled through binding configurations, which I'll discuss next.

Logical and Physical Addressing

When you introduce a router to the application architecture, it is always best if the client can send messages using the correct To header for the service while still sending the message to the router. One way to achieve this is to configure the client to use ClientViaBehavior, as in the configuration shown in Figure 8. This tells the client proxy to generate a message with a To header according to the endpoint's logical address, but to send it via the router's physical address. The problem is that this couples the client to the existence of a router.

Figure 8 Using ClientViaBehavior (Close Figure 8)
<client>
  <endpoint address="http://localhost:8000/MessageManagerService"
    binding="wsHttpBinding"
    bindingConfiguration="wsHttpNoSecurity"
    contract="localhost.IMessageManagerService"
    name="basicHttp"
    behaviorConfiguration="viaBehavior"/>
</client>
<behaviors>
  <endpointBehaviors>
    <behavior name="viaBehavior">
      <clientVia viaUri="http://localhost:8010/RouterService"/>
    </behavior>
  </endpointBehaviors>
</behaviors>

Another way to address this is to have the service configure the listenUri attribute for its endpoints so that the logical address of the service is that of the router while the physical address is specific to the service. Consider this service configuration:

<endpoint address="http://localhost:8010/RouterService"
  contract="MessageManager.IMessageManagerService"
  binding="wsHttpBinding"
  bindingConfiguration="wsHttpNoSecurity"
  listenUri="http://localhost:8000/MessageManagerService"/>

The resulting service metadata publishes the router address to clients, so client endpoints reflect the router address. I don't really like this solution because it couples the service to the router and ideally the service doesn't need to know about it either.

The alternative is to have services use a logical address that is a type of URI that is not tied to the router or service, and then manually tell clients the physical address to send messages to since it won't be part of the metadata. Here is an example of this type of endpoint configuration:

<endpoint address="urn:MessageManagerService"
  contract="MessageManager.IMessageManagerService"
  binding="wsHttpBinding"
  bindingConfiguration="wsHttpNoSecurity"
  listenUri="http://localhost:8000/MessageManagerService"/>

In either case, the service will receive a To header matching its endpoint configuration and the router will receive the message first.

The router should really bear the burden of configuration so that clients and services are not dependent on its presence. Thus, the To header will never match the router's logical address. By default, services use the EndpointAddressMessageFilter to determine whether a message's To header matches any of its configured endpoints. Since a router can't expect this to work, it should install the MatchAllMessageFilter.

The ServiceBehaviorAttribute supports this through the AddressFilterMode property, which can be set to one of the AddressFilterMode enumerations: Exact (the default), Prefix, or Any. Since the router prefix can't be guaranteed to match all services it receives messages for, it makes sense to allow all To headers to pass through, like so:

[ServiceBehavior(InstanceContextMode =
  InstanceContextMode.Single,
  ConcurrencyMode = ConcurrencyMode.Multiple,
  AddressFilterMode=AddressFilterMode.Any)]
public class RouterService : IRouterService

By default, the To header will always be updated to match a proxy's logical address, based on its endpoint configuration—regardless of whether the To address is already set to the correct value. To suppress this behavior so that the router can forward messages to the service using the original To header, the router must use a binding configuration with manual addressing. This is not a property that can be set on any of the standard bindings, so you must use custom binding to achieve this.

The following code snippet illustrates a customBinding section that sets this feature for the HTTP transport channel:

<customBinding>
  <binding name="manualAddressing">
    <textMessageEncoding />
    <httpTransport manualAddressing="true"/>
  </binding>
 </customBinding>

This facilitates the addressing flow shown in Figure 9 where headers are not altered.

Figure 9 Addressing Semantics through a Router with Manual Addressing
Figure 9 Addressing Semantics through a Router with Manual Addressing(Click the image for a smaller view)
Figure 9 Addressing Semantics through a Router with Manual Addressing
Figure 9 Addressing Semantics through a Router with Manual Addressing (Click the image for a larger view)

MustUnderstand Headers

So far I have focused on a simple routing implementation to illustrate core router design considerations that also impact addressing, filtering, and binding configuration. This simple routing solution only works if the service does not enable security, reliable sessions, or any other rich protocol for its bindings. Figure 10 shows a simplified view of the binding protocols I'm assuming for the discussion thus far.

Figure 10 Service Contract and Endpoint Configuration
Figure 10 Service Contract and Endpoint Configuration(Click the image for a smaller view)
Figure 10 Service Contract and Endpoint Configuration
Figure 10 Service Contract and Endpoint Configuration (Click the image for a larger view)

Figure 11 shows the same view for a pass-through router, when the service requires security and reliable sessions. Enabling these protocols means that the client and service channels will exchange additional messages to establish sessions, request security tokens, and other related messaging. Since the router allows all messages to pass through, these protocol-specific messages will also pass through to the service—and that is a good thing.

Figure 11 Pass-Through Configuration with Security and Reliable Sessions
Figure 11 Pass-Through Configuration with Security and Reliable Sessions(Click the image for a smaller view)
Figure 11 Pass-Through Configuration with Security and Reliable Sessions
Figure 11 Pass-Through Configuration with Security and Reliable Sessions (Click the image for a larger view)

A problem arises, however, when the messages to and from the service include headers that must be understood by their recipient. Since the pass-through router purposely does not have security or reliable sessions enabled, those channels are not present to process associated protocol headers.

You can instruct the router service to ignore MustUnderstand headers by setting the ValidateMustUnderstand property of the ServiceBehaviorAttribute to false, as shown here:

[ServiceBehavior(InstanceContextMode =
  InstanceContextMode.Single,
  ConcurrencyMode = ConcurrencyMode.Multiple,
  AddressFilterMode=AddressFilterMode.Any,
  ValidateMustUnderstand=false)]
public class RouterService : IRouterService

This will address incoming messages from the client, but it won't address messages returned from downstream services.

To address this, you must also modify the router implementation to specify this behavior when initializing the channel factory for calling the downstream service, like so:

using (ChannelFactory<IRouterService> factory =
  new ChannelFactory<IRouterService>("serviceEndpoint"))
{
  factory.Endpoint.Behaviors.Add(new MustUnderstandBehavior(false));
  IRouterService proxy = factory.CreateChannel();

  // remaining code
}

Now, protocol and service messages can freely flow between client and service, through the router—assuming that the protocol used is HTTP.

Another complication arises when duplex protocols such as TCP or named pipes are employed. This means that messages can be initiated by the service to the client, such as when reliable sessions are enabled. There is an advanced router configuration you can use to handle this special case, but I'll address that scenario and its practicality in Part 2 of this series.

Send your questions and comments to sstation@microsoft.com.

Michele Leroux Bustamante is Chief Architect of IDesign Inc., Microsoft Regional Director for San Diego, and a Microsoft MVP for Connected Systems. Her latest book is Learning WCF. Reach her at mlb@idesign.net or visit design.net. Michele blogs at dasblonde.net.


Windows with C++: Windows Imaging Component Basics

Windows with C++
Windows Imaging Component Basics
Kenny Kerr

Contents
Getting Started
Decoding Images
Encoding Images
The WIC Imaging Factory
Working with Streams
WIC via WPF
What's Next?

T
he Microsoft® Windows® Imaging Component (WIC) is an extensible framework for encoding, decoding, and manipulating images. Originally designed for Windows Vista® and Windows Presentation Foundation (WPF), today WIC not only ships with Windows Vista and the Microsoft .NET Framework 3.0 and later, but it is also available as a download for Windows XP and Windows Server® 2003 for use in native applications.

One of several powerful native frameworks that power WPF, WIC in this context is the framework used in the implementation of the System.Windows.Media.Imaging namespace. It is, however, also ideally suited for native applications written in C++ because it provides a simple-yet-powerful API exposed through a set of COM interfaces.

WIC supports different image formats using an extensible set of imaging codecs. Each codec supports a different image format and typically provides both an encoder and decoder. WIC includes a set of built-in codecs for virtually all of the major image formats including PNG, JPEG, GIF, TIFF, HD Photo (HDP), ICO, and of course the Windows BMP.

HDP is the only format you may not have heard about. It was originally called Windows Media Photo and was developed in conjunction with Windows Vista to overcome some of the limitations with existing formats and provide better performance and higher image quality. For more information on HDP, check out the specification at microsoft.com/whdc/xps/wmphoto.mspx. Fortunately, WIC provides great support for this new image format, so applications don't have to know the specifics of the formats to use them.

This month I'll show you how to use WIC to encode and decode different image formats and a few things in between. Next time I'll explore some of the more advanced features and show you how to extend WIC with your own imaging codecs.

Getting Started

The WIC API consists of COM interfaces, functions, structures, and error codes, as well as GUIDs identifying various codecs, containers, and formats. All of the declarations you will need are included in the wincodec.h and wincodecsdk.h header files that are provided as part of the Windows SDK (and included with Visual Studio® 2008). It will also be necessary for you to link to the WindowsCodecs.lib library, which provides the various definitions you may need. You can add the following to your project's precompiled header to make it all available:

#include <wincodec.h>
#include <wincodecsdk.h>
#pragma comment(lib, "WindowsCodecs.lib")

Since the WIC API consists primarily of COM interfaces, I use the Active Template Library (ATL) CComPtr class to handle the creation and management of the interface pointers. If you would like to do the same, you'll need to also include the atlbase.h header file that defines the CComPtr template class:

#include <atlbase.h>

The WIC API also uses the COM library, so the CoInitializeEx function must be called on any thread that will use the API.

Finally, the WIC API makes use of HRESULTs to describe errors. The samples in this article use the HR macro to clearly identify where methods return an HRESULT that needs to be checked. You can replace this with your own error-handling strategy—whether that is throwing an exception or returning the HRESULT yourself.

Decoding Images

Decoders are represented by the IWICBitmapDecoder interface. WIC provides a few ways to create a decoder object, but at a minimum you can simply use a particular decoder's CLSID to create an instance. The following example creates a decoder for a TIFF image:

CComPtr<IWICBitmapDecoder> decoder;
HR(decoder.CoCreateInstance(CLSID_WICTiffDecoder));

Figure 1 lists the codecs included with WIC and the CLSIDs you can use to create the different decoders. Once the decoder is created, it needs to be initialized with a stream containing the pixels and optional metadata in a format understood by the decoder:

Figure 1 CLSIDs for Built-In WIC Codecs (Close Figure 1)

Format | Decoder | Encoder
BMP | CLSID_WICBmpDecoder | CLSID_WICBmpEncoder
PNG | CLSID_WICPngDecoder | CLSID_WICPngEncoder
ICO | CLSID_WICIcoDecoder | Not available
JPEG | CLSID_WICJpegDecoder | CLSID_WICJpegEncoder
GIF | CLSID_WICGifDecoder | CLSID_WICGifEncoder
TIFF | CLSID_WICTiffDecoder | CLSID_WICTiffEncoder
HDP | CLSID_WICWmpDecoder | CLSID_WICWmpEncoder

CComPtr<IStream> stream;

// Create stream object here...

HR(decoder->Initialize(
  stream,
  WICDecodeMetadataCacheOnDemand));

I'll discuss streams later in this article, but IStream is just the traditional COM stream interface used by many APIs including the XmlLite parser I covered in the April 2007 issue of MSDN® Magazine (msdn.microsoft.com/msdnmag/issues/07/04/Xml).

The second parameter to the Initialize method describes how you would like the decoder to read the image information from the stream. WICDecodeMetadataCacheOnDemand indicates that the decoder should only read the image information from the stream as it is needed. This can be particularly useful if the image format happens to contain or support multiple frames. The alternative is WICDecodeMetadataCacheOnLoad, which indicates that the decoder should cache all image information immediately. All subsequent requests to the decoder would then be fulfilled directly from memory. This has some further implications for managed code that I'll talk about later.

With the decoder initialized, you can freely query the decoder for information. The most common thing you're likely to ask for is the set of frames that make up an image. The frames are the actual bitmaps that contain pixels. You can think of the image format as a container for frames. As I've mentioned, some image formats support multiple frames.

The GetFrameCount function is used to determine the number of frames in the image:

UINT frameCount = 0;
HR(decoder->GetFrameCount(&frameCount));

Given the number of frames, individual frames can be retrieved using the GetFrame method:

for (UINT index = 0; index < frameCount; ++index)
{
    CComPtr<IWICBitmapFrameDecode> frame;

    HR(decoder->GetFrame(index, &frame));
}

The IWICBitmapFrameDecode interface returned by GetFrame derives from the IWICBitmapSource interface that represents a read-only bitmap. IWICBitmapFrameDecode provides information associated with the frame such as metadata and color profiles. IWICBitmapSource provides the bitmap's size and resolution, pixel format, and other optional characteristics such as its color table. IWICBitmapSource also provides the CopyPixels method that can be used to actually read pixels from the bitmap.

You can get the dimensions of the frame expressed in pixels using the GetSize method:

UINT width = 0;
UINT height = 0;
HR(frame->GetSize(&width, &height));

And you can get the resolution of the frame expressed in dots per inch (dpi) using the GetResolution method:

double dpiX = 0;
double dpiY = 0;
HR(frame->GetResolution(&dpiX, &dpiY));

Although the resolution has no impact on the pixels themselves, it does affect how the image can be displayed when using a logical coordinate system such as that used by WPF.

The last critical attribute of the frame is the pixel format. The pixel format describes the layout of pixels in memory and also implies the range of colors or color space that it supports. The GetPixelFormat method returns the pixel format:

GUID pixelFormat = { 0 };
HR(frame->GetPixelFormat(&pixelFormat));

The pixel formats are defined as GUIDs whose names pretty clearly describe the memory layout. For example, the GUID_WICPixelFormat24bppRGB format indicates that each pixel uses 24 bits (3 bytes) of storage with 1 byte per color channel. In addition, the ordering of the red (R), green (G), and blue (B) letters indicates the order of the bytes from least significant to most significant. As an example, the GUID_WICPixelFormat32bppBGRA format indicates that each pixel uses 32 bits (4 bytes) of storage with 1 byte per color channel as well as 1 byte for the alpha channel. In this case, the channels are ordered with the blue (B) channel being least significant and the alpha (A) channel being the most.

The actual pixels can be retrieved using the CopyPixels method.

HRESULT CopyPixels(
  const WICRect* rect,
  UINT stride,
  UINT bufferSize,
  BYTE* buffer);

The rect parameter specifies a rectangle within the bitmap to copy. You can set this parameter to zero and it will copy the entire bitmap. I'll talk about the stride in a moment. The buffer and bufferSize parameters indicate where the pixels will be written and how much space is available.

The stride can be one of the more confusing aspects of bitmaps. Stride is the count of bytes between scanlines. Generally speaking, the bits that make up the pixels of a bitmap are packed into rows. A single row should be long enough to store one row of the bitmap's pixels. The stride is the length of a row measured in bytes, rounded up to the nearest DWORD (4 bytes). This allows bitmaps with fewer than 32 bits per pixel (bpp) to consume less memory while still providing good performance. You can use the following function to calculate the stride for a given bitmap:

UINT GetStride(
  const UINT width, // image width in pixels
  const UINT bitCount) { // bits per pixel
  ASSERT(0 == bitCount % 8);

  const UINT byteCount = bitCount / 8;
  const UINT stride = (width * byteCount + 3) & ~3;

  ASSERT(0 == stride % sizeof(DWORD));
  return stride;
}

With that out of the way, you can call the CopyPixels method as follows, assuming the frame represents a 32 bpp bitmap:

const UINT stride = GetStride(width, 32);
CAtlArray<BYTE> buffer;
VERIFY(buffer.SetCount(stride * height));

HR(frame->CopyPixels(
  0, // entire bitmap
  stride,
  buffer.GetCount(),
  &buffer[0]));

My example uses the ATL CAtlArray collection class to allocate the buffer, but you can obviously use whatever storage you like. To handle larger bitmaps more efficiently, you can call CopyPixels a number of times to read different portions of the bitmap.

Encoding Images

Encoders are represented by the IWICBitmapEncoder interface. As with decoders, WIC provides a few ways to create encoders, but at a minimum you can simply use a particular encoder's CLSID to create it. This code, for example, creates an encoder for a PNG image:

CComPtr<IWICBitmapEncoder> encoder;
HR(encoder.CoCreateInstance(CLSID_WICPngEncoder));

Figure 1 lists the CLSIDs you can use to create the different encoders included with WIC. After the encoder is created, it needs to be initialized with a stream that will ultimately receive the encoded pixels and optional metadata:

CComPtr<IStream> stream;

// Create stream object here...

HR(encoder->Initialize(
  stream,
  WICBitmapEncoderNoCache));

The second parameter to the Initialize method is less interesting, as WICBitmapEncoderNoCache is the only flag now supported.

With the encoder initialized, you can now start adding frames. The CreateNewFrame method creates a new frame that you can then configure and write pixels to:

CComPtr<IWICBitmapFrameEncode> frame;
CComPtr<IPropertyBag2> properties;

HR(encoder->CreateNewFrame(
  &frame,
  &properties));

CreateNewFrame returns both an IWICBitmapFrameEncode interface representing the new frame as well as an IPropertyBag2 interface. The latter is optional and can be used to specify any encoder-specific properties such as the image quality for JPEG or the compression algorithm for TIFF. For example, here's how you can set the image quality for a JPEG image:

PROPBAG2 name = { 0 };
name.dwType = PROPBAG2_TYPE_DATA;
name.vt = VT_R4;
name.pstrName = L"ImageQuality";

CComVariant value(0.75F);

HR(properties->Write(
  1, // property count
  &name,
  &value));

The value for image quality must fall between 0.0 for the lowest possible quality and 1.0 for highest possible quality.

With the encoder properties set, you need to call the Initialize method before configuring and writing to the frame:

HR(frame->Initialize(properties));

The next step is to set the dimensions and pixel format for the new frame before you can write pixels to it:

HR(frame->SetSize(width, height));
GUID pixelFormat = GUID_WICPixelFormat24bppBGR;
HR(frame->SetPixelFormat(&pixelFormat));
ASSERT(GUID_WICPixelFormat24bppBGR == pixelFormat);

The SetPixelFormat parameter is an [in, out] parameter. On input it specifies the desired pixel format. On output it contains the closest supported pixel format. This usually isn't a problem unless the format is set at run time, perhaps based on the pixel format of another bitmap.

Writing pixels to the frame is achieved using the WritePixels method, as shown here:

HRESULT WritePixels(
  UINT lineCount,
  UINT stride,
  UINT bufferSize,
  BYTE* buffer);

The lineCount parameter specifies how many lines of pixels are to be written. This implies that you can call WritePixels a number of times to write the complete frame. The stride parameter indicates how the pixels in the buffer are packed into rows. I described how you can calculate the stride in the previous section.

After you've called WritePixels one or more times to write the complete frame, you need to tell the encoder that the frame is ready by calling the frame's Commit method. And once you've committed all the frames that make up the image, you need to tell the encoder that the image is ready to be saved by calling the encoder's Commit method.

So far I've covered the basics of encoding and decoding images. Before I move on, I want to drive it home with a simple example. Figure 2 shows a CopyIconToTiff function that demonstrates how to read the individual bitmaps that make up an icon and copy them to a multiframe TIFF image.

Figure 2 CopyIconToTiff (Close Figure 2)
 HRESULT CopyIconToTiff(
   IStream* sourceStream,
   IStream* targetStream) {

  // Prepare the ICO decoder

  CComPtr<IWICBitmapDecoder> decoder;
  HR(decoder.CoCreateInstance(CLSID_WICIcoDecoder));

  HR(decoder->Initialize(
    sourceStream,
    WICDecodeMetadataCacheOnDemand));

  // Prepare the TIFF encoder

  CComPtr<IWICBitmapEncoder> encoder;
  HR(encoder.CoCreateInstance(CLSID_WICTiffEncoder));

  HR(encoder->Initialize(
    targetStream,
    WICBitmapEncoderNoCache));

  UINT frameCount = 0;
  HR(decoder->GetFrameCount(&frameCount));

  for (UINT index = 0; index < frameCount; ++index) {
    // Get the source frame info

    CComPtr<IWICBitmapFrameDecode> sourceFrame;

    HR(decoder->GetFrame(index, &sourceFrame));

    UINT width = 0;
    UINT height = 0;
    HR(sourceFrame->GetSize(&width, &height));

    GUID pixelFormat = { 0 };
    HR(sourceFrame->GetPixelFormat(&pixelFormat));

    // Prepare the target frame

    CComPtr<IWICBitmapFrameEncode> targetFrame;

    HR(encoder->CreateNewFrame(
      &targetFrame,
      0)); // no properties

    HR(targetFrame->Initialize(0)); // no properties

    HR(targetFrame->SetSize(width, height));
    HR(targetFrame->SetPixelFormat(&pixelFormat));

    // Copy the pixels and commit frame

    HR(targetFrame->WriteSource(sourceFrame, 0));
    HR(targetFrame->Commit());
  }

  // Commit image to stream

  HR(encoder->Commit());

  return S OK;
}

In this example, I've simplified things further by taking advantage of an alternative to the WritePixels method. Instead of first copying the pixels from the source frame and then writing them to the target frame, I'm using the WriteSource method that reads the pixels directly from a given IWICBitmapSource interface. Since the IWICBitmapFrameDecode interface derives from IWICBitmapSource, this provides an elegant solution.

The WIC Imaging Factory

WIC provides an imaging factory for creating various WIC-related objects. It is exposed through the IWICImagingFactory interface and can be created as follows:

CComPtr<IWICImagingFactory> factory;
HR(factory.CoCreateInstance(CLSID_WICImagingFactory));

I've already shown how you can create a decoder for a specific image format given a CLSID identifying an implementation. Of course, as you probably can tell, it would be much more useful if you didn't have to specify a particular implementation or even hardcode the format of the image.

Fortunately, WIC provides the solution. Before creating a decoder, WIC can examine a given stream for patterns that might identify the image format. Once a best match is found, it will create the appropriate decoder and initialize it with the same stream. This functionality is provided by the CreateDecoderFromStream method:

CComPtr<IWICBitmapDecoder> decoder;

HR(factory->CreateDecoderFromStream(
  stream,
  0, // vendor
  WICDecodeMetadataCacheOnDemand,
  &decoder));

The second parameter identifies the vendor of the decoder. It is optional but may be useful if you prefer a particular vendor's codec. Keep in mind that it is only a hint, and if the particular vendor does not have a suitable decoder installed, then a decoder will still be chosen regardless of vendor.

IWICImagingFactory also provides the CreateDecoderFromFilename and CreateDecoderFromFileHandle methods that provide the same functionality given a path to a file and a file handle, respectively. Alternatively, you can create a decoder without specifying a CLSID or a stream but instead indicating the image format. The CreateDecoder method does just that:

CComPtr<IWICBitmapDecoder> decoder;

HR(factory->CreateDecoder(
  GUID_ContainerFormatIco,
  0, // vendor
  &decoder));

HR(decoder->Initialize(
  stream,
  WICDecodeMetadataCacheOnDemand));

Similarly, the CreateEncoder method allows you to create an encoder for a particular image format without regard for its implementation, like so:

CComPtr<IWICBitmapEncoder> encoder;

HR(factory->CreateEncoder(
  GUID_ContainerFormatBmp,
  0, // vendor
  &encoder));

HR(encoder->Initialize(
  stream,
  WICBitmapEncoderNoCache));

Figure 3 lists the GUIDs identifying the implementation-independent image formats, also known as container formats.

Figure 3 Container Format GUIDs (Close Figure 3)

Format | GUID
BMP | GUID_ContainerFormatBmp
PNG | GUID_ContainerFormatPng
ICO | GUID_ContainerFormatIco
JPEG | GUID_ContainerFormatJpeg
GIF | GUID_ContainerFormatGif
TIFF | GUID_ContainerFormatTiff
HDP | GUID_ContainerFormatWmp

Working with Streams

You are free to provide any valid IStream implementation for use with WIC. You can, for example, use the CreateStreamOnHGlobal or SHCreateStreamOnFile functions that I described in my XmlLite article, or even write your own implementation. WIC also provides a flexible IStream implementation that may come in handy.

Using the imaging factory I introduced in the previous section, you can create an uninitialized stream object as follows:

CComPtr<IWICStream> stream;
HR(factory->CreateStream(&stream));

The IWICStream interface inherits from IStream and provides a few methods for associating the stream with different backing storage. For example, you can use InitializeFromFilename to create a stream backed by a particular file:

HR(stream->InitializeFromFilename(
  L"file path",
  GENERIC_READ));

You can also use InitializeFromIStreamRegion to create a stream as a subset of another stream or use InitializeFromMemory to create a stream over a block of memory.

WIC via WPF

As I've mentioned, WIC provides the framework on which the WPF imaging functionality is based. The various imaging classes are defined in the System.Windows.Media.Imaging namespace. To show you just how easy it is to use from managed code, Figure 4 shows the CopyIconToTiff function from Figure 2 rewritten in C# using the WPF wrapper classes.

Figure 4 CopyIconToTiff Rewritten in C# and WPF (Close Figure 4)
static void CopyIconToTiff(Stream sourceStream,
                           Stream targetStream) {

  IconBitmapDecoder decoder = new IconBitmapDecoder(
    sourceStream,
    BitmapCreateOptions.None,
    BitmapCacheOption.OnDemand);

  TiffBitmapEncoder encoder = new TiffBitmapEncoder();

  foreach (BitmapFrame frame in decoder.Frames) {
    encoder.Frames.Add(frame);
  }

  encoder.Save(targetStream);
}

The BitmapCacheOption.OnDemand value corresponds to the WICDecodeMetadataCacheOnDemand decoder option used in native code. And in a similar way, the alternative BitmapCacheOption.OnLoad value corresponds to the WICDecodeMetadataCacheOnLoad decoder option.

I've already described how these options influence when the decoder reads the image information into memory. There is, however, an additional side effect that you should be aware of when dealing with these options in managed code. Consider what happens when you specify BitmapCacheOption.OnDemand. The decoder will hold onto a reference to the underlying stream and may read from it at some point after the bitmap decoder object has been created. This assumes that the stream is still available. You need to be careful that your application doesn't close the stream prematurely. It's a matter of managing the lifetime of the stream so that it is not closed before the decoder is finished with it.

This doesn't affect native code because the IStream interface is a standard COM interface whose lifetime is controlled by reference counting. Your application may have released all of its references to it, but the decoder will hold onto one as long as is necessary. What's more, the stream is only closed after all interface pointers have been released.

What's Next?

WIC provides an incredibly powerful and flexible framework on which to base your imaging needs. With a generous set of codecs and a simple API, you can start taking advantage of many of its features in no time.

In my next column I'm going to explore some of the more advanced features offered by WIC. I'll show you how you can develop your own codecs and illustrate the registration and discovery process, including the pattern-matching facilities, in detail. While I'm at it, I'll also correct a limitation in one of the built-in codecs.

Send your questions and comments for Kenny to mmwincpp@microsoft.com.

Kenny Kerris a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. Reach Kenny at weblogs.asp.net/kennykerr.


Going Places: Provisioning Mobile Devices

Going Places Provisioning Mobile Devices and Mike Calligaro
SUMMARY
Learn how you can set up every mobile device in your company with a few lines of code and some XML--thanks to the provisioning APIs in the Windows Mobile SDK.
Contents
Why Provision?
Configuration Service Providers
Getting Your XML to the Device
Who Wants to Type XML?
All Provisioned and Ready to Go

W
elcome to Going Places—a new column devoted to mobile device development. The main focus will be on Windows Mobile® phones and Tablet PCs, although any topic related to mobility is fair game.

My name is Mike Calligaro, and I've been a developer at Microsoft for 14 years. Aside from a two week stint in the Windows® desktop group, I've spent my entire career in what is now the Windows Mobile division. I've worked on a wide range of products, including an early version of interactive TV, the Sega Dreamcast, and the Windows CE operating system. I've spent the past eight years working on various parts of Windows Mobile.

Today I'm going to talk about programmatically configuring Windows Mobile devices (or "provisioning" to people who like to use fancy words). If you're an IT manager or support technician, and are manually configuring Windows Mobile devices for your users, stop. This article is for you. If you're a mobile app developer and wish you had an easy way to add your Web page to the browser's favorites, this will help you too.

Putting on a polyester suit for a moment, what if I told you that, with a few lines of code and some XML, you can set up a VPN, set the device's date and time, configure an e-mail account, add a file to the device, set a ring tone, change a registry key, and much, much more? Now how much would you pay? A million dollars? Two million dollars? Well, in this special MSDN® Magazine offer, we're making these capabilities available to you for the low, low price of free. That's right, free. So call now while I throw this digital fish into a blender.

Hopefully I'm a better developer than I am a salesman. In all seriousness, the provisioning APIs are part of the standard Windows Mobile SDK, which is a free download available from the Windows Mobile Developer Center at msdn.microsoft.com/windowsmobile.

Why Provision?

Although the name Windows Mobile came later, Microsoft released its first platform for cellular phones in 2002. Getting into the phone space meant implementing a lot of things that mobile operators wanted but weren't accustomed to doing, such as the ability to configure devices remotely. Now, there wasn't a whole lot to configure on the typical 2002-era phone, but we needed to support that basic level of configuration to be competitive. Once we had the infrastructure in place, we found it convenient to do a lot of remote configuration.

For instance, due to the requirements of my job, I end up using a number of different phones and many different OS versions. During the more unstable times in the development lifecycle, I've occasionally had to set up various phones multiple times a day. And even when things are running smoothly, it's not uncommon for me to configure at least one phone a week. So rather than manually type in my Exchange Server address and set my ring tone, I put together an XML blob that does most of my setup. I keep the XML file and a little program to process it on a storage card. That way I'm always ready when someone hands me a new phone to try.

I'm sure that being able to set Penny Arcade as a browser favorite was never a mobile operator requirement, but I'm glad that we used the "I have a hammer, so everything looks like a nail" mentality in this case.

To work this configuration magic, all you need is a single API (DMProcessConfigXML) and some XML. For example, say you want to set Live Search as a browser favorite. Start with the following XML:

// WAP provisioning XML to add a new browser favorite.
LPCWSTR g_wszFavoriteXml =
    L"<wap-provisioningdoc> "
    L"   <characteristic type=\"BrowserFavorite\"> "
    L"      <characteristic type=\"Live Search\"> "
    L"         <parm name=\"URL\" value=\"http://m.live.com/\"/> "
    L"      </characteristic> "
    L"   </characteristic>"
    L"</wap-provisioningdoc>";

Given that, here's all the code you need in order to set the favorite using the XML:

HRESULT AddFavorite() {
   HRESULT hr         = E_FAIL;
   LPWSTR wszOutput   = NULL;

   // Process the XML.
   hr = DMProcessConfigXML(g_wszFavoriteXml,
     CFGFLAG_PROCESS, &wszOutput);

   // The caller must delete the XML returned from DMProcessConfigXML.
   delete [] wszOutput;
   return hr;
 }

I'm simplifying things a bit here for the sake of space. In your own code you should do security and error checking. And you're more likely to be reading XML from a file rather than having it compiled into your code. But, in a nutshell, this is what you need to do.

There's also a managed wrapper for DMProcessConfigXML. In C#, you can use ConfigurationManager.ProcessConfiguration instead. Here's the same code in C#:

XmlDocument configDoc = new XmlDocument();
configDoc.LoadXml(
    "<wap-provisioningdoc>"+
    "<characteristic type=\"BrowserFavorite\">"+
    "<characteristic type=\"Live Search\">"+
    "<parm name=\"URL\" value=\"http://m.live.com/\"/>"+
    "</characteristic>"+
    "</characteristic>"+
    "</wap-provisioningdoc>"
    );
ConfigurationManager.ProcessConfiguration(configDoc, false);

Configuration Service Providers

DMProcessConfigXML sends the XML it receives to various Configuration Service Providers (CSPs). These providers read the XML and act on it. If there's some sort of configuration you want to change that isn't supported, the likelihood is that we haven't written a CSP for it yet. We're interested in hearing feedback on new CSPs you'd find useful for us to add.

The definitive list of CSPs is available from msdn2.microsoft.com/bb737536.aspx. Also, there are a number of XML examples available from msdn2.microsoft.com/bb737572.aspx.

Unfortunately, with regard to the availability of CSPs, there is still a difference between Windows Mobile Professional (Pocket PC) phones and Windows Mobile Standard (Smartphone) phones. You will find that some of the CSPs are available on Standard but not Professional. (No, the irony isn't lost on us.) We're working to unify the platforms in the future. Also, OEMs and mobile operators have the ability to selectively disable CSPs to lock down functionality they don't want changed. To be safe, you should always test any configuration you plan to deploy on the phones you're thinking about supporting.

How about a few examples? First, here's a blob of XML that sets the time zone, turns on the alarm, and sets it:

<wap-provisioningdoc>
   <characteristic type="clock">
      <parm name="TimeZone" value="4"/>
      <parm name="AlarmOn" value="1"/>
      <parm name="AlarmTime" value="06:30:00Z"/>
   </characteristic>
</wap-provisioningdoc>

The characteristic type tells you that it uses the Clock CSP. Each CSP has an MSDN page that describes the parameters you can change as well as the possible values you can change each parameter to. For instance, the documentation for the Clock CSP is available at msdn2.microsoft.com/bb737261.aspx, and it describes the possible values for TimeZone, AlarmOn, and AlarmTime as well as two parameters not in this example (date and time).

Be sure to look at the docs for the correct values to assign to the parameters. I'm sure they make sense to someone, but often not to mere mortals like me. For instance, the TimeZone value for GMT -8 is 4, but the value for GMT -5 is 35. Go figure.

Figure 1 shows an example of some of the things you can do with the Sync CSP. (Note that the documentation for the Sync provider is at msdn2.microsoft.com/bb737700.aspx.) As you see, you can choose what data to sync, where to sync it from, how often to sync, and so on. There are something like 50 parameters related to syncing that you can configure with this provider.

Figure 1 Using the Sync CSP (Close Figure 1)
<wap-provisioningdoc>
   <characteristic type="Sync">
      <characteristic type="Connection">
         <parm name="Server" value="testserver"/>
         <parm name="User" value="testuser"/>
         <parm name="Domain" value="thephone-company"/>
         <parm name="AllowSSLOption" value="1"/>
      </characteristic>
      <characteristic type="Settings">
         <parm name="PeakStartTime" value="0800"/>
         <parm name="PeakEndTime" value="1800"/>
         <parm name="PeakFrequency" value="0"/>
         <parm name="BodyTruncation" value="5120"/>
      </characteristic>
      <characteristic type="Calendar">
         <parm name="Enabled" value="1"/>
      </characteristic>
      <characteristic type="Contacts">
         <parm name="Enabled" value="1"/>
      </characteristic>
      <characteristic type="Mail">
         <parm name="Enabled" value="1"/>
      </characteristic>
   </characteristic>
</wap-provisioningdoc>

While these examples only scratch the surface, hopefully they give you an idea of how the CSPs work.

Getting Your XML to the Device

So you've put together some great XML, and you've written a little app that opens an XML file and sends the data to DMProcessConfigXML. But you won't be doing any provisioning until you can get the XML and the app onto the device.

You have a few options here. The simplest is to put the file and the application on a storage card, pop the card into a device, and then run the application using File Explorer. This is what I do to configure my devices.

But what if you're an IT manager with hundreds of devices to configure? The extra steps of launching File Explorer and navigating to the app would take too much time. In that case, you could make the configuration app run automatically when you insert the storage card.

Create a folder called 2577 (don't ask) at the root of the storage card and put your configuration app into that folder. Name the app autorun.exe. If the app automatically opens a standard XML file name (for example, config.xml), then all you'll need to do to configure the device is insert the card. The AutoRun trick also works when the device boots, so it will work even if you have a phone that requires you to remove the battery to insert the storage card.

Of course, there are more options. If, after the initial setup, you plan to do further configuration with your devices, you could put the configuration app on the device and associate an extension with it. Then you could e-mail configuration files with the associated extension to your users. They would only need to open the attached configuration file to configure their devices. Alternately, your users could browse to a Web site that downloads the configuration file for them.

It's also possible to use PushRouter to send Wireless Application Protocol (WAP) configuration files over SMS to devices. PushRouter is beyond the scope of this article, but this is how mobile operators send configuration files to phones on their networks.

If you're an independent software vendor (ISV), you can use the CSPs from within your setup program to do any necessary device configuration, from adding files to changing registry keys to adding browser favorites to changing the home screen. The simplicity of the configuration format also makes it easy to have a matched XML file for your uninstaller that undoes what your installer did.

If you have a connection to the device through ActiveSync® or the Windows Mobile Device Center, you can configure the device without putting any files on it. The desktop tool rapiconfig.exe is located in the tools directory of the Windows Mobile SDK. This tool will read an XML file and use RAPI to configure the device currently connected to ActiveSync. After you've docked the device, go into the tools directory in the SDK and (assuming your XML file is called config.xml) type this:

rapiconfig config.xml

For more information on RapiConfig, see msdn2.microsoft.com/bb737541.aspx.

Finally, you can create either a CAB or a CAB Provisioning Format (CPF) file. Both CABs and CPFs can be copied to a device and executed directly without the need for a program. The desktop tool makecab.exe is used to create both file types and is also available in the Windows Mobile SDK.

The main difference between a CAB and a CPF file is that the CPF won't open any dialog boxes when it is executed (it runs silently). This turns out to be the CPF's main drawback as well. Depending on the security settings on the device and whether or not the file is signed, CABs and CPFs might need to show a dialog box asking the user if it's OK to provision the device. Because the CPF is required to run silently, it can't show this dialog box. So it fails. In the end, if you're planning to sign the configuration file, use a CPF. If not, use a CAB.

Making either is very simple. You just pass a normal XML file to makecab and give it the name of the output file you want. The one trick is that the XML file must be named _setup.xml. If you call it something else, makecab will succeed, the CAB or CPF file will be generated fine, and then it will fail with a cryptic error when you try to use it. Beware, the documentation doesn't make this very clear.

The syntax for making a CAB file is:

makecab _setup.xml config.cab

Unlike the XML file, there are no such restrictions on the name of the generated CAB file. For instance, instead of config.cab, you can choose whatever name you want for the generated CAB file. The syntax for making a CPF file is, likewise:

makecab _setup.xml config.cpf

Again, replace config.cpf with whatever you want the CPF file to be called. Yes, that's correct, the only difference in the two is whether you name the output file .cab or .cpf. And, again, in both cases the XML file must be called _setup.xml.

Go to msdn2.microsoft.com/bb737689.aspx for more information on CPF files and to msdn2.microsoft.com/bb416436.aspx for more information on signing CPF files.

Who Wants to Type XML?

Hand-typing XML is fine for savvy IT professionals, but what if you want to allow your users to choose their own configuration from a set of options? Wouldn't it be cool to have a Web site with a bunch of checkboxes and such? The users would browse to the site, choose what they want, and it would generate the XML file.

For this scenario, the Microsoft® .NET Framework provides the XmlDocument class, making it easy to programmatically generate XML. For example, remember the clock XML above? The code in Figure 2 creates it on the fly.

Figure 2 Creating the Clock Configuration (Close Figure 2)
void CreateXML()
{
    XmlDocument xmlDoc = new XmlDocument();
    XmlElement root = CreateRoot(xmlDoc);

    XmlElement type = CreateCharacteristic(xmlDoc, "clock", root);
    CreateParm(xmlDoc, "TimeZone", "4", type);
    CreateParm(xmlDoc, "AlarmOn", "1", type);
    CreateParm(xmlDoc, "AlarmTime", "06:30:00Z", type);

    xmlDoc.Save("config.xml");
}

XmlElement CreateRoot(XmlDocument xmlDoc)
{
    XmlElement root = xmlDoc.CreateElement("wap-provisioningdoc");
    xmlDoc.AppendChild(root);
    return root;
}

XmlElement CreateCharacteristic(XmlDocument xmlDoc, string type, XmlElement parent)
{
    XmlElement element = xmlDoc.CreateElement("characteristic");
    element.SetAttribute("type", type);
    parent.AppendChild(element);
    return element;
}

XmlElement CreateParm(XmlDocument xmlDoc, string name, string value, XmlElement parent)
{
    XmlElement element = xmlDoc.CreateElement("parm");
    element.SetAttribute("name", name);
    element.SetAttribute("value", value);
    parent.AppendChild(element);
    return element;
}

CreateRoot, CreateCharacteristic, and CreateParm are helper functions that you can lift wholesale. Then just make your own CreateXML that calls into them. This example hardcodes the parameter information, but in your Web page the values would come from the user's UI choices. After generating the XML, the Web site downloads it to the device for the user to execute.

Conceivably, you could also write a configuration app that runs on the device. After the user selects the appropriate configuration, code like that in Figure 2 would generate the XML and send it directly to ConfigurationManager.ProcessConfiguration. So you wouldn't need to go through the steps of sending the XML to a file and then reading it back in.

All Provisioned and Ready to Go

If you've ever had to enter an Exchange Server address using the numeric keypad on your phone, you owe it to yourself to move to the comfort of your desktop keyboard and compose some XML. No one should have to go through that pain. I hope you've found this article to be ... liberating. Happy provisioning.

Send your questions and comments to goplaces@microsoft.com.


{ End Bracket }: The Magic of Software

{ End Bracket }
The Magic of Software
Julia Lerman

Software developers often enjoy laughing at the wildly unrealistic portrayals of high-tech computing in movies and TV today, such as the transparent computer screens in the Steven Spielberg movie Minority Report or the flashy fingerprint analysis software programs used on CSI. But perhaps the feats performed on these shows are closer to actualization than we realize.

In 2004, Carter Maslan, from the Platform Evangelism team at Microsoft, wrote some of the first conceptual applications for what was then code-named "Longhorn" (now Windows Vista®), using many of the tools that are now part of the Microsoft® .NET Framework 3.0. After watching a video on the first sample, which was a concept application for the real estate market, I remember thinking that the fantastic imaginings of the TV crime shows were actually coming to life—that life was, in fact, imitating art.

Recently, I was invited to attend an all-day event at Microsoft called Mix n Mash 08, which is a precursor to the Microsoft MIX08 conference to be held in Las Vegas in March. MIX is all about new Web technologies offered by Microsoft. At the Mix n Mash event, we were able to view some already-public projects that Microsoft is working on and discuss their current state and future plans.

One of the highlights was the Microsoft Surface computer (microsoft.com/surface). Surface computing has the visual magic of Windows Presentation Foundation (WPF) at its core, combined with a multi-touch surface in a table-sized format.

The applications that have been built for it so far range from visually exciting and entertaining to serious business applications. Surface is truly astonishing. The first demonstration included an image of pebbles under water. Running your hand over the tabletop makes the water image move just as real water would—but your hand doesn't get wet!

Another application we enjoyed was finger painting—a wonderful escape for adults. Then we watched as a Zune was placed on Surface, which immediately revealed the device's stored media and allowed us to interact with the media right from the table top. Yet another application transformed Surface into a restaurant table where we were able to order and pay for food and drinks. Surface is so advanced that it looks like a next-generation concept of what computing can do for you, yet it is here today.

At the end of the day, we each had an opportunity to ask a question of Bill Gates, who is leaving the helm of Microsoft in July 2008 to devote his time to the Bill and Melinda Gates Foundation. I asked him about his vision for computing and what he hoped to see carried out after his departure. He explained to us that much of the technical motivation at Microsoft is based on desire to realize the ultimate goal of computer science. That goal, he said, is that someday computers will be "hyper-intelligent," and "as soon as you [achieve] that, then you just ask them what you should do, and they tell you." While this will take decades yet, Microsoft is currently targeting what Gates described as computers that function somewhat like a personal assistant.

Gates also spent some time explaining the company's quest process, in which a group of people is tasked with imagining what computers could do for particular communities (for example, developers, information workers, or consumers) within the next 10 years. The ideas generated during these quests are sent to Microsoft Research for further exploration. Periodic quest summits track the progress of these projects. For example, Microsoft Research is currently working on audio watermarking, color barcodes, and memory retention research, as well as several hundred other initiatives based on truly innovative research in technology.

Speaking of fantastic, one of my favorite absurdities on TV today is when the characters of the crime shows seem to invent pixels that don't exist. It's hilarious to watch them zoom in on a photograph and detect critical, case-solving information in the reflection of someone's sunglasses. However, since I have returned from Mix n Mash 08, I am laughing less and finding myself more and more intrigued by what these television shows seem to be proposing. Certainly, criminal investigators are already benefitting from the fingerprint and facial recognition software, but both of these technologies have a long way to go before they are as effective as they appear on the fictional crime shows. Perhaps the Pattern Recognition and Machine Learning team at Microsoft Research will work on this. As Gates said to us, "What is the magic of software going to do?"

Julia Lerman, a .NET consultant, has been building software for more than 20 years. She is well-known in the .NET community as conference speaker, author, Microsoft .NET MVP, and leader of the Vermont .NET User Group. Her upcoming book is titled Programming Entity Framework. Julia blogs at thedatafarm.com/blog.