December 2006 December 2006
Mobilize: Explore The New Features In Windows Embedded CE 6.0
Paul Yao presents an overview of Windows Embedded CE 6.0. Paul Yao
Graphics To Go: Make A Mobile Imaging App With The .NET Compact Framework 2.0
This article focuses on developing for Pocket PCs, a skill which can then be transferred to Smartphone application development. Rob Pierry
Windows Workflow: Build Custom Activities To Extend The Reach Of Your Workflows
This article covers the core components required for building custom activities in Windows Workflow Foundation. Matt Milner
Extend ASP.NET: Simplify Data Binding In ASP.NET 2.0 With Our Custom Control
The wwDataBinder control addresses simple control data binding--binding simple form controls like textboxes, checkboxes, and the selected values of list controls to individual data or object values. Rick Strahl
BizTalk Adapters: Integrate E-Mail Processing Into Your Business Solutions
Use BizTalk Server to monitor e-mail for new messages, trigger confirmations, and persist e-mail and attachments to a custom database. Alex Starykh
Locale Hero: Enable Global Apps With Locale Builder And Windows Vista
Everything you need to know about locales in Windows Vista. Kieran Snyder and Shawn Steele
Columns
Editor's Note: A Look Into the Future: Josh Trupin takes a look at mobility, the theme of this month’s issue.Joshua Trupin | Toolbox: Issue and Defect Tracking, Automate IT Tasks, and More: In this issue, issue and defect tracking, automating IT tasks, and more.Scott Mitchell | Advanced Basics: TableLayoutPanels: This month Ken Getz writes a demo-creation system for Windows-based applications, which he calls a switchboard. Ken Getz
CLR Inside Out: Inside Diagnostic Tools for .NET: In this installment of CLR Inside Out, a look inside the CLR Profiling API and tools that use it.Jonathan Keljo | Data Points: RSS Feeds on a Smartphone: John Papa builds a Windows Mobile 5.0 application that reads RSS feeds and loads them into an ADO.NET DataSet.John Papa | Cutting Edge: The Client Side of ASP.NET Pages: This month Dino Esposito dissects the client-side source code generated by ASP.NET pages.Dino Esposito
Test Run: String Permutations: The ability to programmatically create and use string permutations is essential in software testing, as James McCaffrey explains. Dr. James McCaffrey | Service Station: Web Service Software Factory: Web Service Software Factory is designed to help you build Web service solutions that follow known architecture and design patterns, as Aaron Skonnard explains here.Aaron Skonnard | .NET Matters: Deserialization Progress, and More: Build a standard progress bar that indicates how much deserialization has completed and how much still remains. Plus, System.Collection.Generics performance.Stephen Toub
Netting C++: The Design Space of the Common Type System: Stanley Lippman moves Text Query Language to the Microsoft .NET Framework using the C++/CLI language extensions in Visual C++ 2005. Stanley B. Lippman | { End Bracket }: And Along Came 10…: IntroducingOn10.net (http://on10.net), the Channel 9 answer for the technology enthusiast who isn’t necessarily a programmer. Duncan Mackenzie


Mobilize
Explore The New Features In Windows Embedded CE 6.0
Paul Yao

This article is based on a prerelease version of Windows Embedded CE 6.0. All information herein is subject to change.
This article discusses:
  • Windows Embedded CE 6.0
  • Embedded development
  • Kernel innovations
  • Shared source implementation
 This article uses the following technologies:
Windows CE
Contents
Overview of Windows Embedded CE 6.0
API Deployment Scheme
Kernel-Mode Operation
Memory Architecture
Device Emulator
Cell Technologies for CE 6.0
Location-Based Programming
Security Enhancements
Development Tools
Cool Tools
Registry File Editor
Runtime Image Viewer
Shared Source and Windows CE Development
Conclusion
The first version of Windows CE shipped in 1996. Since then, many new types of portable and embedded products have emerged-enabled by myriad advances in semiconductor components. In 1994, worldwide semiconductor sales hit the $100 billion mark, according to the Semiconductor Industry Association (SIA), and that number is expected to hit $250 billion for 2006. Along the way, the Windows® CE team has kept its fingers on the pulse of developers working in the embedded space, pushing the boundaries in each generation.

Overview of Windows Embedded CE 6.0
For each of its previous five versions, Windows CE has served as the foundation for a wide range of embedded devices. The newest version, Windows Embedded CE 6.0, continues the push forward, providing a whole new set of features-some of which you need to understand, and others you may just want to be aware of. (See the sidebar "Inside Embedded Innovation at Microsoft".)
Inside Embedded Innovation at Microsoft
The process of conceiving, planning, developing, and delivering a new version of a successful product is a huge undertaking—especially when that product is an operating system, albeit a small one. In the case of Windows CE, it involves a development team with upwards of 700 individuals. When planning new features, Microsoft solicits customer feedback in several ways. The best known feedback mechanism is the beta software program, whereby prerelease software is shipped to customers. But beta programs are not the only way to get input. The design and planning phases are bolstered with customer visits and design reviews to inject customer insights into the product development process.
At the start of the product planning process, the Windows CE planning team hits the road to visit customers around the world to learn about their current and anticipated product development needs. These visits occur shortly after the previous product version ships, because innovation is an ongoing process. On the basis of these visits, a draft development plan is created and reviewed internally.
With a draft development plan in place, the next step involves validating the plan at a design review where hundreds of people from outside Microsoft gather to provide input. I participated in a design review in June of 2005 for the CE 6.0 plan. This week-long event started with participants signing non-disclosure agreements (NDA) to facilitate open and honest discussion on both sides. The actual mechanics of a design review are pretty interesting. Major feature areas are first described by members of a development team. Each presentation ends with a question and answer period, and each participant uses a wireless voting box that was provided at registration. Immediately after each vote, the results are displayed for everyone to view and discuss.
By the end of the week, each participant may have provided hundreds of opinions on how the development team should set priorities. The value of the exercise can best be measured by the new insights that development teams have gotten when they put their tough questions in front of customers who are relying on the presence of some feature in a future product version.

The specific features of interest to you will depend on the type of gadgets you are working with. If you're a designer of headless industrial controllers, it's likely you have a different set of needs from a software developer creating the next generation of sewing machine applications. Both are likely to define what's interesting in ways that might not make sense to a developer of mobile phone applications. You get the point.
While Windows CE is the foundation for many types of gadgets, you often get the impression that it's split into two camps: Windows Mobile®-powered devices and custom embedded devices (in other words, everything else). But can one platform support both standard and custom devices? Yes, Windows Mobile and custom embedded devices are linked by a vast technology portfolio that includes a small but capable multithreaded kernel, a rich family of programming APIs-Win32, MFC, and the .NET Compact Framework-and a powerful, extensible set of development tools.
Windows Embedded CE 6.0 (CE 6.0) introduces a new kernel that overcomes limitations of earlier kernels, while squeezing more performance out of those tiny CPUs. In this article I'll describe the new CE 6.0 operating system capabilities beginning with those that pertain to the new kernel: a new API deployment scheme, new use of kernel mode, and the memory architecture. I'll then cover other new features of CE 6.0.
Finally, I'll go over the new tools for configuring, building, and testing the myriad Windows CE configurations. The discussion of tools is aimed squarely at developers of embedded applications who create custom Windows CE configurations. But if you're a Windows Mobile developer and are tempted to skip this second half, I urge you to keep reading, paying special attention to the discussion of Windows CE shared source. It's a mandatory resource for anyone who wants the inside scoop on Windows CE that only the source code can provide.

API Deployment Scheme
One of the strengths of Windows CE is that its development benefitted from the features and optimizations of the desktop versions of Windows. Starting with the first version of Windows CE and including every version up to Windows CE 5.0, the Win32® APIs were deployed using client/server architecture similar to that found in early versions of Windows NT®.
In Windows NT 3.5, for example, all GUI functions reside in a single process: csrss.exe, the client-server runtime subsystem. When functions like CreateWindow or TextOut are called, the CSRSS process is triggered through an interprocess communication mechanism called local-procedure calls (LPCs). Owing to performance penalties of so many context switches between the heavyweight Windows NT processes, this architecture was formally abandoned in Windows NT 4.0 (released in 1996). At the time, all the formerly user-mode API libraries were transferred into kernel mode.
A similar change is taking place with the release of Windows Embedded CE 6.0. For all previous versions, Windows CE also used a client/server mechanism to connect a caller to the target Win32 function. There are similarities, but it is important to note the differences between the Windows CE implementation and the one on Windows NT. One difference is that Windows CE used a much more lightweight process than Windows NT, and so the performance cost was not as high. Unlike Windows NT, Windows CE was designed to be configurable. And the use of processes to deploy system APIs allowed the demands for ROM and RAM to be gated by how much of the overall operating system was needed for a specific device. For example, a device could be configured to run with just one API process, the system kernel (nk.exe). Other devices that need GUI support would also need to run the GUI (gwes.exe) process.
Although the new model does provide some performance improvements, the real motivating factor for making the change is to remove limitations that could bottleneck development on the next generation of devices. Those limitations, which are well known to Windows CE programmers, include the limit on total number of processes (32), and the small virtual address space (32MB) of previous generations of Windows CE kernels.
Many things have changed in the decade since the first kernel was deployed. Devices today can have more capable hardware: faster CPUs, more storage, a color LCD display screen, and so on. Today's designs don't need the tiny memory footprint of yesterday's models, and so a redesign was in order. With CE 6.0, system APIs are moved out of their own user-mode processes and are placed into kernel-mode DLLs instead. Figure 1 summarizes the changes between Windows CE 5.0 and CE 6.0.
Figure 1 Changes to Major Modules in Windows Embedded CE 6.0

Windows CE 5.0 Process | Windows Embedded CE 6.0 DLL | Description
nk.exe (OAL + kernel) | nk.exe (OAL) kernel.dll (kernel) | OEM code is getting separated from CE kernel code starting with CE 6.0
filesys.exe | filesys.dll | Registry, file system, and property databases
device.exe | device.dll | Manages kernel-mode device drivers
device.exe | udevice.exe | New to CE 6.0, a separate process to manage user-mode device drivers
gwes.exe | gwes.dll | Graphical and windowing event subsystem
services.exe | servicesd.exe | Host process for system services
services.exe | services.exe | Command-line interface to configure services
While this reorganization of the operating system was underway, another important change was made: OEM code was separated from operating system code. Previously, a hardware designer created a low-level set of routines called the OEM Adaptation Layer (OAL), and that component was statically linked to the operating system kernel. The OAL and the kernel appeared as a single executable, nk.exe. In CE 6.0, these two components are now built into two separate modules: the kernel (kernel.dll) and the OEM Adaptation Layer (nk.exe).

Kernel-Mode Operation
To run Windows CE, a CPU must support two privilege levels. The higher privilege level is referred to as kernel mode; the lower privilege level is known as user mode. Any memory that is allocated for either code or data is assigned to one of these two modes. Putting code in user mode helps the overall environment run more robustly and more securely. However, these benefits are not free, because mixing user-mode and kernel-mode code is generally slower than running the system entirely in kernel mode.
In previous versions, Windows CE could be configured for all kernel-mode operation, or for mixed-mode operation using both kernel mode and user mode. With CE 6.0, only the mixed operating mode is supported with all applications being loaded into user-mode memory, and all OS components loaded into kernel-mode memory. As shown in Figure 2, some components have both a user-mode and a kernel-mode incarnation. This deployment helps minimize the cost of calling across the privilege boundary, at the expense of a larger operating system image.
Figure 2 User-Mode and Kernel-Mode System Libraries

Component | User-Mode | Kernel-Mode
Core system library | coredll.dll | k.coredll.dll
Device driver helpers | ceddk.dll | k.ceddk.dll
Credential provider | credsrv.dll | k.credsvr.dll
IP network helper API | iphlpapi.dll | k.kphlpapi.dll
Multimedia timer | mmtimer.dll | k.mmtimer.dll
Process, thread, and memory helpers | toolhelp.dll | k.toolhelp.dll
Windows Sockets | ws2.dll | k.ws2.dll
Windows Sockets service provider interface | wspm.dll | k.wspm.dll

Memory Architecture
The new kernel brings a completely new memory architecture that eliminates the previous limits on available processes and address space size. Previous kernels supported a maximum of 32 processes, each of which occupied its own process slot. CE 6.0 raises the limit on the number of processes to 32K. (This is a theoretical maximum, based on a limit on the storage of kernel handles. CE 6.0 can accommodate a maximum of 64K kernel handles, and a process requires a minimum of two handles-one for the process itself, and one for the process's thread. The practical maximum is lower because other types of kernel objects occupy space in the kernel handle table.)
Prior to CE 6.0, each Windows CE process had a 32MB virtual memory address space. With CE 6.0, each process has a 2GB address space. In addition to having a larger address space, the structure of process address spaces is different. Prior to CE 6.0, a single virtual address space was divided into the 32 slots, with no overlap between process address spaces. With CE 6.0, each process gets its own, truly private process address space. This change makes the CE 6.0 process address space appear very similar to the process address space in desktop versions of Windows (such as Windows XP).
No doubt you're wondering what impact the new memory architecture will have on existing software. Generally speaking, most changes are limited to device drivers. While the size of the process address space is much larger, well-behaved applications shouldn't notice, since memory is allocated using the same allocation APIs, and data is still stored using 32-bit virtual memory pointers. The new memory architecture does make some things easier, particularly for applications that needed to allocate very large blocks of memory-greater than 10MB, for example-such as might be needed to hold a high-resolution image read in from a digital camera sensor. Previously the need for large data blocks had to be satisfied by allocating with the shared memory allocation functions (MapViewOfFile, for example). Starting with CE 6.0, large allocations can just as easily be satisfied with a call to VirtualAlloc.
Aside from the increase in size of the process address space, the other change introduced with this memory architecture is the truly private process address space. Prior to CE 6.0, it was generally quite easy for one process to look into the address space of another process. In the now defunct slotted address scheme, the currently running process is mapped into slot zero. This was done by fixing up all pointers to reference memory in the range of zero to 32MB. By calling a function like MapPtrProcess, the currently running process can take a pointer from another process and access that memory directly in the process slot (in the range 32MB-64MB for slot 1, 64MB-96MB for slot 2, and so forth). With the implementation of a private process address space, no application process can look into the address space of any other application process. Instead, you will need to use the standard Win32 shared memory APIs to enable two processes to read and write to the same memory.
One reason that this new architecture is likely to have the biggest impact on device drivers is that existing Windows CE device drivers often use functions like MapPtrProcess to read or write directly in an application's address space. In CE 6.0, drivers that still need to access data in an application's address space need to run in kernel mode. This is accomplished by linking to the appropriate set of kernel libraries (k.coredll.dll) instead of their user-mode counterparts.
In some cases, it might be necessary for a device driver to run in user mode instead of in kernel mode. For example, there might be a driver for which you don't want to grant full kernel mode privileges. Such a driver can be run in CE 6.0 through the use of a user-mode device driver process-udevice.exe.
Now let's take a look at some of new features that are available with CE 6.0. Some are developer tool enhancements, while others are new features that you can include in custom embedded devices that you create.

Device Emulator
Many stories told by science fiction authors describe machines that behave like humans. And yet, stories are seldom written to extol the glory of machines that mimic other machines. Isn't there something miraculous about a desktop system being able to simulate the behavior of a mobile phone?
Such impersonation is actually nothing new. In fact, it is almost as old as computers themselves. In the 1960s, the then-new IBM 360 could emulate its predecessor, the IBM 1401. In the 1970s, a DEC-10 emulated a MITS Altair to debug the first Microsoft BASIC interpreter. Today, of course, there's the Microsoft Virtual PC, where emulation becomes virtualization.
Windows CE has seen three generations of emulators. The first supported Windows CE by directly calling the roughly equivalent Windows NT functions. It was a good first start, but left much to be desired. The second generation emerged as a custom embedded platform using hardware virtualization, running the same low-level instructions as a real device. It was a very capable emulator, but had a drawback for developers who were focused on deploying to non-x86 processors since it was built using the I86 instruction set. A third-generation emulator, which ships with CE 6.0, provides instruction-level emulation of the ARM V4I instruction set. This emulator ships with the CE 6.0 Platform Builder as the Device Emulator.
This newest generation supports the ARM V4I instruction set. While the x86 instruction set dominates on the desktop, it has not made the same inroads in the embedded world. That honor belongs to the instruction sets designed and licensed by ARM Holdings, Ltd. of the UK. The benefit of emulation at the machine instruction set level is binary compatibility. Instead of building the x86 executable files for the emulator and a second set of ARM executables for actual devices, you can create a single set of binaries and run them both. In addition to being more convenient, it also enables a higher degree of testing confidence because you test everywhere with the same executables without worrying about errors in your setup scripts, or compiler or linker bugs.

Cell Technologies for CE 6.0
To enable machine-to-machine communication, CE 6.0 includes the interfaces needed to connect to mobile phone networks. Previously, Windows CE did not provide support for making phone calls and sending SMS (text) messages. With CE 6.0, Microsoft has made available a select set of the components for connecting to mobile phone networks.
CE 6.0 provides cellcore.dll, which extends the Win32 API to support a variety of mobile phone services such as initiating a data connection, sending an SMS message, and so on. Another key component provided with CE 6.0 is ril.dll, the driver for the Radio Interface Layer (RIL). This component provides the low-level interface for connecting an application layer to the mobile phone hardware. Previously, to build a mobile phone on Windows CE, you had to develop your own interface layer. This is not an easy thing to do, but something that a few stalwart pioneers have done.
Among the other supporting elements in CE 6.0 are the low-level components for the Wireless Application Protocol (WAP), including a kernel-mode driver (wapdrv.dll) and a user-mode API (wap.dll). You should note that the browsers provided with CE 6.0 do not support WAP browsing, but instead are HTML-only browsers. The Pocket Internet Explorer® that ships on Windows Mobile-powered devices (which does support WAP) is different from the HTML-only version that ships with CE 6.0.
To enable the creation of such wireless data terminals, there is rich support for SMS messaging. This is important because the mobile phone network uses SMS to send notifications for services like voice mail, fax, and e-mail. And so CE 6.0 provides a rich set of SMS service providers to enable the receipt of SMS broadcast messages, notification messages, and status messages.

Location-Based Programming
Before computers became mobile, there was little interest in figuring out the computer's location. Once delivered to its air-conditioned vault, a mainframe computer never needed to figure out where it was. With today's pocket-size systems and onboard automobile computers, the subject of location is suddenly much more interesting.
Microsoft introduced support for Global Positioning System (GPS) APIs with Windows Mobile 5.0. Those same APIs, and the drivers to support them, are now available with CE 6.0.

Security Enhancements
A major theme of the past few years has been finding ways to improve security of all computer systems. Security is one of the main themes of Windows Vista™, after all. And the CE 6.0 team has also made security an important aspect of their work.
Before discussing improvements that have been made with CE 6.0, it's worth pointing out the security features that already exist in Windows CE. The centerpiece of Windows CE security is provided by the ability for a Windows CE-powered device to maintain tight control over which applications and DLLs are allowed to load and run. Through the OEMCertifyModule function, a device can prevent any unauthorized code from running. A common method for identifying authorized modules is through the use of digital certificates. Device security can be set up in a variety of different configurations. For example, in one configuration any unknown modules-those without valid certificates-can be denied any system access. Alternatively, this mechanism can be turned off so that all modules have complete access to all system services.
Another core security feature is the Cryptographic API, which allows applications to encrypt and decrypt blocks of data using a variety of encryption algorithms. Encryption algorithms are identified by symbolic names with the prefix CALG_, such as CALG_DES and CALG_AES. A second is support for secure sockets layer (SSL), which provides secure HTTP connections. In addition, Virtual Private Networking (VPN) is supported through the point-to-point tunneling protocol (PPTP). And to enable secure connections with server systems, Windows CE provides support for a variety of authentication mechanisms, including the Windows NT LAN Manager protocol and the more robust Kerberos authentication protocol.
Other features from previous versions of Windows CE include the Credential Manager, support for Smart Cards, the Data Protection API (DPAPI), support for public key certificates (PKI), and also the Local Authentication Subsystem (LASS). So what are the new security features with CE 6.0?
As I discussed earlier in this article, there is strict separation of user-mode code from kernel-mode code. In CE 6.0, enhanced validation checks of the Protected Server Library (PSL) and I/O Control's (IOCTL) parameters that are passed from user mode to kernel mode are performed, improving kernel mode security and stability.
The perimeter security of a system can be vastly improved by supporting a secure loader. A secure loader ensures that only trusted code gets to run on a system. In Windows CE 5.0, the operating system provided a loader hook but OEMs had to write their own secure loader. CE 6.0, in contrast, ships with a built-in secure loader. The trust decisions of the loader are certificate-based-this means that all code that runs on the system has to be signed. The secure loader, if enabled, inspects the code signature and if the signature is signed by a trusted certificate then that code is allowed to run. If not, the module load fails. The OEM has control over what certificates are trusted and thus has control over the code that gets to run on the system.
Another area in which security improvements have been made is in the Windows CE OS Design (or the New Platform) Wizard. When a platform is configured with a feature that may compromise the security of the device, a security warning is issued. For example, Figure 3 shows the notification displayed when the Object Exchange (OBEX) Protocol is included in a platform. Details of the potential compromise are provided, to help platform developers address the potential problems early in the platform development process.
Figure 3 A Security Warning from the Design Wizard
Figure 3 A Security Warning from the Design Wizard
A final feature that was added to CE 6.0 security is support for a secure boot loader. This feature ensures that downloaded operating system (NK.BIN) images contain valid digital signatures before allowing the OS images to be installed and run on a system.

Development Tools
Prior to CE 6.0, the Windows CE team shipped a standalone product called Platform Builder. With CE 6.0, the platform development tools are integrated into Visual Studio® 2005. Visual Studio is, of course, the premier development tool for both client and Web development; now this same tool is available to support Windows CE. Features that have benefited application developers-like IntelliSense®, syntax checking, syntax highlighting, outline view, and function completion-can be used in the development of custom Windows CE platforms.
Some Platform Builder terms have changed to better fit into the Visual Studio paradigm. A Windows CE 5.0 Workspace is now a Solution in CE 6.0. A Windows CE 5.0 Project (an IDE-defined application or DLL) is now a Subproject in CE 6.0.
It was strange at first to start Visual Studio 2005 when I wanted to configure and build a new CE 6.0 platform, but I quickly learned how Windows CE 5.0 settings and commands mapped to Visual Studio. The command to create a new Windows CE platform is similar in the two versions. For Windows CE 5.0, you would select the File | New Platform... menu item. For CE 6.0, you select the File | New | Project... menu. With CE 6.0 tools installed, a new project type gets added to the Visual Studio New Project window. The project type is named "Platform Builder." Figure 4 shows this change, along with a new project template for "OS Design."
Figure 4 New Project Window with the Platform Builder Project Type
Figure 4 New Project Window with the Platform Builder Project Type
By selecting the project's location and name and clicking OK, you summon the Windows CE OS Design Wizard. Aside from the new name for what is being created-an "OS Design" instead of "Platform Builder Workspace"-the new CE 6.0 wizard is identical to the old Windows CE 5.0 wizard.
Apart from a few refinements, the remainder of the OS Design Wizard will be familiar to anyone with experience with the Windows CE 5.0 Platform Builder. As in the earlier version, the first step in defining a new platform is the selection of one or more Board Support Packages (BSPs). After that, the wizard prompts for a design template, which serves as a platform's initial configuration. Some of the Windows CE 5.0 design templates have been renamed; in some cases older design templates appear as design template variants. What was known as Tiny Kernel in Windows CE 5.0 has been renamed Small Footprint Device in CE 6.0, but otherwise it represents a minimum set of operating system features for a headless Windows CE-powered platform.
The Windows CE OS Design Wizard creates an OS Design (.pbxml), which is a Visual Studio project. The OS Design is added to a solution (.sln) file. After the wizard finishes, the newly created solution file is opened. When a Windows CE design solution is open, Visual Studio changes its menus to display the familiar Platform Builder menus: Build and Debug (see Figures 5 and 6). Changes are also made to the Target menu.
Figure 5 Build Menu
Figure 5 Build Menu
Figure 6 Debug Menu
Figure 6 Debug Menu
If you are new to Visual Studio, here are two tips to help get you started: the Solutions Explorer window plays a key role; if it is not open, summon it with View | Solution Explorer. Many configuration details are available by drilling into the contents of this window.
Context (or popup) menus also play a key role in Visual Studio; summon these menus by selecting and clicking the right mouse button. For example, you can open an OS configuration window by selecting an OS Design solution from the Solution Explorer window and selecting the Properties menu item.

Cool Tools
The most constrained resource in the world of software development is people time. And because better tools make for more productive people, the investment in development tools never ends. Now that the Windows CE Platform Builder-the centerpiece of embedded development-has been integrated into Visual Studio 2005, the Platform Builder team will be able to focus more efforts on adding cool new tools instead of on maintaining the IDE, since Visual Studio has its own dedicated development team.
You can already get a taste of the cool future tools that will come out of this group with two new capabilities being introduced with CE 6.0. One is a graphical registry file editor, which makes it easier to edit and also add new registry keys and registry values into .reg files. A second cool tool is the runtime image viewer for peeking inside nk.bin files. This tool even allows you to compare two different nk.bin files against each other. And finally, there is shared source. While technically not a tool, the phenomenon of open source that benefits Windows CE developers is something I'm very excited about.

Registry File Editor
An important aspect of configuring Windows CE components involves creating settings for the system registry. Previously, registry (.reg) files had to be edited in their raw text form, like the one in Figure 7. Visual Studio provides a new registry file editor, as shown in Figure 8, which is a lot more pleasant to use. When necessary, you can still edit the plain-text version of the registry file by clicking on the Source button at the bottom of the registry editing window. But the default editing mode resembles the familiar registry editing tools used by developers.
Figure 7 Typical Registry Files for a Windows CE Component
IF BSP_NONLED !

; HIVE BOOT SECTION

[HKEY_LOCAL_MACHINE\System\Events]
    "SYSTEM/NLedAPIsReady"="Notification LED APIs"

; END HIVE BOOT SECTION

[HKEY_LOCAL_MACHINE\Drivers\BuiltIn\NLed]
   "Prefix"="NLD"
   "Dll"="nleddrvr.dll"
   "Flags"=dword:8                      ; DEVFLAGS_NAKEDENTRIES
   "Order"=dword:0
   "IClass"="{CBB4F234-F35F-485b-A490-ADC7804A4EF3}"

ENDIF BSP_NONLED !
Figure 8 Visual Studio Registry File Editor
Figure 8 Visual Studio Registry File Editor

Runtime Image Viewer
Another nice development feature is the runtime image viewer. By convention, a Windows CE image is written to the nk.bin file. In the past, it was difficult to know exactly what was in a given nk.bin file. And it was just about impossible to compare the contents of one nk.bin to a second, leaving a lot of troubleshooting to the realm of trial-and-error, guesswork, and wracking your memory for the details of long-forgotten changes. The image viewer provides help for these kinds of problems.
Figure 9 shows the contents of a Windows CE image that was built using the Device Emulator BSP and the Small Footprint Device design template. You can view header information for any executable module (*.exe and *.dll) file by selecting the module and viewing the results in the Visual Studio Properties window. The image viewer allows you view the contents of any file included in the FILES section of a .bib file.
Figure 9 Using the Runtime Image Viewer to See nk.bin
Figure 9 Using the Runtime Image Viewer to See nk.bin

Shared Source and Windows CE Development
With CE 6.0, Microsoft continues a program that began with Windows CE 3.0-the shared source program. Taking a cue from the open source movement, Microsoft began by making available a significant portion of the source code for Windows CE to anyone willing to download the Evaluation edition of the Windows CE Platform Builder.
Developers who build applications to run on Windows CE have gotten tangible benefits from other open source projects. Perhaps the best known is the OpenNETCF project, an extension to the .NET Compact Framework. The OpenNETCF project is available for download.
Microsoft has directly sponsored several source code projects such as the Bluetooth project, which provides managed code classes to simplify the use of Bluetooth in Windows CE-based applications. Another is the webcam driver used to kick-start designs with webcams when no drivers were publicly available. A third project sponsored by Microsoft is the digital video recorder (DVR) project, to support the creation of Windows CE-powered video recorders.

Conclusion
The world of Windows CE development keeps getting better. Whether your interest is application development and Windows Mobile-powered devices, device-driver development or low-level BSP programming, you'll keep finding innovations that make your development life easier and more efficient. A new kernel has eliminated some nagging limitations on available processes and process address space. Integration with Visual Studio has opened up all of the productivity enhancements that have been baked into Visual Studio 2005.
If you liked Windows CE before, you're going to love it now. If you are new to the world of Windows CE, you've probably guessed by now that there's a lot of potential waiting for you to tap into. And if you want to join a discussion of the possibilities, check out the Windows Embedded Developer Interest Group.

Paul Yao is a Business Alliance Manager with Utimaco Safeware, The Data Security Company. Paul is a contributing editor to MSDN Magazine, and coauthor of eight books including the first book published on Windows programming. Paul is founder of the Windows Embedded Developers Interest Group (WE-DIG), and founder of The Paul Yao Company.


Graphics To Go
Make A Mobile Imaging App With The .NET Compact Framework 2.0
Rob Pierry

This article discusses:
  • Managed code for mobile apps
  • Testing with device emulators
  • Graphics with the .NET Compact Framework
  • Interacting with Pocket Outlook
 This article uses the following technologies:
The .NET Compact Framework 2.0, Windows Mobile 5.0, Visual Studio 2005
Contents
Working with Images
Drawing Shapes and Text
Device Emulator Manager
Using the Camera
Checking System State
Working with Pocket Outlook
Advanced Synchronization Ideas
Conclusion
With the introduction of the .NET Compact Framework, Microsoft brought all of the advantages of managed code development to mobile applications. Developers already familiar with writing Windows® Forms applications could quickly get up to speed on programming for mobile devices. Version 2.0 of the Microsoft® .NET Compact Framework is now available and with it development has become even easier through the introduction of new tools (like the Device Emulator Manager) and a broadening of its coverage. In particular, version 2.0 adds more classes and methods for working with graphics-including, among other enhancements, angled text, custom pens, image manipulation, and interacting with a device's built-in camera. This article will cover these and other new enhancements that are available with Visual Studio® 2005 and the .NET Compact Framework version 2.0.
When you use the .NET Compact Framework to develop for Windows Mobile® 5.0, there are two main types of device you can target: Smartphones and Pocket PCs. While the tools for developing for each of these platforms may be the same, the development experience for each has been customized for the class of device it targets. Pocket PCs tend to be larger PDA devices that may optionally include phone functionality. Smartphones, on the other hand, are smaller devices that are primarily phones with PDA functionality included secondarily. Generally, Pocket PCs are more powerful, offering touch screens, higher screen resolutions, more memory, and faster processors. This article focuses on developing for Pocket PCs. But once you know how to develop for Pocket PCs, developing for Smartphones should be quite easy.
Before getting started, you need to install the Windows Mobile 5.0 SDK for Pocket PC. The SDK includes emulators for Pocket PC devices, documentation, and samples. Once that is installed, you're ready to get started.
In this article, I'll guide you through the development of a simple imaging application that supports a background image with text and lines drawn over it. The background image can either be loaded from an existing picture or captured using the device's built-in camera. Both of these tasks-opening and capturing pictures-will make use of standard dialogs that are new to version 2.0 of the Compact Framework. For the overlay, I'll demonstrate how to use the newly supported features of custom pens and angled text to draw on top of the background. The application will also let you save your pictures after you've drawn on them. As a quick aside, I'll show you how to add support for e-mailing pictures to a Pocket Outlook® contact.

Working with Images
At a fundamental level, the application must be able to store the background image and the foreground overlay of text and lines, and it must be able to form these elements into a single image (see Figure 1). Internally, the images remain separate elements until the application saves the final picture. This is to support the basic editing capabilities-replacing the background while keeping the overlay and clearing the overlay while keeping the background.
Figure 1 Seperate Background and Foreground Layers
Figure 1 Seperate Background and Foreground Layers
When the main form loads, it creates two instances of the Bitmap class, one for the background (which defaults to light gray) and one for the overlay (which defaults to transparent). When the form paints itself, it draws the background and then the overlay.
To support transparency, the application uses the ImageAttributes class. In general, this class can be used to set a range of colors to be transparent, as well as to replace colors with other colors and to perform other manipulations targeted at bitmaps, brushes, pens, and the like. Unfortunately, in the Compact Framework, only the transparency settings are supported. The application uses a specific color to represent transparent and then uses ImageAttributes to prevent that color from being drawn when the form is painting. In this example, you'll use Color.AliceBlue as transparent and then you will make sure to not allow the user to choose this as a color for drawing text and lines. The application only needs to call the SetColorKey method of an instance of ImageAttributes and pass Color.AliceBlue for both parameters since you want a range of exactly one color to be transparent. Here's the code for drawing the overlay onto the background:
private void DrawOverlay(Graphics g, Rectangle region)
{
    using(ImageAttributes attribs = new ImageAttributes())
    {
        attribs.SetColorKey(_transparentColor, _transparentColor);
        g.DrawImage(_overlay, region, region.X, region.Y,
            region.Width, region.Height, GraphicsUnit.Pixel, attribs);
    }
}
To keep things simple, the application limits the image size to the visible area of the device's screen, available through the ClientRectangle property. Bear in mind that this value isn't necessarily constant. On devices with slide-out keyboards, for example, the client rectangle can be different depending on whether the keyboard is open (320×188) or closed (240×268). To work around this, the application defaults the images to the size of the client rectangle when the application is started. This underscores the importance of testing your Windows Mobile applications on actual hardware since the emulator can't cover all hardware configurations.
While the solid light gray background is very pleasant, it would be nice if users could select their own pictures to use as backgrounds. The application uses the new SelectPictureDialog class to accomplish this. Figure 2 shows the interface of SelectPictureDialog. This class is available in the Microsoft.WindowsMobile.Forms assembly, which you'll have to add a reference to if you want to use the dialog. The class has properties that let you set the dialog's title (Title), control whether or not the user can take a new picture with the built-in camera (CameraAccess), filter which pictures are shown by file type (Filter) or whether they are protected by Digital Rights Management (ShowDrmContent and ShowForwardLockedContent), set the initial folder to display (InitialDirectory), and prevent the user from changing to any other folder (LockDirectory). After the dialog is closed, you can check the FileName property to get the file name of the picture the user selected.
Figure 2 Selecting a Picture
Figure 2 Selecting a Picture
Once the user selects a picture, the application replaces the current background with the selected picture, scaled down to fit on the screen. After grabbing a Graphics object for the background image, the DrawImage method is used to draw the newly selected picture onto the background. The second parameter represents the rectangle of the destination onto which you will be drawing (in this case, this is the entire background). The third parameter is the rectangle of the source image (meaning the entire source image), which is being mapped into the destination rectangle. This method is very flexible. How you're using it now automatically makes smaller pictures get stretched and larger pictures get shrunk to fit the background, but you could very easily just take a background-sized chunk out of the new picture by passing the background rectangle as both parameters.
Now that the application is able to create and draw the background and overlay, the task of saving is pretty simple. Instead of painting on the form, you use the SaveFileDialog to prompt the user for the new file name. It creates a copy of the background and then draws the overlay on top of it. Once you have the final image, you use the Save method (new in version 2.0 of the Compact Framework) to write the image out to disk in JPEG format. (BMP, GIF, and PNG formats are also supported.) Here's the code for generating and saving the image:
using (Bitmap newImage = new Bitmap(_background))
{
    using (Graphics g = Graphics.FromImage(newImage))
    {
        DrawOverlay(g, new Rectangle(
            0, 0, _overlay.Width, _overlay.Height));
    }
    newImage.Save(dlgSave.FileName, ImageFormat.Jpeg);
}

Drawing Shapes and Text
Now the user can customize the background by selecting an image, but you still need to allow the user to draw on the overlay. The Graphics object is used for drawing shapes. This has methods like DrawEllipse, DrawLine, and DrawRectangle-all of which take a Pen as a parameter. The Pen dictates the color and thickness used to draw the shapes. Previously, only Pens of a specified color could be created, but the Compact Framework version 2.0 exposes the Width property and another constructor, allowing you to specify both the width and color to use.
In the application, SelectPen.cs implements a form that lets the user choose a color and width for drawing shapes. The available colors are listed in a ComboBox and the width is specified using a NumericUpDown control. After the user makes selections, the main form sets the current pen to a new instance of the Pen class with the specified color and width. All subsequent drawing uses this pen. In this form, remember that you have to make sure that the color that represents transparent in the overlay (AliceBlue) isn't listed in the ComboBox. Otherwise the user may inadvertently draw invisible shapes.
The Draw Line option under the Draw menu shows how the current pen can be used to draw a line on the image. First, the user selects the two end points of the line. The application then obtains a Graphics object for drawing on the overlay and uses the DrawLine method with the current pen to draw a line connecting the two points.
The last step is to call Invalidate with the whole ClientRectangle as a parameter. This results in the form repainting itself, causing the background and the updated overlay to be drawn on the form. Strictly speaking, the only portion of the form that needs to be updated is the rectangle that contains the line that was just drawn. Therefore, one optimization is to invalidate only this rectangle instead of the entire client area.
When looking at the Graphics object, you may have noticed the DrawString method. This method draws a string in the specified font and color at a specified location. Since version 2.0 of the Compact Framework supports angled text, you can let the user select two points to define the line along which text will be drawn and then provide a dialog that allows the user to enter the text and select a size.
Angled text is supported via the LogFont class. This name stands for logical font, which is a structure utilized internally by Windows CE, the underlying operating system upon which Windows Mobile 5.0 is based. To adjust the angle of the text, you set the Escapement property of a LogFont instance to an angle in degrees, multiplied by 10. For example, for text that runs vertically from bottom to top, set the Escapement to 900 (90 degrees from the positive x axis).
The code for creating a Font via a LogFont is provided in the GetFont method of the text entry dialog, TextDialog.cs. There is a small bit of trigonometry at the top used to derive the angle for the text, given two points. Note that the y-axis of the device is inverted, having 0 at the top of the screen rather than at the bottom. Once you adjust for this and adjust for the range of the arctan function, you have a suitable angle for the Escapement property.
Keep in mind that when working with LogFont, you must always set the Orientation property to the same value as Escapement. To set the Height of the LogFont, you take the size provided by the user and adjust for the DPI of the device. (You can determine the DPI using the new DpiX and DpiY properties of the Graphics object.) When you provide this as a negative number, the Framework selects the font with the point size closest to the absolute value of the property. There are more options here, which are explained in the MSDN® documentation. Figure 3 shows the code used for getting a font via LogFont.
Figure 3 Getting a Font via LogFont
public Font GetFont(Point p1, Point p2, float dpi)
{
    int y = -(p2.Y - p1.Y); //adjust for inverted y axis
    int x = p2.X - p1.X;
    int angle = (int)(Math.Atan2(y, x) * 180 / Math.PI);

    //adjust for quadrant
    if (y < 0) angle = 360 + angle;

    LogFont lf = new LogFont();
    lf.Height = (int)(-EnteredSize * dpi / 96);
    lf.Escapement = angle * 10;
    lf.Orientation = lf.Escapement;
    lf.FaceName = Font.Name;

    return Font.FromLogFont(lf);
}
Once the LogFont has been set up appropriately, you can create an instance of Font by calling the static FromLogFont method of the Font class. Now that you have an appropriately sized font, some text, and a point, you can easily output it to the image using the DrawString method.

Device Emulator Manager
If you've done any mobile development with Visual Studio, you're probably already familiar with the Device Emulator. The emulator looks and acts just like a Pocket PC, letting you simulate running your programs on a real mobile device. The emulator can be configured to a variety of screen sizes, memory configurations, and available ports. It can even be configured for simulated phone functionality.
In Visual Studio 2005, emulators are handled via the Device Emulator Manager, which you can access via the Tools menu. When the Device Emulator Manager starts up, it lists all available emulators, grouped by target operating system. Note that there are several choices listed under the Windows Mobile 5.0 Pocket PC SDK, including the base emulator, one with phone functionality, and one with a VGA (640×480) resolution screen.
Right-clicking on an emulator and selecting the Connect option causes the emulator to start up. The Device Emulator Manager then displays a green arrow next to the emulator. Figure 4 shows the Device Emulator Manager with an emulator running.
Figure 4 Device Emulator Manager with an Emulator Running
Figure 4 Device Emulator Manager with an Emulator Running
A feature new to Visual Studio 2005 and ActiveSync® 4.0 is the ability to create a partnership between the desktop machine and the emulator, as if it were an actual device. To do this, open ActiveSync and choose File | Connection Settings. Make sure Allow connections to one of the following is checked and select DMA from the dropdown (the emulator talks to ActiveSync over DMA). Press OK. Now, in the Device Emulator Manager, right-click on the running emulator (the one with the green arrow next to it) and choose Cradle. ActiveSync will activate and then prompt you to create a partnership with the new device. The Device Emulator Manager will change the green arrow to a different icon indicating that the device is cradled. To disconnect the emulator from ActiveSync, simply right-click on the emulator in the Device Emulator Manager list and choose Uncradle.
Although the Device Emulator Manager has made emulators more powerful in Visual Studio 2005, the emulators aren't without limitations. As I mentioned earlier, the emulator doesn't currently simulate devices with slide out keyboards. Another critical limitation comes up when you start adding support for backgrounds captured via a device's camera. Right now the emulator doesn't support camera functionality at all. If you try to access the camera dialog while running on the emulator, you will get an Invalid-OperationException.

Using the Camera
Now it's time to let the user take a picture to serve as the background. To run this portion of the application, you'll need a Windows Mobile 5.0 Pocket PC with a camera. Once you've connected the device to your machine and optionally created a profile in ActiveSync, change the Target Device from Windows Mobile 5.0 Pocket PC Emulator to Windows Mobile 5.0 Pocket PC Device in the dropdown on the Device toolbar and then run the project as usual. (Note that breakpoints and debugging continue to work when you deploy to a real device just like during debugging within the emulator.)
When the user selects New Background from the File menu, the application creates and shows an instance of the new CameraCaptureDialog. This is in the same assembly as the SelectPictureDialog, Microsoft.WindowsMobile.Forms. The dialog lets you fully configure the initial settings for the camera, although users can still adjust them via the regular options menu. The Mode property specifies whether the user can only take a single picture (CameraCaptureMode.Still), a video (CameraCaptureMode.VideoOnly), or a video with audio (CameraCaptureMode.VideoWithAudio). Since you are looking to capture a background image, you want the application to set the dialog to still picture mode and to use the highest quality for still images (using the StillQuality property). You can also set properties indicating an optional time limit for the length of the video (VideoTimeLimit) and for ensuring that the video is suitable for sending in a multimedia message (VideoTypes).
Optionally, the Resolution property can be set to control the size of the picture being captured. In general, arbitrary values aren't supported. Instead, you should use values appropriate to the capabilities of your device (such as 640×480 or 1280×1024). Since you'll be scaling the picture down to fit onto the screen, leave the Resolution at its default value.
Once the user has taken a picture, the ShowDialog method returns with a result of DialogResult.OK. Now, to load the newly taken picture, you can use the same scaling technique you used earlier to resize an existing image to the background. Since only the background image is replaced, any lines or text the user has already drawn remain intact. Once the picture has been drawn onto the background, you call Invalidate with the entire client rectangle, and this causes the screen to repaint with the new picture now displayed as the background.

Checking System State
You can show the CameraCaptureDialog and then, if the device doesn't have a camera, have the InvalidOperationException raised. But it is more graceful to determine ahead of time whether the camera capability exists and then simply disable the menu option if the device does not offer camera functionality. You can do this by querying the device for its general configuration and monitoring real-time properties for changes.
Information about the device is exposed through the SystemState class, which is contained in the Microsoft.WindowsMobile.Status assembly. You can use this information in your application by adding a reference to the assembly and its dependency, Microsoft.WindowsMobile. The SystemState class has a host of static properties that let you determine fixed configuration information and real-time information like the current date and time or the remaining charge in the battery.
The application only needs to check the CameraPresent property once at startup, and then it can disable the New Background menu option if a camera isn't available. The SystemState class, however, exposes a Changed event that you can handle to be notified when the requested state changes. Instead of using the static properties, you can create an instance of SystemState, passing the specific system property to be monitored into the constructor and then just adding a handler to the Changed event. You could, for example, extend the application to monitor the DisplayRotation system property so the application is notified when the user slides out the device's keyboard and the display changes from portrait to landscape. Or the application could monitor the ActiveSyncStatus property to possibly take some action when the device is being synchronized.
The SystemState class also takes the ComparisonType and ComparisonValue properties into account before raising the Changed event. By default, all changes in the monitored property result in the Changed event being raised. But if, for instance, you set the ComparisonType property to ComparisonType.Greater and the ComparisonValue property to 2, the Changed event will only be raised if the system property you're monitoring changes to a value greater than 2. Using other comparison types-like Equals, Contains, or StartsWith-you get finely grained control over which events your application handles.
The main limitation to SystemState is that your application has to be running when the status changes in order for it to be notified. You also need to make sure you declare the instance of SystemState whose Changed event you're handling as a class-level field, instead of a local method variable, so that it isn't garbage collected, causing you to miss events.

Working with Pocket Outlook
Windows Mobile 5.0 provides a new assembly, Microsoft.WindowsMobile.PocketOutlook, for interacting with the messaging accounts, tasks, calendar items, and contacts stored on a device. You can use this assembly in your application, allowing the user to select a contact with an e-mail address and then send the contact a picture.
Using the new ChooseContactDialog, you can prompt the user to either select an entire contact or a specific property of a specific contact (as shown in Figure 5). One of the most useful features of this dialog is the ability to filter the list of available contacts. For this application, you only want to show contacts with e-mail addresses, since the feature is intended for e-mailing a picture to the contact. The RequiredProperties property of the dialog contains an array of ContactProperty enumerations that control the filtering. The application uses a single ContactProperty with the value ContactProperty.Email1Address to indicate that only contacts with a valid e-mail address should be shown. This enumeration contains all the properties of the contact, such as Birthday, BusinessAddress, and HomeTelephoneNumber. Using RequiredProperties, you can easily filter the list of contacts down to a manageable list that is relevant to how the contact will be used in this application.
Figure 5 Selecting a Contact
Figure 5 Selecting a Contact
You use an instance of the EmailMessage class to actually send the message. You simply add a new Recipient to the collection in the To property, setting the Subject and Body properties, and add the image to the collection of Attachments. The main interaction with all messaging accounts, the calendar, and other Outlook functions is accomplished through OutlookSession. Once you create an instance of OutlookSession, you can use the EmailAccounts property to grab the first e-mail account and send the message. Based on the status and settings of the device, the e-mail is either sent immediately or the next time the user has network connectivity and starts a send operation. Figure 6 shows the code for sending the e-mail.
Figure 6 Using OutlookSession to Send E-Mail Messages
if (dlgContact.ShowDialog() == DialogResult.OK)
{
    //send an email
    EmailMessage msg = new EmailMessage();
    msg.To.Add(new Recipient(dlgContact.SelectedContact.Email1Address));
    msg.Subject = "Here is an image I created...";
    msg.BodyText = "...using Windows Mobile 5";
    msg.Attachments.Add(new Attachment(filename));

    using (OutlookSession outlook = new OutlookSession())
    {
        outlook.EmailAccounts[0].Send(msg);
    }

    MessageBox.Show("Send successful.");
}

Advanced Synchronization Ideas
Although sending an e-mail message works fine for the simplified example, it isn't always the correct choice for transferring data. For more complex scenarios when you need to move data between a device and a desktop, ActiveSync would be more appropriate. Even though Microsoft has greatly expanded the coverage of the Compact Framework in version 2.0, some areas still remain outside the realm of managed code. Interacting with ActiveSync and communicating with a device from a desktop machine still require the use of unmanaged code. But there are ways of getting started without writing unmanaged code.
ActiveSync synchronizes data between a mobile device and a desktop through a partnership and various service providers (such as the Outlook provider for synchronizing mail and calendar information). Each provider is comprised of a set of DLLs (one on the device and one on the desktop) that implement specific COM interfaces. These providers are very capable, but also complicated to develop. For basic scenarios where intricate synchronization is less of a concern, however, there is an easy way to be notified when a mobile device has been connected or disconnected without implementing a service provider.
In the registry there is a list of programs to be run automatically when a device either connects or disconnects. The keys are located at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows CE Services\AutoStartOnConnect and \AutoStartOnDisconnect. To make use of these keys, just create a string value with a unique name that contains the path to the executable you want to run. For example, if you add a string with the name NotepadTest and the value notepad to AutoStartOnConnect, Notepad will launch every time a device is connected to the system. If you want to pass command-line arguments to the program, enclose the name of the program in double quotes. A value of "notepad" c:\test.txt will start Notepad with the contents of test.txt when a device connects. Alternatively, Windows Mobile 5.0 allows you to start and stop the ActiveSync process from the device programmatically through the unmanaged functions ActiveSyncStart and ActiveSyncStop.
Now that you know how to start an application on the desktop whenever a device is connected, the next step is to communicate with the device, which is accomplished via the Remote Application Programming Interface (RAPI). The .NET Framework does not currently include a managed version of RAPI. Since RAPI is just a native API, .NET supports the ability to call the functions it contains via Platform Invoke. Creating a .NET-based application that uses this API and is automatically started when a device connects is an easy way to handle moving data back and forth between a mobile device and a desktop machine.

Conclusion
The Compact Framework version 2.0 (along with Visual Studio 2005) builds on the successes of the first release to make developing mobile applications even easier. And the applications you produce will be even more capable. Since developers can use the same tools and classes to develop mobile applications as they use to develop other managed code applications, more developers can more quickly get into writing powerful mobile applications.
Meanwhile, the new APIs allow developers to take fuller advantage of all the capabilities of increasingly powerful mobile devices. They can write apps that interact with a device's camera and include features for communicating with other devices. Ultimately, developers can create the most well-suited solutions for their needs, utilizing the appropriate features on the appropriate devices, without having to worry about extra development effort and costs.

Rob Pierry is a Senior Consultant with Geniant, a leading enterprise IT consulting company specializing in service-oriented architecture. He can be reached at rpierry+msdn@gmail.com.


Windows Workflow
Build Custom Activities To Extend The Reach Of Your Workflows
Matt Milner

This article is based on a prerelease version of the .NET Framework 3.0. All information herein is subject to change.
This article discusses:
  • Building basic and composite activities
  • Asynchronous and event-based activities
  • Customizing the design experience
  • Validation and error handling
 This article uses the following technologies:
.NET Framework 3.0
Contents
Building a Basic Activity
Dependency Properties
Building Composite Activities
Asynchronous and Event-Based Activities
Handling Errors
Design Experience
Validation
Conclusion
Custom workflow activities are one of the most important aspects of the Windows® Workflow Foundation and there are many features to consider when building them. In their January 2006 article, "Simplify Development with the Declarative Model of Windows Workflow Foundation," Don Box and Dharma Shukla discussed the core ideas of building workflows and the interactions between the workflow runtime and activities (see msdn.microsoft.com/msdnmag/issues/06/01/WindowsWorkflowFoundation). In this article, I will cover the core components required for building custom activities for your business domain, including runtime responsibilities, design time experience, and asynchronous activity development.
When building a custom activity, the first choice you have to make is whether to build a simple or a composite activity. Simple, or basic, activities are those whose logic and execution are encapsulated within the code of the activity. Examples of simple activities include the SendEmail activity in the Windows SDK or an activity to query data from a database.
Composite activities, on the other hand, achieve their goals by relying on the execution of child activities. You can author composite activities in two different ways depending on your requirements. One option is to build a composite activity that allows users to add child activities at design time and that controls the execution of those children. For example, the While activity included in Windows Workflow Foundation lets you add a child activity and then configure the While activity's looping functionality. The While composite activity focuses on managing the execution of its children. The other option for building a composite activity is to create an activity that derives from an existing composite activity-the Sequence activity, for example-and add a group of activities as children to define a reusable component.
A reusable activity, such as a manager approval activity, has several steps that are most easily modeled with a collection of existing activities. The manager approval activity could use activities such as IfElse to determine whether the manager is online. If she is online, then an instant message could be sent; otherwise an e-mail would suffice. This activity encapsulates all of the complex logic of requesting approval, which is in and of itself a workflow. Users of this activity can be relieved of having to deal with the complexities and details of actually contacting the manager, simply by using this composite activity in their workflows.
To begin to understand the concepts of building activities, it helps to start with a very basic activity and then gradually build it up into something more complex. This article will make use of two primary activities for the purpose of explaining these concepts: an activity to create a user in the ASP.NET membership store, and a custom Switch activity to model execution similar to the code concept of the same name. The full source for these activities and others can be found in the code download for this article.

Building a Basic Activity
In many ways, building an activity is similar to building a component. After all, an activity is a reusable component that happens to take advantage of the features of Windows Workflow Foundation, the framework in which it executes. The first step in building an activity, then, is to determine its function and its interface. In the case of the CreateUser activity, the function is to create a user through the membership provider model of ASP.NET. The interface will require the properties necessary to create a user: namely things like e-mail address, username, and password.
The code to handle the function of a basic activity is written in the overridden Execute method from the base Activity class. In addition to the code specific to the function of the activity, it is your responsibility as the activity developer to return the status of your activity. For basic activities, this will almost always be a status of Closed. Returning Closed is the indication to the workflow runtime that your activity has completed its execution and the runtime is free to schedule the next activity in the workflow. Figure 1 shows a basic activity used to create a user in the Membership system of ASP.NET 2.0. For more details on the activity lifecycle and the contract between the runtime and activities, refer to the article by Don Box and Dharma Shukla mentioned earlier.
Figure 1 Create User Basic Activity
public string Password
{
    get { return _password; } set {_password = value; }
}

public string Email
{
    get { return _email; } set{ _email = value; }
}

public string UserName
{
    get { return _username; } set{ _username = value; }
}

public string MembershipProviderName
{
    get { return _provider; } set{ _provider = value; }
}

protected override ActivityExecutionStatus Execute(
    ActivityExecutionContext executionContext)
{
    MembershipProvider mp = String.IsNullOrEmpty(MembershipProviderName)
        ? Membership.Provider :
        Membership.Providers[MembershipProviderName];

    MembershipCreateStatus status;
    mp.CreateUser(UserName, Password, Email, null, null, true,
        Guid.NewGuid(), out status);

    return ActivityExecutionStatus.Closed;
}

Dependency Properties
Providing properties on an activity is important to allow consumers of the activity the ability to configure its execution. With standard Microsoft® .NET properties or fields, as shown in the previous example, a consumer of the activity must write code in the workflow to set the values of the properties. But consider a scenario where you want to provide a mechanism for less technical staff to build workflows. You do not want all users to have to write code simply to connect data from one activity to another. The declarative Windows Workflow Foundation model lets you make these scenarios easier, and one of the features that supports this model is dependency properties.
Dependency properties are a means of storing the values of an activity or a workflow's state. Unlike standard properties where the value is stored within the class itself, the value of a dependency property is stored in a dictionary maintained by the DependencyObject base class.
Dependency properties enable several key features, but perhaps the most important is known as activity binding. As the name suggests, activity binding is the act of binding a property on an activity to a property on another activity or on the workflow itself. When you bind a dependency property to another property, the value is automatically propagated for you. For example, if a workflow has a property defined for the user name to be created and you had defined a UserName property as a dependency property, then you could bind the UserName property to the property on the workflow. When the workflow is started with parameters, the value passed into the workflow will automatically be passed to your activity through the binding. Figure 2 portrays this relationship.
Figure 2 Binding with Dependency Properties
Figure 2 Binding with Dependency Properties
This type of binding brings to components the same powerful model that a great many people have come to depend on in UI development. Much like data binding in ASP.NET and Windows Forms, activity binding makes connecting components much simpler. It also makes the idea of a non-programmer building a workflow much more palatable. With properly designed activities, users should be able to drag them onto a design surface and connect the values between activities without having to write any code.
Defining dependency properties in your activity is a two-step process. The first step consists of specifying and registering the dependency property definition, which you do by creating a static variable and setting its value by calling the Register method of the DependencyProperty class. The second step is to define a standard .NET property with get and set methods, where you rely on the base class to store or retrieve the value of the dependency property. Note that there is a code snippet for Visual Studio® 2005 that makes this process easier. The code in Figure 3 shows the UserName property redefined as a dependency property.
Figure 3 UserName Property as Dependency Property
public static DependencyProperty UserNameProperty =
    System.Workflow.ComponentModel.DependencyProperty.Register(
        "UserName", typeof(string), typeof(CreateUserActivity));

[Description("The new username for this user")]
[Category("User")]
[Browsable(true)]
[DesignerSerializationVisibility(
    DesignerSerializationVisibility.Visible)]
public string UserName
{
    get { return (string)base.GetValue(
              CreateUserActivity.UserNameProperty); }
    set { base.SetValue(CreateUserActivity.UserNameProperty, value); }
}
Note also that in this example the .NET property itself is attributed with markers indicating how the property should appear in the property browser. This is most noticeable in the Visual Studio designer where the category and description readily appear when the activity is highlighted. These attributes are all standard .NET designer attributes and are well documented in the MSDN® library.
Activity binding is one feature enabled by the use of dependency properties. Other features include property promotion, change notification, and state management. Because the values of the properties are stored in a central dictionary, the state of the activity can more easily be serialized. Likewise, this central storage makes it easier to monitor changes to those properties, allowing other activities or workflow code to register for changes and take action when a particular value is modified.
A common question is, when should a developer use dependency properties instead of standard .NET properties? The simple answer is that dependency properties should be used for any property that would benefit from being bound to another property or field. The truth is, there is no good argument for not using dependency properties, and with the code snippet support they are easy to write.

Building Composite Activities
While basic activities provide a great deal of value, it is often the case that you will want to provide more complex reusable components. One type of such a component is the reusable activity that encompasses several basic activities to represent a reusable interaction or flow of logic. For example, an activity that first requests approval from a manager before creating a user would be a composite activity. Wrapped up in this one activity would be the request for permission, receiving the response, and then creating the user. The goal in building this composite activity would be to allow a user to add it to a workflow without having to worry about the complexities of the approval process or how it works.
This simple type of composite activity reuses the execution control of an existing base class, often the SequenceActivity class. This is such a common scenario that the Visual Studio Extension for Windows Workflow Foundation comes with a visual designer for this type of activity. Simply add a new activity to your project and you have the start of a composite activity deriving from the Sequence activity, with the ability to define the steps in your reusable component much as you define a workflow. Using the property browser, you can change the base class to any other composite activity to change the execution semantics but continue to use the designer to build the activity logic. Figure 4 shows the activity designer being used to create a new composite activity based on the Sequence activity.
Figure 4
When you are encapsulating activities, the need arises to expose properties of the contained activities to consumers of your composite activity. In the previous example, the user of the composite activity needs to be able to provide values for the username, e-mail address, roles, and other details required to properly configure the contained activities. The Visual Studio Extensions provide a mechanism referred to as property promotion to make this easier. When using the composite activity designer, if you right-click an activity, the context menu contains an option to promote the bindable properties on the activity. When you choose this option, dependency properties are declared on the composite activity and the properties of the contained activity are bound to those values. This exposes the properties you require to be set without forcing you to write the code to connect the values. You can determine what the exposed property will be called to consumers of your composite activity and continue to hide the implementation details of your activity.
In the case where you want to have more control over the execution of the child activities, you write an activity derived from CompositeActivity and, in the Execute method, handle the scheduling of execution for your child activities. As an example, consider the Switch activity. The IfElse activity available as part of the .NET Framework 3.0 provides the model for conditionally executing a single branch of activities. The Parallel activity provides a model for executing all branches. Only the ConditionedActivityGroup (CAG) activity provides a model for conditionally executing multiple branches. Switch is simpler than CAG and provides a good mechanism for showing how to control the execution of child activities to allow for conditionally executing multiple branches of activities based on rules or code decisions. Figure 5 shows the Switch activity being used in a workflow. Note that each branch of the activity has a property named Condition that can be set to a declarative rule condition or a code condition just like the conditions on a branch of an IfElse activity. Only when the condition is present and evaluates to True does that branch of activity execute.
Figure 5 Conditional Execution with the Switch Activity
Figure 5 Conditional Execution with the Switch Activity
As a composite activity, the Switch activity is made of up a number of sequences to be executed conditionally. Therefore, the first step is to create another activity, the SwitchBranch activity, which derives from the Sequence activity and defines an ActivityCondition property named Condition. It is this property that will allow a user to set the code or declarative rule condition that will indicate if the sequence should execute. The SwitchBranch takes full advantage of its base activity and provides no execution logic of its own.
In the Execute method of the Switch activity each of the enabled child activities is checked and the condition evaluated. If the condition has been set and evaluates to True, then that child branch must be scheduled for execution. This is done by calling the ExecuteActivity method on the ActivityExecutionContext. This is the most common place for composite activities to differ in how they manage the children. Whether you want them to be scheduled in parallel, sequence, or reverse order, the Execute method is where you can identify the schedule and which children should or should not be executed.
The Execute method for the Switch activity returns a status of Executing to the runtime. This indicates that the activity has scheduled its children but it is not finished with its work. This keeps the runtime from executing the next activity until it is notified that this activity has completed. When all of the child activities have completed, the Switch activity must be able to notify the runtime that it is complete. Therefore, before scheduling the child branch for execution, the Switch activity registers for a change notification on the activity so that it knows when the child activity closes. Figure 6 shows the code used to register for this notification. Note that this change notification takes advantage of the ability to monitor changes to dependency properties as noted earlier. The activity must implement the IActivityEventListener interface to get notified of these changes.
Figure 6 Register for Change Notification
foreach(Activity child in EnabledActivities)
{
    SwitchBranchActivity childBranch = child as SwitchBranchActivity;
    if (childBranch != null &&
        childBranch.Condition.Evaluate(childBranch, executionContext))
    {
        childBranch.Closed += OnChildClosed;
        executionContext.ExecuteActivity(childBranch);
    }
}
return ActivityExecutionStatus.Executing;
In the OnEvent method of the IActivityEventListener interface, when the activity is notified that a child branch has closed, it checks to see whether there are any more executing children. If not, then the activity can ask the ActivityExecutionContext to close it. This is the same as returning Closed from the Execute method and lets the runtime know the activity has completed all of its tasks. This is another common place for differences as each activity will have to determine the specific conditions under which it has completed its work.
Each composite activity will likely have different semantics for how it should execute the child activities to meet the needs of the author and the business case. The example shown here is intended to provide insights into the core concerns for an activity developer when creating composite activities. Keep these in mind as you build your activities and consider whether the example applies to your particular case.

Asynchronous and Event-Based Activities
While some activities can complete their work in a simple synchronous manner, there are many cases where an activity will start some work and then wait for the response. Likewise, you may have a need to create an activity that can listen for an external event such as a new file being created or a Windows Management Instrumentation (WMI) event being raised. Because of the way that workflows are managed in the runtime, you should not directly create event handlers for external resources in your activities. For example, if you registered an event handler in your activity for changes to the file system, and the workflow needs to enter an idle, serialized state, your handler goes away with it. Any events raised will then not get to your activity.
Windows Workflow Foundation provides infrastructure to help alleviate this problem of communicating with a workflow that might be currently persisted. The communication model is built on a system of queues: activities generally register to receive messages on a queue and services in the host application send messages on queues. Your custom activities can use this model both to handle external events as well as to communicate completion of asychronous activity execution. This allows your activity to execute up to a point, then wait for stimulus in order to continue executing. Take a look at Figure 7 which depicts the communication model between code in the host application and code or activities in the workflow.
Figure 7 Communicating with Activities Asynchronously
Figure 7 Communicating with Activities Asynchronously
To allow your activity to listen for messages arriving on a queue, you need to first make sure the queue exists. If it does not exist, you must create it. This is usually done in the Initialize or Execute methods of the activity depending on when the queue needs to be available to receive messages. The WorkflowQueuingService provides the methods necessary to create, find, or delete workflow queues. Finally, your activity must register to receive these notifications by registering for the QueueItemAvailable event on the workflow queue itself. Once you have ensured that the queue exists and have registered for events, your activity will be notified when there are items available in the queue and you can then dequeue the item and process it.
When creating an asynchronous activity, you use the steps I've just described to prepare your activity to receive messages on the queue, and when your asynchronous activity completes, you enqueue a message to notify your activity and optionally send it the resulting data. Because the OnEvent method receives a reference to the ActivityExecutionContext by way of the Sender parameter, the activity can be closed by calling the CloseActivity method.
As an example, consider a simple ReadLine activity. In this example, the ReadLine activity would create a queue in its execute method and register for notification of data arriving on the queue-implementing the IActivityEventListener interface to receive said notification. The host application or a custom runtime service would retrieve user input from the console and enqueue it for the activity. An example ReadLine activity is included in the code for this article and uses this pattern as its implementation.
To create an activity that can act as an event sink-like the HandleExternalEvent activity-you also implement the IEventActivity interface. This interface defines the core responsibilities of your activity if it wants to listen for events:
public interface IEventActivity
{
    void Subscribe(ActivityExecutionContext parentContext,
        IActivityEventListener<QueueEventArgs> parentEventHandler);
    void Unsubscribe(ActivityExecutionContext parentContext,
       IActivityEventListener<QueueEventArgs> parentEventHandler);

    IComparable QueueName { get; }
}
The QueueName property must return an IComparable value that can uniquely identify your activity when messages are enqueued. This same queue name will need to be used by the code which enqueues the message to the workflow runtime.
This interface allows the activity to be instructed to subscribe to events before it executes and lets the activity know when to unsubscribe. In the subscribe and unsubscribe methods, it is the responsibility of the activity to ensure that a queue is created using the QueueName and deleted at the end of processing. In addition, this is the opportunity for your activity to register information with any local services which will be executing logic on behalf of the activity and responding by enqueuing a message.
A local service is a class that you define and add to the workflow runtime from the host and which can be leveraged by either your host code, the workflow, or your activities. The local service can maintain event handlers or other listeners as long as the host application is running and then ensure that the appropriate data gets to the workflow by enqueuing the message. The information you pass the local service should include information about the queue name, the workflow instance ID, and any information the service requires to initiate work or return a result to your activity.
The sample code for this article includes a custom event activity that creates a queue and registers a handler to get notified when an event arrives on the queue. The host program simply enqueues an item after a short delay to show how items are enqueued; in a real scenario your local service would most likely be responsible for sending the data, but this shows that the host can also enqueue data for an activity.
The general pattern when working with event-based activities is to check for items on the queue in the Execute method and, if there is no data, then subscribe to items being available on the queue and return a status of executing. Then in the handler for items arriving, read the data off of the queue and optionally raise any events on your activity before closing the activity.
By implementing these interfaces your activity can now participate in the workflow in different ways. For example, in a state machine workflow your activity can now be the event that appears as the first activity in an event driven sequence, allowing your event to trigger a sequence of activities and potentially a transition to a new state. For a complete example of this type of activity, take a look at the FileWatcher activity in the Windows SDK.

Handling Errors
Another responsibility of the activity developer is the handling of error conditions. Error management in activities has many of the same qualities as any other code, but there are a few special considerations for managing the activity execution. When an error occurs in your activity execution logic, you should raise an exception, providing as much detail as necessary just as you would in any other component. This signals the runtime that an error has occurred and it can interact with activities in the workflow to allow them a chance to shut down cleanly or the runtime may invoke special handlers for errors.
An activity's HandleFault method is called anytime an exception is raised from your activity execution code. In the HandleFault method, you should write the code required to free up any resources or cancel any asynchronous operations that your activity may have initiated. Note that the exception will still get raised to the runtime; this method is strictly for allowing your activity to clean up after itself. Composite activities also support FaultHandlers, a special attached child which is executed as a post-processing convenience after the composite activity has transitioned from the faulting state to the closed state. You can use the FaultHandlers view to model the handling of faults that arise from child activities, so making your exceptions specific to the problem can allow for better error-handling code in the workflow.
The default behavior for the HandleFault method as defined in the Activity class is to call the Cancel method on the activity. This centralizes the cleaning up of resources but may not be appropriate in all cases as your activity may have specific resources to clean up depending on the type of exception raised. If you have no specific need to handle resources, perform other logging, or clean up, then the default behaviors will work for you in most cases.
Note that, in general, cancellation is used to implement semantics wherein a composite activity has spawned a number of children and in order to meet its desired semantics it needs to cancel the ongoing execution of some of its children. Cancellation is a positive thing and used to support forward execution, whereas faults are negative and are used to implement exceptional cases. Cancellation and fault handling are two completely different things, and the base class implementation of HandleFault calls the Cancel method simply to enable very basic scenarios.

Design Experience
Many activities are designed to be consumed in a model-based user interface such as the designer included with the Visual Studio Extensions for Windows Workflow Foundation. The design experience for activities is in many ways more important than the execution fundamentals since it is the most visible to other developers and consumers of your activity. Windows Workflow Foundation allows the visualization of workflows in many different scenarios, not just development, so it is important that your activity, not only convey the semantics of its responsibilities, but also that it seems logical and useful to users.
There are several components to the design-time experience, including appearance, interactivity, and validation. These design components are themselves .NET classes and are associated to the activity through custom attributes applied to the activity class. The first such class is the activity designer, which provides the core developer entry point for creating a rich design-time experience. The designer class allows you to participate in the design experience through interaction with the property browser, context menus, mouse clicks and movements, and the actual painting of the activity on the design surface.
You create a designer by creating a class derived from the ActivityDesigner base class or any number of composite activity designers available in the Windows Workflow Foundation libraries. Here's the SwitchActivityDesigner being applied to the Switch activity:
[Designer(typeof(SwitchActivityDesigner),
    typeof(IDesigner))]
public partial class SwitchActivity :
    System.Workflow.ComponentModel.CompositeActivity,
    IActivityEventListener<ActivityExecutionStatusChangedEventArgs>
...
I will focus on several key features that are most often used in custom designers. The first of these features is the ability to provide custom context menu items on your activity to allow the user to initiate some action. By overriding the Verbs property in your designer, you can specify a collection of menu items that should be added to the context menu for your activity. This allows a user to right-click or otherwise bring up the context menu and initiate your action. For each verb that you add to the collection, you provide an event handler to be called when a user clicks the item in the menu. Additionally, you pass in a member of the DesignerVerbGroup enumeration indicating how you want the particular verb grouped in the context menu. This gives you the ability to group related activities by their purpose, such as editing or setting options. In the sample code for the Switch activity, a verb gets added that allows the user to reset all of the conditions on the child branches. Here's the code to add the verb to the context menu:
protected override ActivityDesignerVerbCollection Verbs
{
   get
   {
       ActivityDesignerVerbCollection newVerbs =
            new ActivityDesignerVerbCollection();
        newVerbs.AddRange(base.Verbs);
       newVerbs.Add(new ActivityDesignerVerb(this,
            DesignerVerbGroup.Edit, "Clear all conditions",
           new EventHandler(ClearConditions)));
        return newVerbs;
   }
}
In addition to the Verbs property, you can also override the DesignerVerbs property, which allows you to add verbs to the error context menu to indicate problems with the current configuration. Finally, you can override the SmartTagVerbs property to add items to the smart tag that you will find under the names of most composite activities. This item is usually used only if you need to present an alternate view of your design surface similar to the cancellation and fault handler views.
Probably the most common design-time need for an activity developer is to be able to customize the appearance of an activity on the design surface. Windows Workflow Foundation provides several mechanisms for creating the appearance you want, from very simple options where you set only a few properties, to the most robust that allow you to paint the activity directly using a System.Drawing.Graphics object. I will start by explaining the simplest options, which will suffice in many situations.
The first point to understand about the appearance of a workflow activity is that it takes part in what is known as a designer theme. A theme is simply a definition of what each activity should look like in the designer. If you have the Visual Studio Extension for Windows Workflow Foundation installed and select Options from the Tools menu, you will see a category for the Workflow Designer that shows two installed themes. Click the New button and you can get sense of the properties available for each activity. You can save a theme and apply it in the designer in Visual Studio, but you can also use it if you rehost the workflow designer control in your own application. Using themes, you can completely change the workflow design surface and each activity hosted in the designer.
The default theme for the designer inspects an activity and allows it to provide theme information. In your activity development, you accomplish this by associating a custom DesignerTheme with your custom designer using an attribute. In the constructor for your designer theme, you can then set values for properties such as BackColorStart, BackColorEnd, ForeColor, and BackgroundStyle to change the coloring of the activity and its text. CreateUserActivityDesignerTheme provides an example of updating these values:
public class CreateUserActivityDesignerTheme : ActivityDesignerTheme
{
    public CreateUserActivityDesignerTheme(WorkflowTheme theme)
        : base(theme)
    {
        this.BackColorEnd = Color.BurlyWood;
        this.BackColorStart = Color.Bisque;
        this.BackgroundStyle = LinearGradientMode.ForwardDiagonal;
        this.ForeColor = Color.Black;
 }
}
In addition to the color and design, you can specify an icon to be drawn in the default position on the activity. The icon is connected to the activity using the ToolboxBitmap attribute and specifying an image file (a 16×16 bitmap or PNG file generally) that is built into the assembly as an embedded resource. In addition to being used in the designer, this same image is shown when the activity is added to the toolbox manually or by an installer, which makes sense given the name of the attribute. Figure 8 shows the CreateUser activity within a workflow showing the theme and icon support.
Figure 8 Custom Theme and Icon for an Activity
Figure 8 Custom Theme and Icon for an Activity
When creating designers for composite activities, in addition to dealing with issues of themes and icons, you may also have to draw connectors between your child activity designers or otherwise change the appearance of how you contain those children. However, there are several base classes that may provide the functionality needed to reduce the amount of code you have to write. For the Switch activity, for example, the designer derives from the ParallelActivityDesigner, which provides for drawing each of the child activities as branches and creates a designer verb to add a new branch. The Switch activity designer just has to override a method and return a new SwitchBranchActivity. This designer can prove useful for any composite activities you have with branches and is used internally by the IfElse, Parallel, and Listen activities.
The base class for the ParallelActivityDesigner class is the StructuredCompositeActivityDesigner class, which provides much of the rendering logic for connectors when your composite activity contains other designers. A sibling class to StructuredCompositeActivityDesigner is FreeFormActivityDesigner. This designer is the basis for the StateMachine workflow designer and can be used when the child activities can be freely placed in your composite activity as opposed to structurally placed. Use one of these two designers as the basis for your own.
One final note: when you create a composite activity using the activity designer, the activity will show up on the design surface rendering all of its child activities. Thus, a consumer of your composite activity will see all of the internal activities in your implementation. This may be your intention, but if not you can overcome this by creating a designer for your composite activity that derives from the base ActivityDesigner class. This provides the simple rendering of a single activity. You can attach a designer theme and ToolboxBitmap attributes to further customize the look, providing true encapsulation.

Validation
When creating an activity, or any component for that matter, it is good practice to validate the input of the user and make sure your component is configured correctly. In fact, it would be even better if you could stop the user from incorrectly configuring your component in the first place. There are two key areas of development where you get the chance to make sure the activity is being configured correctly: in the designer and with a custom validator for your activity.
When creating a custom designer for your activity you control the activities that can be added to your composite activity, or the activities to which your activity can be added. It is important to note that this behavior only applies to the design environment and does not come into play if your activity is added through code.
Overriding the CanBeParentedTo method on the base ActivityDesigner class allows you to filter the types of activities your activity may be added to. If you return false from this method, users see a red slash, the international symbol for No, indicating that they cannot add your activity. The SwitchBranch activity, for example, will only allow itself to be parented to a SwitchActivity, ensuring that it is only run in the correct context. Overriding the CanInsertActivities method from the CompositeActivityDesigner class lets you control whether a particular activity or set of activities may be added to your composite activity. Using these two methods gives you the power to control the activity structure related to your activity. This becomes very important when you begin looking at the execution needs of your activity as it provides the opportunity to ensure you are set up for successful execution.
The other option for validating your activity is to create a custom validator. The validator class is responsible for inspecting your activity and making sure it is configured according to your requirements, and this validation occurs regardless of whether you are using a designer to add your activity into a workflow. By this point you can probably guess how to create a validator: you create a class that derives from ActivityValidator and then override the Validate method. You then use the ActivityValidatorAttribute to connect your validator to your activity.
The CreateUser activity has a property to hold the password. However, it's not a good idea to configure the password statically in the workflow. Instead, the validator component ensures that the password property is bound to some other value so that its value is set at runtime. Figure 9 shows the CreateUserValidator class testing the property to make sure it is bound.
Figure 9 CreateUserValidator
public override ValidationErrorCollection Validate(
    ValidationManager manager, object obj)
{
    ValidationErrorCollection baseErrors = base.Validate(manager, obj);

    CreateUserActivity user = obj as CreateUserActivity;
    if (user == null) throw new ArgumentException(
        "Activity must be a CreateUser activity");

    // make sure we are in a valid designer experience
    if (user.Parent != null &&
        !user.IsBindingSet(CreateUserActivity.PasswordProperty))
    {
        baseErrors.Add(new ValidationError(
            "Password must be set using activity binding",
            1, false, "Password"));
        baseErrors.Add(new ValidationError("Oops", 2, true));
    }
    return baseErrors;
}
The code first validates that the activity being validated is of the correct type and that it is being validated in a context other than its own compilation. If you do not check the parent property for null, then you will not be able to build your activity in the first place as the validator will run when your activity is compiled. The ValidationErrorCollection is used to pass back a collection of errors to the designer so that the appropriate error messages can be added to the user interface.

Conclusion
As you can see, Windows Workflow Foundation provides a rich environment for you to build reusable, easily designed activities for use in your workflows, whether they're leaf activities (key to implementing domain specific logic) or composite activities (key to creating control flow patterns reflective of real world interactions between entities). In fact, Windows Workflow Foundation does not treat any of its out-of-the-box activities any differently than the custom activities you write.
From the activity execution to the design experience, you have access to the same resources as the Microsoft team that built the base activity library. Building robust activities enables you to expose your applications and services to the processing model of Windows Workflow Foundation and to take advantage of all the services it has to offer. I encourage you to take advantage of the community Web site (wf.netfx3.com) to find activities and submit your own activities when you build them.?

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. As an instructor for Pluralsight, Matt teaches courses on Workflow, BizTalk Server, and Windows Communication Foundation. Matt lives in Minnesota with his wife Kristen and his two sons. Contact Matt via his blog.


Extend ASP.NET
Simplify Data Binding In ASP.NET 2.0 With Our Custom Control
Rick Strahl

This article discusses:
  • Two-way data binding for ASP.NET
  • Implementing custom extender ASP.NET controls
  • Improving designer support for extender controls
  • Handling and displaying data binding errors consistently
 This article uses the following technologies:
ASP.NET 2.0, Visual Studio 2005
Contents
The Problem with Stock Data Binding
Introducing a Two-Way Data Binding Control
Implementing an Extender Control
Workaround for Designer Notification
How DataBinding and Error Management Works
Handling Binding Errors
Wrapping Up
ASP.NET 2.0 includes many new data binding features that can significantly simplify a variety of data binding scenarios. Unfortunately, one very common scenario-simple control data binding-has changed very little. While this is a common scenario in forms-based Web applications, it remains for the most part a manual and time-consuming process.
Simple control data binding refers to the process of binding a single value to a property of a control-for instance, binding form controls like textboxes, checkboxes, radio buttons, or selected values of list controls to individual data or object values. This could benefit significantly from a consistent approach to handling data binding deterministically, as well as managing error trapping and display consistently. In this article, I introduce a custom extender control that can be used to bind any individual value from data or objects to any control property. It can extend and provide designer support for any existing control on a Web Form.
In the process, I'll demonstrate a flexible and intuitive approach that provides consistent behavior for data binding and unbinding, validation, error handling, and error display in forms-based Web interfaces. I'll also dig into the details of how the control is built and describe some of the interesting benefits.
The wwDataBinder control addresses simple control data binding. In more concrete terms, the control manages bindings that express concepts such as "bind the value of the DataRow's CompanyName field to the TextBox's Text property" or "bind the Item.Entity.Pk property to the SelectedValue property of a DropDown list." The control manages the process of binding the specified data values to the specified control property, as well as unbinding the value back into the underlying data item. This is true two-way data binding.

The Problem with Stock Data Binding
ASP.NET natively provides you with reasonable support for inbound data binding. Its native data binding expressions use <%# Eval("FieldName") %> or direct page level expressions like <%# this.Item.Entity.Description %> to bind data to control properties. You call the DataBind method on the page or any control of the page to initiate the data binding process. This works well for inbound binding.
However, unbinding and getting the data back into the underlying data fields or properties is not as readily available. ASP.NET 2.0 introduces the GridView, FormView, and DetailsView controls, which let you use <%# Bind("FieldName") %> syntax that provides two-way data binding. Unfortunately, these controls are closely tied to an underlying data source control, which must be IEnumerable-based. So binding to an individual entity object, for example, is not supported.
Perhaps the most serious problem with the native unbinding mechanisms is the way that unbinding errors are handled. If there's a problem with the formatting of an input field-say a numeric field that contains an alpha character or a DateTime value that sets a month of 22-you get a page-level exception when you try to save the value. While you can trap the exception in page-level error handling, you can't summarize this information because the first exception will stop the unbinding process. If you have two or three errors on the page, it will take multiple post backs to handle these errors, which is very unattractive from a UI perspective (though you could use validators to prevent, rather than handle, the problem).

Introducing a Two-Way Data Binding Control
I created the wwDataBinder control to manage simple data binding in a more flexible (and I hope more logical) way. The control provides a number of handy features and capabilities:
  • Support for deterministic two-way data binding
  • Binding single object or data values to a control property
  • Unbinding control properties to object or data values
  • Handling data conversion from text to typed data
  • Managing any binding errors in a BindingErrors collection
  • Support for custom validation events
  • Checking for required values in input
  • Displaying error icons next to controls
  • Showing message summary with links to controls
  • Adding binding errors programmatically
To use the control, you simply drop it into an ASP.NET page. The control acts as an extender control for any existing controls on the form, and it adds a DataBindingItem property to the extended control in the Visual Studio® 2005 Properties editor (as shown in Figure 1).
Figure 1 Extending Controls with Data Binding Functionality
Figure 1 Extending Controls with Data Binding Functionality
The control sits on the form as a nondescript grey container (you can see this at the bottom of Figure 1), which isn't rendered at run time. It's displayed on the form merely so you can set properties on the wwDataBinder control itself.
As an extender control, wwDataBinder contains a collection of DataBindingItems that hold the configuration values for each of the extended controls. The collection of items is maintained in the HTML markup and the control looks like this:
<ww:wwDataBinder ID="DataBinder" runat="server"
           OnValidateControl="DataBinder_ValidateControl">
   <DataBindingItems>
      <ww:wwDataBindingItem runat="server"
        ControlId="txtProductName"
        BindingSource="Product.Entity"
        BindingSourceMember="ProductName" IsRequired="True"
        ErrorMessageLocation="RedTextAndIconBelow">
      </ww:wwDataBindingItem>
      ... more DataBindingItems
   </DataBindingItems>
</ww:wwDataBinder>
The collection establishes the individual bindings that are applied by the DataBinder. While wwDataBinder extends existing controls, nothing is actually extended until you explicitly set properties on the DataBindingItem extender property of any given control on the form.
In design mode, the extender exposes a DataBindingItem property-this is for providing a mapping between a data item and a control property. Supported data items are ADO.NET data objects, such as ADO.NET DataRow fields. Alternatively, you can bind to an object and any simple property of an object that can be referenced through the Page object. It's possible to bind this.Customer.Entity.CompanyName, where this refers to the Page object (which is the base object from which any binding expression originates).
Binding is performed using a BindingSource to identify a data item. The BindingSource consists of two properties-BindingSource (which is a string property that describes the base object) and BindingSourceMember (which provides the mapping of the field or property to bind to). Programmatic access can also make use of the BindingSourceObject property to provide a reference instead of a BindingSource string.
Suppose you want to bind to a DataRow object. In order for wwDataBinder to bind to the DataRow, the DataRow needs to be accessible through a Page reference. So this.MyDataRow works. In this case, you'd bind with:
BindingSource: MyDataRow
BindingSourceMember: CompanyName
As shown here, CompanyName is the name of the field to which you should bind.
Also, this.Customer.DataRow will work, where Customer is a business object that has a DataRow property. Here, you'd bind with:
BindingSource: Customer.DataRow
BindingSourceMember: CompanyName
You can bind to a DataTable in the same way. When a DataTable or DataSet is used as the BindingSource object, wwDataBinder assumes the first DataRow for a table or the first table and first DataRow for a DataSet. You can also bind an ADO.NET object more directly, like so:
BindingSource: MyDataSet.Tables["Products"].Rows[0]
BindingSourceMember: CompanyName
Just remember that in order for the wwDataBinder to be able to bind, whatever object you're binding to must be referenced through a property of the Page that is marked as protected or public. Public is preferred since public reflection is supported in Medium Trust, whereas reflection will fail with protected properties. Private properties are not supported for binding.
In addition to ADO.NET types, you can also bind to business entities or types from the .NET Base Class Library (BCL). For example, if you have a business object that lets you access CompanyName with this.Customer.Entity.CompanyName, you can express the BindingSource as:
BindingSource: Customer.Entity
BindingSourceMember: CompanyName
You can also bind directly to Page object members. If you have a DateTime RightNow property on the Page then binding looks like the following:
BindingSource: this
BindingSourceMember: RightNow
Any BindingSource implicitly uses this or me to signify the Page object. But unless you are using the Page itself as the binding source, these are optional and don't need to be specified. A BindingSource is relative to a top-level container object-typically the ASP.NET Page object-but you can supply the top-level container as part of the DataBind and Unbind methods as a parameter. This is useful if you want to use the wwDataBinder in a user control where the top-level container is the UserControl rather than the page.
BindingSourceMembers are always simple types like ints, strings, DateTime, bool, and so on. They can also be Enum values, which are translated into strings and work for two-way binding.
The job of a BindingSource is to establish a binding between a data item or property and the control property to which the value is bound. This mapping is used both for binding to a control property and from a control property back into the data item or property. Inbound binding is established by calling the wwDataBinder.DataBind method, and unbinding is done by calling wwDataBinder.Unbind. The process is deterministic in that you explicitly call these methods from your code-this provides a lot of flexibility in terms of how binding is handled.
A DataBindingItem exposes additional properties that are associated with the data binding process, as shown in Figure 1. The BindingProperty determines the field of the control you are binding to-the default is Text, but if you're binding to a CheckBox, you likely bind to Checked or if you're binding to a list control to SelectedValue.
Data binding can be one-way, two-way, or none. Input controls like textboxes generally use two-way binding, while display controls like labels use one-way binding. None is useful if you want a control to participate in binding error management, but you don't want to bind the control to a value.
The wwDataBinder control automatically manages both binding and unbinding errors. When a control cannot be bound or unbound, the control generates a BindingError in the BindingErrors collection, but no exception fires. You can query this collection for individual binding errors and retrieve a summary of errors with the ToHtml method. By default, each control with an error also displays an error icon next to it. Figure 2 shows an example of the form with a few binding errors.
Figure 2 Displaying Binding Errors
Figure 2 Displaying Binding Errors
BindingErrors can either be automatically generated by the control during the binding or unbinding process or can be manually added to the BindingErrors collection from your own code. Automatic BindingErrors are generated when an input value cannot be converted back to the bound data item or property, usually due to a data conversion error. For example, the Reorder Expected date shown in Figure 2 is an invalid date and the error is automatically captured by the control since the date conversion fails. Each DataBindingItem also has an IsRequired property and, not surprisingly, an item marked as IsRequired cannot be left blank. If one is left empty, a binding error is automatically generated and added to BindingErrors.
The other two errors on this form are manual errors. The Dairy Products error is handled through a ValidateControl event that is hooked up to the wwDataBinder. The event fires for each control bound and the event handler allows for checking each control after it has been unbound. If you decide a value is not valid in your code, you can set the BindingItem's BindingErrorMessage programmatically and return false to let the binder know that this DataBindingItem is in error and needs to display on the error list.
The last error is a purely programmatic error that is added by using wwDataBinder.AddBindingError and specifying the error message and Control instance or ID in the Page code. This allows assigning an arbitrary error message to a control that is data bound and makes the control show up both with the icon and in the error listing.
All of this provides for automatic and fully programmatic binding error generation so there's lots of flexibility on how errors and error icons are displayed. In addition, the control automatically picks up any Validator controls on the page and picks up their error messages and control IDs and incorporates them into the binding error mechanism.
If a control has an error, an icon is displayed next to it by default. The icon display is optional and you can turn off the warning icons for the DataBinder as a whole, or you can change the location and format for each of the individual DataBindingItems through the ErrorMessageLocation property. By default, a WebResource is used-it points to an embedded image resource in the compiled control assembly. Or you can override the image explicitly with the ErrorIconUrl property on wwDataBinder.
The assembly also includes a wwErrorDisplay control, which is used to display the error summary shown in Figure 2. This control is optional, but I like to use it for consistency. In the example, I have simply assigned the BindingErrors.ToHtml method's string output to the text property. The control also has easy message display methods, like ShowError and ShowMessage, which display one-line messages with an icon, providing a consistent way to display messages on a Page.
You'll notice that the error summary has links for each error displayed. These links jump to the individual control that is in error and highlight it, making it easier to spot and fix errors.
Let's take a look at the code required to run the form I've already shown you. The pattern used with the wwDataBinder control works something like the following. To bind, you hook up control bindings, mapping data to control properties. You then load your data into page level object references and call the DataBind method to bind the controls on the form.
To unbind, first make sure the bound data objects are available and loaded. Call the Unbind method to unbind the control values into the data. Check for BindingErrors, handle any custom server side validation, and display any binding and validation errors on failures. Finally, save the data.
My example uses a few business objects that load internal DataSet objects to keep the sample code short and concise, so this code isn't littered with SQL syntax. The business objects return their data in DataSets. This is convenient for the examples here because I can bind the controls directly to these data items. Figure 3 shows the full page code for the sample.
The sample starts off in OnInit loading up the various lists displayed on the form by calling the LoadLists method. Note that the item, category, and supplier listings are all using standard list-based binding for list content, and this code is unrelated to the wwDataBinder.
The Item business object is the main object in the Page. I'll bind to an individual record that can be expressed either as this.Item.DataRow or this.Item.Entity, where DataRow is a standard ADO.NET DataRow and Entity is a simple managed object with single properties for each field. (The download for this article available on the MSDN®Magazine Web site provides two sample forms-one that binds against the DataRow and one that binds the Entity.)
In the Page_Load of the first page access, the first product in the dropdown list is selected by and used for the Item.Load method. Item.Load takes a primary key parameter for a ProductId and loads up a DataTable/DataRow with a single record. The business object that provides this functionality exposes an Item.DataRow property. Item.DataRow is the main BindingSourceObject on this form, so controls bind to a BindingSource Item.DataRow and BindingSourceMember of ProductName, StockOnHand, and so on. After the item is loaded, you can data bind the controls with the data from the DataRow, like this:
this.Item.Load(FirstItemPk);
this.DataBinde r.DataBind();
this.DataBinder contains a collection of DataBindingItems, which now bind the controls of the form.
The DataBinder loops through the collection of DataBindingItems and calls each individual DataBind method, binding the data item to the specified control property. Note that some controls on the form are dropdown list controls that are filled with list binding and are also bound by the DataBinder, which binds the SelectedValue to select an item in the list.
The data binding happens only when IsPostBack is false so that values are not rebound after a user has changed the value and saved the changes. The form is rebound only when a new item selection is made in the txtProductId_SelectedIndexChanged event handler, which again loads a new item and then DataBind on the DataBinder property to rebind all the controls.
If you have controls that do not post their values back-labels or other display-only controls-you can also bind individual items by explicitly calling DataBinder.GetDataBindingItem(Controlid).DataBind. Each DataBindingItem performs its own binding and unbinding, and this method call triggers the explicit binding.
At this point, the page is bound. The user can now make changes. When it's time to save the data from the page, you can unbind the control values back into the underlying binding sources. In my example, most controls bind back in the Item.DataRow object, but you can easily bind to multiple objects with a single DataBinder. Each DataBindingItem determines its own BindingSource.
The first step to unbind is to ensure the object you are binding back to is in the proper context. In my example, you need to make sure that the business object has the data for the current item loaded. And since you only bound the form on the first hit (!PostBack), you now have to reload the data so the DataBinder has something to bind back to. Once the item is loaded, you can call the DataBinder's Unbind method to retrieve the control values back into the data item or property. When the call returns, the properties have been unbound into the DataRow. Note that the data has not been saved yet-unbinding only restores the data values, it doesn't bind right back into the database.
Before the data can be saved, you should first check for BindingErrors and any validation errors in the business object. BindingErrors are automatically generated if there are invalid values in controls and when the DataBinder cannot convert the entered values back into their strongly typed counterparts. BindingErrors are also created when IsRequired is set to true on a field and the field is then left blank. Finally, BindingErrors are also created if you create custom validation handlers and the DataBinder_Validate method returns false for any DataBindingItem.
The typical steps for unbinding boil down to the following minimal code:
Item.Load(ItemPk);
DataBinder.Unbind();

if (DataBinder.BindingErrors > 0)
{
    ErrorDisplay.Text = DataBinder.ToHtml();
    return;
}

Item.Save();
The sample includes a bit more code to demonstrate BindingErrors related features-adding binding errors from custom code and even from business object validation. Pay close attention to the DataBinder.AddBindingError method, which allows you to add BindingErrors to the control so that you can have a consistent error display mechanism, both for binding errors as well as for application-generated errors. AddBindingError requires that the control is already bound to the DataBinder, but you can also dynamically add a control to the binder at run time and then add a binding error, as shown with the txtProductId code in Figure 3. In that case, use AddBinding to programmatically add a binding then call AddBindingError to attach an error message.
Figure 3 Sample Form Using wwDataBinder Control
public partial class NorthwindInventoryData : System.Web.UI.Page
{
    public busItem Item = null;
    public busCategory Category = null;
    public busSupplier Supplier = null;

    public DateTime RightNow = DateTime.Now;

    bool PageError = false;

    protected override void OnInit(EventArgs e)
    {
        base.OnInit(e);

        this.Item = new busItem();
        this.Category = new busCategory();
        this.Supplier = new busSupplier();

        // *** This content goes into ViewState
        if (!this.IsPostBack) this.LoadLists();
    }

    protected void Page_Load(object sender, EventArgs e)
    {
        if (this.PageError) return;

        // *** I only databind on initial load
        if (!this.IsPostBack)
        {
            int FirstItemPk = (int)Item.DataSet.Tables[
                "TItemList"].Rows[0]["ProductId"];

            if (!this.Item.Load(FirstItemPk))
            {
                this.ErrorDisplay.ShowError(Item.ErrorMessage);
                this.Item.New();
            }

            // *** Go do it: Bind the Binding Items to the Controls
            this.DataBinder.DataBind();
        }
    }

    protected void btnSubmit_Click(object sender, EventArgs e)
    {
        // *** Reload Item record to override only data that's changed
        if (!this.Item.Load(int.Parse(this.txtProductId.SelectedValue)))
        {
            this.ErrorDisplay.ShowMessage("Invalid product id selected");
            return;
        }

        // *** Unbind Controls into the Item.DataRow
        bool IsError = this.DataBinder.Unbind(this);

        // *** Check for Business Rule validation Errors
        if (!Item.Validate())
        {
            // *** Add any Validation Errors to BindingErrors
            foreach (ValidationError Error in Item.ValidationErrors)
            {
                DataBinder.AddBindingError(
                    Error.Message, Error.ControlID);
            }
        }

        // *** Manually add a binding error to a non-bound control
        if (this.txtProductId.SelectedIndex == 0)
        {
            // *** If the item isn't already bound, I can create a binding
            // *** if already bound, this isn't necessary
            this.DataBinder.AddBinding(this.txtProductId);

            // Assign a binding error to it
            this.DataBinder.AddBindingError(
                "First Item in list can't be " +
                "updated (manual binding)", this.txtProductId);
        }

        // *** If there are BindingErrors display them
        if (DataBinder.BindingErrors.Count > 0)
        {
            this.ErrorDisplay.Text = DataBinder.BindingErrors.ToHtml();
            return;
        }

        if (!Item.Save())
            this.ErrorDisplay.ShowError(Item.ErrorMessage,
                "Couldn't save Item");
        else
            this.ErrorDisplay.ShowMessage("Inventory saved.<br />" +
                "Updated saved Time: " + this.RightNow.ToString());
    }

    protected void txtProductId_SelectedIndexChanged(object sender,
        EventArgs e)
    {
        int Pk = -1;
        int.TryParse(this.txtProductId.SelectedValue, out Pk);
        if (Pk == -1)
        {
            this.ErrorDisplay.ShowError("Invalid Selection");
            return;
        }

        // *** Rebind the item after change was made
        this.Item.Load(Pk);
        this.DataBinder.DataBind();
    }

    protected bool DataBinder_ValidateControl(wwDataBindingItem Item)
    {
        if (Item.ControlInstance == this.txtCategoryId)
        {
            DropDownList List = Item.ControlInstance as DropDownList;
            if (List.SelectedItem.Text == "Dairy Products")
            {
                // *** Add a BindingErrorMessage to the DataBindingItem
                Item.BindingErrorMessage =
                    "Dairy Properties are bad for you";

                // *** Return false to tell that validation failed
                return false;
            }
        }

        return true;
    }

    protected void LoadLists()
    {
        if (this.Item.GetItemList("TItemList") < 0)
        {
            this.ErrorDisplay.ShowError(Item.ErrorMessage,
                                  "Couldn't load items");
            this.PageError = true;
            return;
        }

        if (this.Category.GetCategoryList("TCategoryList") < 0)
        {
            this.ErrorDisplay.ShowError(Item.ErrorMessage,
                "Couldn't load categories");
            this.PageError = true;
            return;
        }
        if (this.Supplier.GetSupplierList("TSupplierList") < 0)
        {
            this.ErrorDisplay.ShowError(Supplier.ErrorMessage,
                "Couldn't load suppliers");
            this.PageError = true;
            return;
        }

        // *** Bind the lists
        this.txtCategoryId.DataSource =
           Category.DataSet.Tables["TCategoryList"];
        this.txtCategoryId.DataTextField = "CategoryName";
        this.txtCategoryId.DataValueField = "CategoryId";
        this.txtCategoryId.DataBind();

        this.txtProductId.DataSource = Item.DataSet.Tables["TItemList"];
        this.txtProductId.DataTextField = "ProductName";
        this.txtProductId.DataValueField = "ProductId";
        this.txtProductId.DataBind();

        this.txtSupplierId.DataSource =
            Supplier.DataSet.Tables["TSupplierList"];
        this.txtSupplierId.DataTextField = "CompanyName";
        this.txtSupplierId.DataValueField = "SupplierId";
        this.txtSupplierId.DataBind();
    }
}

        this.Item.Load(Pk);
        this.DataBinder.DataBind();
    }

    protected bool DataBinder_ValidateControl(wwDataBindingItem Item)
    {
        if (Item.ControlInstance == this.txtCategoryId)
        {
            DropDownList List = Item.ControlInstance as DropDownList;
            if (List.SelectedItem.Text == "Dairy Products")
            {
                // *** Add a BindingErrorMessage to the DataBindingItem
                Item.BindingErrorMessage =
                    "Dairy Properties are bad for you";

                // *** Return false to tell that validation failed
                return false;
            }
        }

        return true;
    }

protected void LoadLists()
    {
        if (this.Item.GetItemList("TItemList") < 0)
        {
            this.ErrorDisplay.ShowError(Item.ErrorMessage,
                                  "Couldn’t load items");
            this.PageError = true;
            return;
        }

        if (this.Category.GetCategoryList("TCategoryList") < 0)
        {
            this.ErrorDisplay.ShowError(Item.ErrorMessage,
                "Couldn’t load categories");
            this.PageError = true;
            return;
        }
        if (this.Supplier.GetSupplierList("TSupplierList") < 0)
        {
            this.ErrorDisplay.ShowError(Supplier.ErrorMessage,
                "Couldn’t load suppliers");
            this.PageError = true;
            return;
        }

        // *** Bind the lists
        this.txtCategoryId.DataSource =
           Category.DataSet.Tables["TCategoryList"];
        this.txtCategoryId.DataTextField = "CategoryName";
        this.txtCategoryId.DataValueField = "CategoryId";
        this.txtCategoryId.DataBind();

        this.txtProductId.DataSource = Item.DataSet.Tables["TItemList"];
        this.txtProductId.DataTextField = "ProductName";
        this.txtProductId.DataValueField = "ProductId";
        this.txtProductId.DataBind();

        this.txtSupplierId.DataSource =
            Supplier.DataSet.Tables["TSupplierList"];
        this.txtSupplierId.DataTextField = "CompanyName";
        this.txtSupplierId.DataValueField = "SupplierId";
        this.txtSupplierId.DataBind();
    }
}
The sample in Figure 3 uses a business object and binds to a DataRow. There's another example in the code download that instead binds to the business object using an Entity object. So rather than binding to the Item.DataRow it binds to Item.Entity where Entity is a simple .NET object that has properties to match the properties of the Item table. The process doesn't change at all; only the Data source string changes to Item.Entity instead of Item.DataRow. The wwDataBinder control figures out at run time which type it is dealing with and based on that binds and unbinds appropriately. If you're not using business objects, wwDataBinder will also work just fine directly against ADO.NET objects.

Implementing an Extender Control
wwDataBinder is implemented as an extender control, which means that it extends the functionality of existing controls on a form. Extender controls are a good mechanism to provide new functionality to existing controls without requiring subclassing. I've used a data binding scheme for many years that is similar to the one I've described in this article. But in the past, I used control subclasses to implement this functionality-I had wwTextBox, wwCheckBox, and wwDropDownList subclasses (and so on) and each of them needed to implement a special data binding interface that provided roughly the same functionality described here. This works fine, but it's not great for extensibility since it relies on adding code to every control that you want to extend. This results in unwelcome code manageability and code duplication issues.
An extender control is much more flexible in that it works with any existing control. Extenders drop onto a form as a single control and then have a collection of items that each track a specific control of the form with extended properties. In the case of the wwDataBinder, the extender property is a wwDataBindingItem object that contains the properties required to bind and unbind values to a control property. If you look again at Figure 1, the DataBindingItem property that is highlighted is the extension property that is provided by the wwDataBinder extender. While designer support for extender properties is nice, it is by no means required and that may be a good thing since designer support for extender controls in the Visual Studio Web designer is tricky to implement. (I'll go into that more in a bit.)
Figure 4 shows the wwDataBinder class hierarchy. wwDataBinder is the top-level control, which contains a collection of wwDataBindingItem objects. Each of these maps to a control instance on a form via the ControlId property. The ControlId property, in turn, holds the ID for the control that is tracked and provides the base reference for finding the control on a form.
Figure 4 wwDataBinder Extender Class Hierarchy
Figure 4 wwDataBinder Extender Class Hierarchy
The idea of the data binding extender is that data binding items are added to the collection. The collection can be filled in a variety of ways: using the Extender DataBindingItem property in the property sheet, using the stock Collection Editor on the DataBinder, using HTML markup for the wwDataBinder control, or calling wwDataBinder.AddBinding at run time.
The extender mechanism is the easiest to use, but it requires some extra implementation code on the control to provide this designer integration. The stock collection editor is also available to add items to the DataBinder and is available simply because the control implements a Collection property and provides a few attributes on the main class and the collection property:
[ProvideProperty("DataBindingItem", typeof(Control))]
[ParseChildren(true, "DataBindingItems")]
[PersistChildren(false)]
[NonVisualControl, Designer(typeof(wwDataBinderDesigner))]
public class wwDataBinder : Control, IExtenderProvider
{
    [PersistenceMode(PersistenceMode.InnerProperty)]
    public wwDataBindingItemCollection DataBindingItems
    {
        get  { return _DataBindingItems; }
    }
    wwDataBindingItemCollection _DataBindingItems = null;

    ...
}
This tells the designer to persist the collection of DataBindingItems as a property, which are automatically parsed and added to the collection when the designer loads the page. This also enables the default collection editor for the sub-controls as shown in Figure 5. All of the designer entry mechanisms results in Visual Studio persisting the output into the HTML markup shown in Figure 6.
Figure 6 Control Markup for a Variety of Binding Sources
<ww:wwDataBinder ID="DataBinder" runat="server"
                 OnValidateControl="DataBinder_ValidateControl">
    <DataBindingItems>
       <ww:wwDataBindingItem runat="server"
          BindingSource="Item.DataRow"
          BindingSourceMember="ProductName"
          ControlId="txtProductName" IsRequired="True"
          ErrorMessageLocation="RedTextAndIconBelow">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingProperty="SelectedValue"
          BindingSource="Item.Entity"
          UserFieldName="Category" BindingSourceMember="CategoryId"
          ControlId="txtCategoryId">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingSource="Item.DataRow"
          BindingSourceMember="UnitPrice"
          ControlId="txtUnitPrice" DisplayFormat="{0:f2}"
          IsRequired="True" UserFieldName="Unit Price">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingProperty="Checked"
          BindingSource='Item.DataSet.Tables["Products"].Rows[0]'
          BindingSourceMember="Discontinued"
          ControlId="chkDiscontinued"
          UserFieldName="Discontinued">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingSource="this" BindingSourceMember="RightNow"
          ControlId="txtRightNow" UserFieldName="Right Now">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingSource="Item.Entity"
          BindingSourceMember="UnitsInStock"
          ControlId="txtUnitsInStock"
          DisplayFormat="{0:f0}">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          ControlId="txtSupplierId" BindingSource="Item.DataRow"
          BindingSourceMember="SupplierId"
          BindingProperty="SelectedValue">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingSource="Item.DataRow"
          BindingSourceMember="ReorderLevel"
          ControlId="txtReorderLevel">
       </ww:wwDataBindingItem>
       <ww:wwDataBindingItem runat="server"
          BindingSource="Item.DataRow"
          BindingSourceMember="ReorderExpected"
          ControlId="txtReorderExpected"
          DisplayFormat="{0:d}" UserFieldName="Reorder Expected">
       </ww:wwDataBindingItem>
    </DataBindingItems>
 </ww:wwDataBinder>
Figure 5 Stock Collection Editor
Figure 5 Stock Collection Editor
Of course, you can also manually enter this markup into the ASPX page so no designer support is necessary at all. However, while both manual markup and the collection editor work fine, entering the Control IDs manually and remembering which controls need to be data bound is a bit tedious. The problem is that the control to be extended is completely separate from the DataBinder. It's important to understand that regardless of their differences, all three mechanisms essentially end up with the same end result, which is the ASP.NET control markup in Figure 6. The markup is the persistence mechanism for the control.
Providing a designer extender provider is by far the best user experience, as it shows the data binding information in context with the control it extends. Realistically, without the designer support, I don't think this control would be user-friendly enough to be used efficiently. Creating this support in ASP.NET controls is tricky because the Web designer doesn't support extender controls as nicely as does the Windows® Forms designer. Specifically, the Web designer doesn't automatically generate startup code for the controls being extended and it doesn't call the Set<extended property> method when the property is updated. So a little bit of extra work is required to build an extender control that works with the Web designer. Let's take a look at how you do this.
Implement IExtenderProvider First, implement the IExtenderProviderInterface on the extender control so that Visual Studio can recognize the control as an extender. This interface consists of a single CanExtend method that returns true or false:
public bool CanExtend(object extendee)
{
   if (!this.IsExtender) return false;

   // *** Don't extend ourself <g>
   if (extendee is wwDataBinder) return false;

   if (extendee is Control) return true;

   return false;
}
Mark the Class with the ProvidePropertyAttribute The class should also be marked up with a ProvideProperty attribute for each property with which you want to extend controls. In this case, wwDataBinder extends with a single complex property called DataBindingItem:
[ProvideProperty("DataBindingItem", typeof(Control))]
Implement Get Method for the Provided Property The designer calls a Get<extender property name> method to get an instance of the property that will be provided to the control. The code for this method should retrieve either an existing instance or create a new blank instance if it can't be matched with a control:

public wwDataBindingItem GetDataBindingItem(Control control)
{
    foreach (wwDataBindingItem Item in this.DataBindingItems) {
        if (Item.ControlId == control.ID) return Item;
    }
    wwDataBindingItem NewItem = new wwDataBindingItem(this);
    NewItem.ControlId = control.ID;
    NewItem.ControlInstance = control;
    this.DataBindingItems.Add(NewItem);
    return NewItem;
}
In theory, you should also implement a Set method. But, unfortunately, the Web designer never calls the Set method, which is why extender controls are difficult to work with in ASP.NET. The Set method is supposed to notify the application that the designer is finished editing a control and so can be used to finalize adding of the control. But since this method never fires, there's no easy way to detect when the extender property has been edited and should be persisted.

Workaround for Designer Notification
You have to go to a bit of extra effort to work around this notification problem. You need to notify the designer of a change every time a property value on the extender property is changed. You do this by implementing a method on the main extender control, like this:
internal void NotifyDesigner()
{
    IDesignerHost Host = this.Site.Container as IDesignerHost;
    ControlDesigner Designer =
        Host.GetDesigner(this) as ControlDesigner;
    PropertyDescriptor Descriptor =
        TypeDescriptor.GetProperties(this)["DataBindingItems"];

    ComponentChangedEventArgs ccea =
        new ComponentChangedEventArgs(this, Descriptor, null,
            this.DataBindingItems);
    Designer.OnComponentChanged(this, ccea);
}
This method forces the designer to persist the DataBindingItems collection into the HTML markup. The final step is to call this method every time a property is changed on one of the DataBindingItems. Here's an example of the ControlId property on wwDataBindingItem:
[NotifyParentProperty(true)]
[TypeConverter(typeof(ControlIDConverter))]
[Browsable(true)]
public string ControlId
{
    get { return _ControlId; }
    set
    {
        _ControlId = value;
        if (this.DesignMode && this.Binder != null)
            this.Binder.NotifyDesigner();
    }
}
private string _ControlId = "";
In this example, this.Binder is a reference of the wwDataBinder control. So every time a property is set, the designer is notified and the entire collection is persisted. This is inefficient and requires code in every property that the designer can set. But it ensures that the designer and HTML markup are always in sync. This proactive persistence is the only way I've found to ensure that the properties are properly written.
The lack of calls for the Set<property extender> method causes a few other minor problems. For starters, empty items get added to the collection because the designer initializes the extender property as soon as a control gets activated. Because there's no notification for a save operation, the value is added to the collection and persisted as soon as the designer accesses the extender properties and you end up with a blank item. There's some workaround code in the class that cleans out the list when items are added in design mode, so there's at most one empty item in the list. It's messy, but it doesn't interfere with anything as the binder ignores items that don't have a BindingSource set.
The second issue is slightly more serious. If you rename a control, the association between the DataBindingItem and the control gets lost and settings are cleared out for the control. Again, the problem is that the designer doesn't notify the name change properly, so there's no chance for the code to intercept the ControlId change and adjust the data binding item. Instead you end up with a new item and an old item that now points at an invalid ControlId. If you run the form, an error occurs during unbinding as the binding can't be resolved. It's easy to fix-you can change the HTML markup manually, rename the ControlId of the old item, and remove the newly added one.
These quirks are minor, but they are a reminder that the Web designer in Visual Studio has some issues with Extender properties and that you need to work around them for designer support.

How DataBinding and Error Management Works
The key features of the wwDataBinder control are data binding and error management. The control's functionality to handle the data binding is straightforward. It relies primarily on reflection. Each of the individual wwDataBindingItem objects has DataBind and Unbind methods, which are responsible for binding of an individual data or property value to an individual property of the target control. The main processing for binding, unbinding, and error detection occurs in these two methods.
The data binding process starts with filling the DataBindingItems collection, which is done automatically by ASP.NET when it reads the ASPX page and parses the definitions in the page into the DataBindingItems collection. You can also programmatically add items using AddBinding method. When all the data has been loaded and is ready for binding, the DataBind method is called on the wwDataBinder object.
wwDataBinder.DataBind is a high-level coordinator, but its core task is to loop through the collection of DataBindingItems and fire their DataBind methods. Most of the work occurs in the DataBind method of the individual wwDataBindingItem objects where reflection is used to parse the string-based values of the DataBindingItem into object references and properties.
The code to do this is tedious, but the source includes a few handy Reflection helper classes that make it easier to retrieve complex hierarchical object references. For example, the following code is used to retrieve an object reference and then assign a value to it:
object Source = wwUtils.GetPropertyEx(WebPage,
    "Item.Entity.ContactInfo");
wwUtils.SetProperty(Source, "Address", "32 Kaiea Place");
Reflection doesn't natively support hierarchy syntax, so this helper function and its SetPropertyEx counterpart make short work of retrieving and assigning complex object references. You can find these helper classes in the wwUtils class provided in the wwDataBinder project and assembly.
Even with these helpers, the code for retrieving a reference to the source object, checking for the type of the object, and then assigning the value is lengthy. There are a number of additional checks for specific type of objects, such as ADO.NET DataRow, DataTable, and DataSet objects, which are handled differently than, for example, properties on an object.
The core process of the DataBind and Unbind methods is to read the value and then reassign it to the target. A lot of things can cause problems in that process too-controls or property names may be incorrect, or unbinding values can be incorrect and therefore won't convert back into the proper source types. Unbinding, in particular, needs to deal with the issue of turning input string values back into typed values. There are a number of elements to consider in this process, such as culture awareness and special types like Enums. The wwDataBindingItem DataBind and Unbind methods are too long to show here, but if you want to see the core of the binding activities, check them out in the provided source code.
When binding errors occur in either of these two methods, special exceptions are thrown up the hierarchy and back to the wwDataBinder.DataBind or Unbind methods. These methods handle the errors and convert them into BindingErrors and ultimately error icons/messages on the page. Figure 7 shows the wwDataBinder.Unbind method so you can get an idea of the high-level flow.
Figure 7 wwDataBinder.Unbind Method
public bool Unbind(Control Container)
{
    bool ResultFlag = true;
    foreach (wwDataBindingItem Item in this.DataBindingItems)
    {
        try
        {
            if (this.BeforeUnbindControl != null &&
                !this.BeforeUnbindControl(Item)) continue;

            // *** This is where all the work happens!
            Item.Unbind(Container);

            // *** Run any validation logic - DataBinder Global first
            if (!OnValidateControl(Item) )
                this.HandleUnbindingError(Item, new
                    ValidationErrorException(Item.BindingErrorMessage));

            // *** Run control specific validation
            if (!Item.OnValidate())
                this.HandleUnbindingError(Item, new
                    ValidationErrorException(Item.BindingErrorMessage));
        }
        // *** This handles any unbinding errors
        catch (Exception ex)
        {
            this.HandleUnbindingError(Item, ex);
            ResultFlag = false;
        }

        // *** Notify that unbinding is done
        if (this.AfterUnbindControl != null)
            this.AfterUnbindControl(Item);
    }

    return ResultFlag;
}

Handling Binding Errors
When errors occur, they are caught and passed to an internal HandleUnbindingError method. This looks for specific errors and tries to create a sensible error message. The data binding operations throw a few known exceptions, including BindingErrorException, which occurs when a value cannot be bound or unbound due to a formatting problem. RequiredFieldException and ValidationErrorException occur if a Validation is implemented and returns false. These errors provide detailed error information, which is used to format the message displayed.
The AddBindingError is a key method that is responsible for creating and adding a BindingError object to the BindingErrors collection, creating the error message text and injecting the error icon next to the control. The BindingErrors collection is an important part of the binding process, as it's the primary mechanism for the developer to check for errors (if DataBinder.BindingErrors.Count > 0), and it provides the user display for error information (DataBinder.BindingErrors.ToHtml).
I ran into a few snags trying to create the icons dynamically at run time. Initially, my code used Literal Control insertion to add the dynamic HTML markup into the page like this:
// *** Generate the HTML to be injected next to the control
string HtmlMarkup = this.GetBindingErrorMessageHtml(BindingItem);

if (!string.IsNullOrEmpty(HtmlMarkup))
{
    LiteralControl Literal = new LiteralControl();
    Control Parent = BindingItem.ControlInstance.Parent;

    int CtlIdx = Parent.Controls.IndexOf(BindingItem.ControlInstance);
    try
    {
        Parent.Controls.AddAt(CtlIdx + 1, Literal);
    }
    catch { }
}
This works fine most of the time, but not when there is <% %> script code on the page. If a container contains script code, you can't add any new controls to it dynamically. If you attempt to do this, you end up with the following error: The Controls collection cannot be modified because the control contains code blocks (such as <% .. %>).
The problem is that ASP.NET renders the results of a custom ParseTree method (that includes hard-coded control definitions, evaluation expressions, and literal text) into the Render code for the control container when script expressions are present in a container. In effect, the Controls list becomes unmovable and adding a new control to this list doesn't work reliably, so ASP.NET throws this error. The code I just showed you catches this error and simply doesn't render the icon next to the control. But the binding error is still added and displays on the error summary.
Since this control is supposed to be a generic data binding control, I created a workaround using client script. Rather than injecting a new control into the page, I created a JavaScript function that takes a ControlId and an HTML string as input and adds the HTML from client code. So rather than embedding a literal control, I embed a call to this JavaScript function in the page. The generated code for a single validation error is shown in Figure 8.
Figure 8 Dynamically Injecting HTML Using JavaScript
<script type="text/javascript">
<!--
AddHtmlAfterControl('txtCategoryId',
 '&nbsp;<img src="images/warning.gif" alt="First item can\'t..." />');
AddHtmlAfterControl('txtUnitsInStock',
 '&nbsp;<img src="images/warning.gif" alt="Invalid numeric..." />');
// -->
</script>
<script type="text/javascript">
<!--
function AddHtmlAfterControl(ControlId,HtmlMarkup)
{
   var Ctl = document.getElementById(ControlId);
   if (Ctl == null) return;

   var Insert = document.createElement('span');
   Insert.innerHTML = HtmlMarkup;

   var Sibling = Ctl.nextSibling;
   if (Sibling != null)
      Ctl.parentNode.insertBefore(Insert,Sibling);
   else
      Ctl.parentNode.appendChild(Insert);
}// -->
</script>
In ASP.NET, this is hooked up with the following code for each BindingError:
Message = this.GetErrorMessageHtmlMarkup();
if (!this._ClientScriptInjectionScriptAdded)
   Page.ClientScript.RegisterClientScriptBlock(
      this.GetType(), "AddHtmlAfterControl",
      @"function AddHtmlAfterControl(ControlId,HtmlMarkup){ ... }");

this.Page.ClientScript.RegisterStartupScript(
   this.GetType(), Item.ControlId,
   string.Format("AddHtmlAfterControl('{0}','{1}');\r\n",
   Item.ControlInstance.ClientID, Message), true);
This code generates the individual AddHtmlAfterControl client-side function calls, which get embedded as literal strings into the StartupScript block of the Page so they fire when the page loads. The message is a full HTML string that includes the image and alt text or other formatting depending on which format was chosen to display error messages. The code also renders the AddHtmlAfterControl function the first time it is hit, so the StartupCode calls can actually find the function. ASP.NET makes this sort of client-side code generation from the server quite easy.
Another interesting feature of the error display of the control is the error list. If you look again at Figure 2, you can see that all the errors displayed in the summary list have links to click on. These take you to the control in error. The code also highlights the control with a red border for three seconds.
All of this is handled with JavaScript, which is generated into the error messages with brute force. The generation of these errors is handled by the BindingErrors.ToHtml method (shown in Figure 9), which loops through all the binding errors and writes the list of errors as an unordered list. Each item's error message is then marked up as a link with HTML and a JavaScript script action that jumps to the ClientId of the control.
Figure 9 Creating Error Messages with Links
public string ToHtml()
{
    if (this.Count < 1) return "";

    StringBuilder sb = new StringBuilder("");
    sb.Append("\r\n<ul>");
    foreach (BindingError Error in this)
    {
        sb.Append("<li style='margin-left:0px;'>");
        if (Error.ClientID != null && Error.ClientID != "")
            sb.Append(

"<a href='javascript:;' onclick=\"var T = document.getElementById('" + Error.ClientID + "'); if(T == null) { return };" +
"T.style.borderWidth='2px';T.style.borderColor='Red';"  +
"try {T.focus();} catch(e) {;} " +
@"window.setTimeout('T=document.getElementById(\'" + Error.ClientID  + @"\');T.style.borderWidth=\'\';T.style.borderColor=\'\';',3000);" +
"\"">" + Error.Message + "</a>\r\n</li>");

        else sb.Append(Error.Message + "\r\n");
    }
    sb.Append("</ul>");
    return sb.ToString();
}
Consistent error handling and display is the sign of a polished application, so having a consistent mechanism to handle errors related to data binding is important. It's even nicer if you can indirectly apply errors from other areas of your application like business object validation errors into the same error display mechanism. For example, in my applications, I use business objects that produce validation errors when calling the Validate method on the business object. It's a simple matter to loop through the collection of validation errors and add them to the binding errors of the data binder:
if (!Item.Validate())
{
    foreach (ValidationError Error in Item.ValidationErrors)
    {
        DataBinder.AddBindingError(Error.Message, Error.ControlID);
    }
}
By doing so, validation errors from the business object now participate directly in the data binding process and I can produce errors from operations that are completely decoupled from the user interface. Well, almost decoupled-notice that there's a ControlId property on the ValidationError object returned from the business object. In the business object, I can choose to assign a common ControlId to my validation errors. Here's a business object Validate method:
public bool Validate()
{
    this.ValidationErrors.Clear();

    if (this.Entity.UnitPrice < 0)
        this.ValidationErrors.Add(
            "Unit price can't be smaller than 0",
            "txtUnitPrice");

    if (this.ValidationErrors.Count > 0)
        return false;

    return true;
}
I'm optionally adding a control name to the Validation error in the business object code. In my applications I usually use consistent control names that match underlying property names. While this introduces some coupling, it also brings together data binding and validation errors in a consistent manner. The coupling is purely optional and if the control name is omitted, the code for adding validation errors to binding errors still works. The summary still displays, but the message has no link to the control and there is no icon next to the control.

Wrapping Up
Data binding is an important part of most Web applications. The control I've described in this article provides a lot of functionality in an unobtrusive way. It also brings together several related tasks like data binding, error handling, and error display in a way that is not easily done with stock ASP.NET functionality. While there are other ways to approach simple control data binding, this is one that I've found to be most productive for my own uses.
I hope you'll find this control as useful as I have. It addresses a number of scenarios that ASP.NET 2.0 doesn't currently handle very well, and it does so in a non-intrusive manner so you don't have to change anything to use it. Extender controls like this one are a great way to extend the functionality of existing controls without requiring subclassing of existing controls. So go ahead and extend the functionality of what ASP.NET already provides.

Rick Strahl is president of West Wind Technologies on Maui, Hawaii. The company specializes in Web and distributed application development, tools, training, and mentoring with a focus on .NET, IIS, and Visual Studio. For more information, visit www.west-wind.com or contact Rick at rstrahl@west-wind.com.


BizTalk Adapters
Integrate E-Mail Processing Into Your Business Solutions
Alex Starykh

This article discusses:
  • The BizTalk orchestration architecture
  • Building custom BizTalk solutions
  • Extracting and processing data from e-mail
  • Creating custom adapters
 This article uses the following technologies:
BizTalk Server 2006
Contents
First, Some Background
Building the Solution
Orchestration
Create a Custom Adapter
It's a common problem everywhere. Data arrives at your doorstep in all sorts of formats, typically initiating some sort of customer service, order fulfillment, or other business process. Often the data arrives in an e-mail message accompanied by other documents. Once the e-mail is received, an automated reply must be sent and the e-mail must be saved to a database or to a line-of-business (LOB) application. Depending on the kind of data received, different e-mail notifications may be needed.
Dealing with the variety of data formats and work processes is no easy task. Fortunately, BizTalk® Server can help. In this article I'll develop a solution that shows how. My solution uses BizTalk Server to monitor a specified e-mail account for new messages, trigger a confirmation when a new e-mail message is received, and then persist the entire contents of the e-mail, including all attachments, to a custom database. It also includes some conditional logic that will send notifications based on the data in the claim. New features in BizTalk Server 2006, including the new POP3 adapter and the enhanced SMTP adapter and variable multipart message support will glue the solution together.
If you want to further refresh your BizTalk knowledge, you may want to look at the useful introductions by Aaron Skonnard in the November and December 2005 issues.

First, Some Background
BizTalk Server is an integration server designed to work with messages. Figure 1 shows a typical message flow. A message is received by one of the locations belonging to a receive port. While one port can have multiple receive locations, there is always one and only one port for each location. Each receive location has a specific configuration for one adapter. For example, a receive location may have a configuration set for a file adapter that indicates what type of files (file mask) to poll from what folder. When the adapter receives a new message, the message moves from the receive location to the receive port, then to a pipeline. The pipeline is simply a sequence of components that is executed for every message. Usually the pipeline performs tasks such as decoding, decrypting, verifying messages, and generating the XML representation.
Figure 1 Typical Message Flow in BizTalk
Figure 1 Typical Message Flow in BizTalk
Once work in the pipeline is complete, the message is submitted to a SQL Server™ database called the MessageBox, which contains information about subscriptions, in-flight and suspended messages, message properties, tracking, queuing, and other important services provided by BizTalk.
BizTalk is built on a publish/subscribe model. Every single message received by BizTalk is automatically routed to subscribers whose subscriptions match, say, all messages received from a certain port. A more complex subscription might target messages of a given type with values promoted from inside the messages.
When a message is published to the MessageBox database, if there is not at least one subscriber for that message, it is immediately suspended. Most suspended messages can be manually resumed for processing later.
Subscribers in BizTalk are either send ports or orchestrations. Each send port's subscription determines what messages will be routed to that port. When a message is destined for a send port it goes through a send pipeline. Usually the send pipeline is responsible for assembling the message into the appropriate format, signing and encrypting the message, and so on. From the pipeline the message goes to the adapter configured on the send port.
Orchestrations represent a business process workflow that executes when an appropriate message is received by BizTalk. When orchestrations send messages, those messages are published again to the MessageBox to be routed to the matching send port.
BizTalk ships with more than 25 different adapters that cover most transport protocols and many common applications, including file, FTP, HTTP, SOAP, SMTP, POP3, SharePoint®, SQL Server, Oracle, SAP and EDI. Most can be used for receive and send ports. Refer to the BizTalk documentation for a complete list of available adapters.
If no built-in adapter matches your needs, BizTalk lets you create a custom adapter. Later in this article I'll show you how.
The new BizTalk Server 2006 POP3 adapter is responsible for polling a message from a specified mailbox, parsing the message, and separating attachments. The adapter then creates a new multipart message with one message part designated as the body and other parts corresponding to each attachment.
The body part represents a default part that is used by the engine for processing because its type is known to BizTalk. But BizTalk doesn't have to understand the other parts of the message, unless we explicitly declare them. In fact one of the new features of BizTalk Server 2006 is the ability to declare only the body of the multipart message at design time, leaving the number of other parts and their types undeclared. This comes in handy when the number of attachments is unknown beforehand.

Building the Solution
Now let's start developing our solution. First we'll set up the SMTP and POP3 servers. If you have them already available for development purposes, use them; otherwise install the SMTP and POP3 services that ship with Windows Server® 2003. You can add those services in Control Panel by selecting Add/Remove programs, then Windows Components, then SMTP and POP3 services. When that's done, add mailboxes for a couple of users to test mail delivery functionality between those users.
Now go to the BizTalk Server Administration console and create a new application called ClaimsProcessingApp.Note that applications in BizTalk represent a logical grouping of different BizTalk artifacts.
You can create the first receive port with one receive location that will use the POP3 adapter by right-clicking on the Receive Ports folder and selecting new one-way receive port, specifying the port name RcvEmailPort. Next, click on the ReceiveLocation tab and create one receive location with a transport type of POP3. Then select the POP3 adapter and click Configure to get the POP3 adapter configuration screen.
The POP3 adapter lets you specify how the message should be parsed when it's received by the adapter. Figure 2 shows some of the most important configuration settings for the POP3 handler.
Figure 2 Configuration Settings for the POP3 Handler
Figure 2 Configuration Settings for the POP3 Handler
In our scenario, the received e-mails have one XML attachment and any number of other attachments, so we'll specify that mime decoding should be used and that the body part content type is text/xml. Don't forget to specify the mail server, user name, and password used for retrieving e-mail messages, and make sure the poll frequency is set to something appropriate for development purposes-let's say once every 15 seconds.
Now let's get ready to test our receive port. We'll use the messaging-only aspect of BizTalk by creating a send port that subscribes to any message received by our receive port. Click on send port, selecting a new static one-way port, and File as the transport adapter for the port. Then specify a physical location where the test messages will be placed. Now specify a subscription on that port: select Filters from the Send Port Properties list (as in Figure 3) and specify the following filtering expression to indicate that all messages received by RcvEmailPort will go to this send port:
BTS.ReceivePortName == RcvEmailPort
Figure 3 Send Port Filter
Figure 3 Send Port Filter
Let's test what we've done so far. Go to the receive location configuration, enable it, and start the send port we just created. Then send e-mail with a couple of attachments including an XML file (any XML file will do) to the monitored account. A few moments later you should see the new file created with the body of the message you sent. If you don't, first verify that the BizTalk host is running and then check the event log for additional troubleshooting steps.
At this point you should be able to retrieve a message from the POP3 server and save the XML portion of that message to a test folder.

Orchestration
Now we'll build the business process that begins as a new message arrives. Again in our scenario, the entire content of the message, including its sender, is saved to a database. Then an automated acknowledgment is sent back to the sender. The last step is to decide if additional e-mail notification is needed based on the claim amount.
Typically this sort of business process is implemented via orchestrations. To build one, start Visual Studio® and create a new project based on an empty BizTalk project type.
To design a new schema that represents the claim data, right-click on the project and select Add New Item | Schema and name it Claim. Add several fields to the claim, including the claim amount as a decimal type. Right-click on the amount and select Show Promotions, then promote that field as a distinguished field (see Figure 4). I've also created a Submitter node in the schema with promoted from, to, and subject elements in that node. We will need those promotions later when we capture details of the original message and decide about additional notification.
Figure 4 The Claim Schema
Figure 4 The Claim Schema
We will also need a sample of the instance that complies with the schema. BizTalk can generate one if you right-click on the schema and select Generate Instance.
Now we're ready to design an orchestration. Much as you added the schema, add a new orchestration to the project, then create a new multipart message type, called ClaimMessageType, with one part-a body part called xmlBody defined by the schema we just designed. Add two messages-incomingClaimMsg and finalClaimMsg-of the same ClaimMessageType. Now add a receive shape, which will receive incomingClaimMsg and will activate a new instance of the orchestration (activate property of the shape needs to be set to true).
The next step is to construct a new, finalClaimMsg, of the same message type but with additional values persisted to the fields in the body of the message. Those fields will be populated based on information from the SMTP header of the message. For example, we will add e-mail Subject, From, and To addresses to the original XML body of the message.
To achieve this we assign messages first using the following expression in a newly added message assignment shape:
finalClaimMsg = incomingClaimMsg
In the same construct message shape, we also assign the additional fields:
   finalClaimMsg.xmlBody.Submitter.from = incomingClaimMsg(POP3.From);
   finalClaimMsg.xmlBody.Submitter.subject =
       incomingClaimMsg(POP3.Subject);
   finalClaimMsg.xmlBody.Submitter.to = incomingClaimMsg   (POP3.To);
and
finalClaimMsg(BTS.DestinationParty) =
    "mailto:" + incomingClaimMsg(POP3.ReplyTo);
The last line sends an acknowledgment back to the sender.
Now we are ready to send finalClaimMsg to the file or database and send an acknowledgment back. To do so, add two send shapes and indicate that finalClaimMsg should be used on both of the shapes. Finally, let's add a decision shape to indicate that with a claim amount of more than 100, an additional notification will be sent to a predefined user. The rule in the decision shape might look like the following:
If (finalClaimMsg.Amount > 100)
The Then branch would consist of only a new send shape with finalClaimMsg.
Now we need to create four logical ports and connect them to the appropriate send or receive shapes. Drag and drop a port shape from the toolbox to the orchestration port surface and go through the port configuration wizard, accepting all default values except port name and the communication's direction. Repeat that process to create one receive and three send ports. Connect the receive and send shapes to newly created logical ports. The final orchestration should look similar to Figure 5.
Figure 5 The Complete Orchestration
Figure 5 The Complete Orchestration
Now adjust the project properties to use a key file for assembly signing and then compile the project. Go to the deployment tab and change the Application Name property to ClaimsProcessingApp. Deploy the solution by right-clicking the project and selecting deploy menu. At this point the project with all schemas, maps, and orchestration is compiled into a .NET assembly that gets placed in the Global Assembly Cache (GAC) as well as registered with the BizTalk management database.
Assuming deployment is successful, let's go back to the BizTalk Management console to bind the orchestration's logical ports to the physical ports we'll create next.
There are two additional ports that use the SMTP adapter to send an e-mail acknowledgment and notification. These ports are very similar except for the actual message text, which can be specified on the port level as opposed to using the dynamic port with the body specified in the orchestration.
The first port, named SendAcknowledgmentBack, uses the SMTP transport with the To address set to %DestinationParty%, which will be replaced by the destination party information we've already assigned in the message assignment shape (finalClaimMsg(BTS.DestinationParty)=...).
The second port, SendAdditionalNotification, uses SMTP transport with the delivery address set to a fixed value.
For the database port, let's temporarily use the file adapter port that was created in the previous part of the solution, but with the subscription filter completely removed. We'll change the file transport to the database transport in the next section. Select the SendResultsOut send port and remove any subscription filters.
Now we need to change a pipeline on the ReceiveEmailMsg from Passthrough to XmlReceive, since BizTalk needs to know what's inside the message's body part. Also, since we are using promoted fields the actual promotion is also done via an XmlAssembler that resides in XmlReceive pipeline. Go to the properties of the deployed orchestration and specify the bindings as shown in Figure 6.
Figure 6 Orchestration Bindings
Figure 6 Orchestration Bindings
Now we're ready to test the solution. Start the application (right-click on the application and select Start) and send a new e-mail message to the account monitored by BizTalk with the sample message generated earlier as an attachment.
If everything is set correctly, we'll receive an acknowledgment back and one additional file will be created in the destination folder with the XML body of the message. If the sample we attached had a claim amount greater than 100, an additional e-mail notification will be sent as well.

Create a Custom Adapter
There's still one major piece of the solution missing. The file that BizTalk creates in the destination folder represents only the XML body of the message. Where are all the attachments and what about storing the entire message in the database? As it turns out, the majority of the BizTalk adapters, including the file adapter, use only the body of the message, completely ignoring the rest of the parts. While this makes sense for many BizTalk implementations, it's not exactly what we need here. Enter custom adapters.
Custom adapters are .NET components registered with BizTalk Server that are responsible for receiving and/or sending messages. While building a generic custom adapter can be fairly complex, Microsoft has created a base library called the Adapter Framework, which makes creating a custom adapter much easier. The framework ships with the BizTalk SDK; its complete source code is available in C#.
Figure 7 highlights the relationship between a custom adapter and the BizTalk Server environment. Not only does the framework enable you to create a run-time environment for the adapter, it also allows custom configuration of settings that are rendered to the user by BizTalk Server Explorer or BizTalk Server Administration console.
Figure 7 Adapter Framework
Figure 7 Adapter Framework
There are multiple variables that influence design requirements for the adapter. Among the most critical are:
  • Communication direction—whether the adapter is a send or receive adapter.
  • Hosting-whether the adapter is hosted by a BizTalk process (in-process host) or outside of the BizTalk environment.
  • Message exchange pattern—which involves three possibilities: one-way asynchronous, in which messages flow in one direction with no responses or other dependencies from previous or subsequent messages; request-response, in which each message flowing into the adapter should generate a response back to the message box (used, for example, when an orchestration initiates a call to a Web service using a SOAP adapter); solicit-response, which is similar to request-response except the direction of the initial message (as when an orchestratin is exposed as a Web service using a SOAP adapter).
  • Batch-supported or batch-unsupported—whether the adapters can send multiple messages simultaneusly, which reduces chattiness between the adapter and messagebox database as well as between the adapter and the outside world.
  • Transaction-aware—whether the adapter supports transactional sending and receiving of messages.
There are several mandatory .NET interfaces that all adapters must implement. These interfaces include IBaseComponent, which returns name and version of the adapter, IBaseTransportType, which details transport type, and IBatchCallback, which receives the status of a submitted batch to the MessageBox database. Depending on the selected variables, the adapter is likely to require implementation of additional interfaces. For example, a batch-aware transactional send adapter is represented by a class that must implement IBTBatchTransmitter. The BizTalk documentation outlines specific requirements for each adapter type implementation, so refer to this resource often.
The send adapter we need to create here has to support transactional interactions with a custom database, relying on a one-way asynchronous pattern. Our adapter will be generic, meaning it won't know much about message content. It will call a stored procedure for each message part, passing message part data and supporting information-contentType, messageId, and a flag indicating if the message part passed is a bodyPart. So each message part will be persisted to the database in a context of a single atomic transaction initiated on a message level. This means that if some part can't be persisted to the database (let's say the call to the stored procedure fails), the transaction will roll back and the entire message will not be saved. In that case, the adapter will repeat the process until the limit set in the adapter's advanced properties is reached. When the limit (by default up to three times every five minutes) is reached, the message will be suspended.
From the adapter point of view, each message is represented by the IBaseMessage interface. Since each message can have multiple message parts, those parts are represented by the IBaseMessagePart interface. The actual body of the message part is accessible via the IStream interface, as shown in Figure 8.
The IBaseMessageContext interface represents a dictionary indexed on property name and namespace. Such an interface-centric design allows easier interoperability with COM-based adapters as well as implementation-specific substitution (for example, based on size of the message) of actual instances.
Figure 8 The IBaseMessagePart Interface
Figure 8 The IBaseMessagePart Interface
While we could proceed along that route and build upon the Adapter Framework, let's take an easier route. The BizTalk SDK ships with several examples of custom adapters. One is a transactional send adapter that does almost exactly what we need. So we will build on that sample instead of starting from scratch.
Let's copy the sample from BizTalk Installation Folder\SDK\Samples\AdaptersDevelopment\TransactionalAdapter to a new folder so it can be safely modified. For my installation BizTalk Installation Folder refers to C:\Program Files\Microsoft BizTalk Server 2006.That folder contains two projects: one represents design-time features of the adapter and the second represents run-time features. We also need to copy the base adapter project located in the BaseAdapter subfolder of the directory I just cited.
We can easily add those three projects to the same solution we used to design the orchestration (though you can create a new solution if you prefer) using File | Add | Existing Project in Visual Studio. Check for any missing strong name key files and replace them with your own if needed.
Since we're building a send adapter, we can safely delete several files designed to support the receiving side of the adapter. Those files include TransactionReceiver.cs and TransactionReceiverEndpoint.cs.
Let's quickly review the code. The entry point for our send adapter is in the TransactionalTransmitter class. In fact, that class is exactly what we will be using later on when we create a .reg file for our adapter registration.
The code of the TransactionalTransmitter class shows that the CreateAsyncTransmitterBatch method is called by BizTalk when there is a new batch to send out. The code creates a new instance of the TransactionalAsyncBatch class, which implements the IBTTransmitterBatch interface that is needed to inform BizTalk that the adapter sends messages asynchronously.
The TransmitMessage method is called for each message in the batch and then the Done method runs. For each batch of messages, the Done method creates a new worker thread where the SendMessage method is eventually executed.
Let's make some changes to the projects. First, to be good citizens we need to modify a unique GUID associated with the adapter in case you need to reuse the original source code again. The GUID is located in TransactionalTransmitter.cs and it's passed in as a parameter in the base constructor of TransactionalTransmitter class.
Also, in TransactionAsyncBatch.cs, the Send method must be changed. That method is called when the adapter is ready to send out a message or message batch. The new code for that method is located in Figure 9.
Figure 9 New Send Method
private void SendMessage(
    IBaseMessage message, TransactionalTransmitProperties properties)
{
    string storedProcedureName = properties.CmdText;
    string connectionString = properties.ConnectionString;

    Guid messageId = message.MessageID;

        using (SqlConnection connection = new SqlConnection(
            connectionString))
    {
        connection.Open();
        for (int i = 0; i < message.PartCount; i++)
        {
            //get one part
            string partName;
            IBaseMessagePart part =
                message.GetPartByIndex(i, out partName);

            //read the part body to an array
            byte[] msgData = new byte[part.Data.Length];
            part.Data.Read(msgData, 0, msgData.Length);

            string contentType = part.ContentType;
            bool isBody = (part.PartID == message.BodyPart.PartID);

            SqlCommand command =
                new SqlCommand(storedProcedureName, connection);
            command.CommandType = CommandType.StoredProcedure;

            command.Parameters.Add(
                new SqlParameter("@MessageID", messageId));
            SqlParameter bodyParam =
                new SqlParameter("@Data", SqlDbType.VarBinary);
            bodyParam.Value = msgData;
            command.Parameters.Add(bodyParam);
            command.Parameters.Add(
                new SqlParameter("@ContentType", contentType));
            command.Parameters.Add(
                new SqlParameter("@IsBody", isBody));

            //execute
            command.ExecuteNonQuery();
        }
    }
}
As you can see from Figure 9, we open a new connection to SQL Server, accepting a connection string as a parameter that can be customized at deployment time, then iterate through each message part of the outgoing message. After reading message content, content type, and a flag indicating whether the part is a body, we create and execute a new SQL command. If you check the code that calls the Send method (the BatchWorker method), you'll find that it creates and controls transactions using the Transaction class new to the .NET Framework 2.0-that's exactly what allows us to run code in a single atomic transaction.
Figure 9 has the complete text of the stored procedure that's invoked by the adapter. As you can see, it's a simple insert statement against a custom table. Be careful not to use GUID as a primary key for your table when you perform many inserts as this may cause future performance problems due to data page fragmentation:
CREATE PROCEDURE [dbo].[MakeMessage]
    @messageId uniqueidentifier,
    @data varbinary(max) = null,
    @contentType varchar(50),
    @isbody bit = null
AS
BEGIN
    INSERT INTO Messages(messageID, data, contentType, IsBody)
    VALUES (@messageId, @data, @contentType, @isBody)
END
Figure 10 AdapterRegistryWizard
Figure 10 AdapterRegistryWizard
Now build the solution. When the build succeeds, we need to create a registry file (*.reg) that summarizes changes needed to register our adapter within BizTalk. Fortunately, that task can be easily done by running the AdapterRegistryWizard from BizTalk Installation Folder\SDK\Utilities\AdapterRegistryWizard. When the wizard starts, type a name for the adapter and the properties namespace, and check the box indicating that the adapter will only send messages (see Figure 10).
On the next screen, select the assembly with the run-time implementation of the adapter-usually it's in the project's bin folder-and select the Transmitter class as an implementation entry point for the adapter. The adapter alias can be any string uniquely identifying the adapter and used for dynamic ports in the orchestration, but it's not used for our solution.
The next screen asks about management interface implementation. It needs to point to the assembly generated by the second project we've copied from the original location. Make sure that the box representing send handler configuration is checked. Specify a location output for the registry file and the wizard will create the .reg file there. I recommend that you review the registry file as it contains useful information for better understanding the adapter registration procedure. Double-click on the .reg file and confirm the registry merge.
The final step is to make the adapter available to the BizTalk management database. We'll do so by expanding the Platform Settings | Adapters folder in BizTalk Administration console and right-clicking and selecting the New | Adapter menu. Our adapter should be available from the dropdown selection; type a name and confirm the changes. Don't worry about restarting the host now as that can be done later.
Now it's time to test the adapter. Stop the application (right-click and select the Stop command), then go to the SendResultOut port and specify a new transport to use: TransactionDbAdapter. Click configuration and specify a connection string and stored procedure name. Restart the host via Platform Settings | Host Instances, select BizTalkApplication, and click Restart.
Now send an e-mail message with several attachments, including an XML form, to the mailbox monitored by BizTalk. As soon as an acknowledgment is received, check the table in the database-you should see several records, one corresponding to each attachment and one for the message body.
As you've seen, triggering a business process when e-mail is received can be easily accomplished with BizTalk Server. In many scenarios there is no need to write any code, but if you must, use the Adapter Framework from the BizTalk Server SDK and take advantage of the samples shipped with the product to jump-start your development.

Alex Starykh is a Technology Architect at Microsoft Technology Center in Virginia. He has more than 12 years of experience in architecture, development, and consulting for commercial companies and government agencies with focus in integration, databases, and custom development. Alex can be reached at alex.starykh@microsoft.com.


Locale Hero
Enable Global Apps With Locale Builder And Windows Vista
Kieran Snyder and Shawn Steele

This article is based on a prerelease version of Windows Vista. All information contained herein is subject to change.
This article discusses:
  • How locale data affect your application
  • Creating custom locales that work
  • Dates, times, numbers, and calendars
  • Text rendering in a locale
 This article uses the following technologies:
Windows Vista
Contents
Microsoft Locale Builder
Creating or Modifying a Locale
Creating Custom Locales that Work
Date and Time Formatting
Numbers, Currency, and Text
How Locale Data Affect Your Application
Time, Dates, and Calendars
Text Rendering in a Locale
When Not to Use Locale Data
Pseudo-Locales for Testing
What's Next?
Internationalization support in Windows Vista will exist on a scale not seen in previous releases of Windows®. When Windows Vista™ ships, it will have built-in enablement infrastructure for over 130 languages. This will offer an enhanced experience not only for languages that Windows has supported before, but also for many more customers around the globe, including users in Latin America, Asia, and Africa. As a result, the expanded internationalization infrastructure introduced in Windows Vista and the Microsoft® .NET Framework 2.0 opens up a new world of potential customers for application developers as well.
Windows enablement support is organized around the concept of locales. A locale is a set of data associated with a particular language-region pairing-for example, English in Canada, Greek in Greece, or Telugu in India)-that may be used to help shape the user experience. Windows Vista will include over 200 unique locales that will provide data appropriate for customers around the world, including calendar support with pertinent date and time formats; text formatting, including a set of relevant Unicode code points, information about text directionality and writing system; font and text rendering support; appropriate numeral and currency standards and formatting; language and region display names; linguistically appropriate sorting behavior; and keyboard layout or other relevant input method.
The locale the user selects informs many aspects of his user experience. Locale choice determines whether the user sees a 12- or 24-hour clock; what keyboard he uses by default while typing; whether algebra.txt or zebra.png shows up first when he looks at a list of file names; and what calendar he sees in Microsoft Outlook®. The effects of locale selection are pervasive, and the range of locale data creates some unique opportunities and challenges for developers who want to make their applications globally aware. By using locale data correctly, a developer can tailor applications to create a linguistically and culturally appropriate user experience for a wide range of customers.
And the range seen in existing Windows Vista locales is just the beginning. With the introduction of Microsoft Locale Builder, you will have the ability to create your own highly customized locale support and share it with customers, colleagues, and broad user communities. Locale Builder allows users to replace a particular piece of data inside a built-in Windows locale or to create a locale for a language-region pairing that is not already supported by Windows. This means that developers will have the opportunity to return highly customized locale data in their applications.
By using the Windows font and rendering support that is available for many of the world's writing systems along with Locale Builder and Microsoft Keyboard Layout Creator, you can create applications that are linguistically accessible for over 90 percent of the world's literate population-that's over 3.5 billion people!
In this article we introduce you to Locale Builder and give you some tips on how to create a custom locale with data that is optimized for application functionality. We also include some development best practices for creating applications that can take advantage of a broad range of locale data to create a customized experience for global customers. Our goal is to enable customers to use our locale infrastructure and tools to provide a globally appropriate user experience across Windows-based applications.

Microsoft Locale Builder
Windows has always provided a way for users to customize their locale data, using the Regional and Language Options control panel to select their locale and customize the time or date formats. However, customization through Regional and Language Options is limited to existing locales and relatively simple variations. Additionally, any changes specified in Regional and Language Options do not persist if the locale selection is changed and then changed back.
Although the level of locale support in Windows Vista represents a significant increase in global coverage, it is still relatively small compared to the number of languages and regions in the world. As we go to press, Ethnologue.com lists 6,912 known living languages. Multiply that times a couple hundred countries. Maintaining data on even a fraction of the possible combinations is logistically challenging, particularly for regions where computers are still rare. Additionally, the data for a specific language or region is not always consistent across the preferences of an individual or a group. In these cases, it makes sense to allow users to build the exact locale they need.
Locale Builder was created to help Windows Vista users and administrators easily create new locales or customize existing locales to meet their own needs. Once you've created a custom locale, you can install it on your machine, include it as a package to be installed on a customer's machine, or share it with other people who might be interested in your locale.

Creating or Modifying a Locale
The first step when creating locale support is to see if Windows already supports the locale you want to produce or one that is similar. It is also possible that other users have created a locale for the language or culture that you need. Engaging with a local university or government may be all you need to find a locale appropriate for your use.
If you cannot find an existing locale appropriate for your needs, you can use Locale Builder to make your own. You may want to modify an existing locale if the current support for your language and region doesn't match your expectations, or if a value changes or is wrong, such as when a country adopts the Euro currency after Microsoft has released the new locale. We use the term replacement locale to describe a locale with customized data that is intended to replace a locale shipped by Microsoft. Replacement locales will change the behavior of the target locale for all users of the computer. For instance, if you create a replacement locale for en-US that uses the 24-hour clock, then all applications calling Windows APIs for en-US time support will use the 24-hour setting.
You can also choose to create an entirely new locale, known as a supplemental locale. Locale Builder guides you to use an existing locale as a template for your new locale. This is useful particularly if the language or region for which you are creating support already exists in another locale.
When you start Locale Builder you will be asked for the name of your new locale and the existing locale on which you'd like to base your custom solution as shown in Figure 1. Names are generally in the RFC 4646 language-country ISO standard form, such as haw-US for Hawaiian (United States). In the case of a replacement locale, the new locale name will be the same as the existing locale name. Locale Builder gives a short preview of what the source locale data look like. Click Next when the name and template locale have been selected.
Figure 1 Selecting the Starting Point for a Custom Locale
Figure 1 Selecting the Starting Point for a Custom Locale
After selecting the basic locale name, you have the opportunity to edit the properties of that locale from the locale formats window as shown in Figure 2. Choose from items such as locale names and standards information, number and currency formatting, time and dates patterns, or other kinds of linguistic and cultural information. Each group allows a range of fields that can be customized to the needs of your particular locale.
Figure 2 Editing Locale Properties
Figure 2 Editing Locale Properties
After the locale has been created, you will want to save a Microsoft Installer .msi file so that you can install it, and you will also want to save a source .ldml file so that you can easily make changes later. Click the Save button in the locale editing window to see the save dialog. Choose your file name and location. The locale name usually makes an appropriate file name.
To install your custom locale, just double-click the .msi that you created. Note that you have to be an administrator to install a custom locale. If you want to share the locale with other users you can share either the .msi or the .ldml file. If you need to install the locale as part of a larger distribution you may want to create an .msm file to use with that distribution's installer.
Once it's installed you can then select your new custom locale from the Regional and Language Options control panel (see Figure 3). Verify that the examples display the content you expect and then test your custom locale in appropriate applications, such as Outlook, Windows Explorer, and any applications important to your needs. You can use the Uninstall a Program Control Panel option to remove a locale previously installed via the .msi installer.
Figure 3 Saving a Custom Locale
Figure 3 Saving a Custom Locale

Creating Custom Locales that Work
So now you know how to use Locale Builder, but the big reason you're building a custom locale is that existing Windows locales don't meet all of your requirements. Maybe there's a locale for your language and region, but you prefer a 24-hour clock. Maybe the default character set is Cyrillic, but new legislation has passed requiring school systems in your region to use Latin. Perhaps your country has recently joined the European Union and is adopting the Euro, and you need a currency symbol update. Or maybe Windows doesn't support your locale at all.
We've received feedback from customers concerning all of these scenarios and more. The fact is that linguistically and culturally appropriate data changes over time, and standards and user expectations change to match. This can create sets of users that have very different target user experiences. If you're building a custom locale, it's because you or your customers want something different from what Windows provides today. And there's a pretty good chance that you have a clear idea of just what it is that you need to add or change, whether it's an entire locale or just a piece of an existing locale.
This section presents some best practices for moving forward once you've identified linguistically and culturally appropriate data for your custom locale. By following these best practices, you can ensure that your locale data are optimized for application functionality.
Windows locales have historically been labeled with numeric identifiers called LCIDs. Windows Vista introduces a set of APIs that use string-based locale identifiers to complement the APIs that take LCIDs. Microsoft is migrating away from LCIDs in favor of string-based identifiers for our new locale support, as the meaning of string-based identifiers is more transparent and strings are more extensible for users creating custom locales. If you use Locale Builder to create a supplemental locale, one thing you'll need to do is select an appropriate identifier.
The identifiers used for Microsoft locales in both Windows and .NET mirror the Internet Engineering Task Force (IETF) standard for locale identifiers as closely as possible. The IETF standard relies on RFC 4646, which in turn uses ISO 639 tags to identify languages and ISO 3166 tags to identify regions. The standard relies on full, descriptive Unicode script names and provides a syntax for combining these tags to identify a set of locale data. Identifiers are of the form:
ll(l)-Ssss-CC
In this syntax, ll(l) denotes the two-letter language tag (or the three-letter tag if there is no ISO two-letter code for your language), Ssss denotes an optional script tag, and CC denotes the region tag. In Windows locales, we only include the script tag if there are multiple locales supporting a particular language-region pairing but using different writing systems. Some examples of Windows locale identifiers are shown in Figure 4.
Figure 4 Examples of Windows Locale Identifiers

Identifier | Meaning
en-CA | English (Canada)
fr-FR | French (France)
ja-JP | Japanese (Japan)
quz-BO | Quechua (Bolivia)
uz-Latn-UZ | Uzbek (Latin, Uzbekistan)
uz-Cyrl-UZ | Uzbek (Cyrillic, Uzbekistan)
Rarely, users may need language codes for concepts that aren't addressed by ISO 639 or RFC 4646. In those cases ISO 639 provides the user-assigned codes qaa-qtz, which you may use. Similarly ISO 3166 provided user-assigned region codes such as QMA to QZZ and 900 to 999. RFC 4646 also allows any label after an x label to be user defined. Some examples of user-assigned codes would be en-QMA, qaa-QZZ, and en-x-Aviation. Should you find that you need custom codes for locales, please follow the standards.
We recommend that you choose your locale identifier so that the intended contents of the locale are as transparent as possible to potential consumers of your locale data. If you create a custom locale to share with others, using an identifier that takes advantage of an international standard is a good way to ensure that you are clear about the set of users your locale is intended to support.
You will also want to choose the display name for your locale such that it clearly states the language, region, and, if appropriate, script that your locale supports. The display name you choose will be visible (and selectable) in the Regional Settings control panel on any machine where the locale is installed, and applications may choose to display it as part of selectable UI. The clearer your display name is, the more likely it is that it will be selected by your target users.

Date and Time Formatting
Windows-based applications use the date, time, and calendar properties that you specify much more frequently than other properties. The calendar type that you select for your custom locale will determine which calendar your locale's users see in every calendaring application that calls into Windows for support, including the Windows calendar and Outlook.
There are a few things to keep in mind when selecting your calendar data. First, short day names really are short. We recommend that super-short day names be no more than two characters long. Calendaring applications will often use these abbreviations to display weekdays at the top of calendar rows where UI space is at a premium, as in the M?ori Gregorian calendar in Outlook 2007 shown in Figure 5.
Figure 5 Māori
Figure 5 Māori
Short day names should also be unique. You'll need to make sure that the super-short day-name abbreviations that you select are not all identical to one another so that calendar labels mean something. If every day in your language starts with P, you'll probably want to include a second code point in your short day name abbreviations.
Abbreviated day-names are longer than short day names, but not by much. Calendars frequently use these day-name abbreviations as well as abbreviated month names, so you will want to be mindful of their usual space constraints on UI. For instance, the Windows calendar uses abbreviated day names to label days of the week as shown in Figure 6.
Figure 6 Yoruba Windows Calendar
Figure 6 Yoruba Windows Calendar
Choose your time and date patterns wisely. Locale Builder offers you a selection from among existing Windows patterns, or you can choose to create a custom pattern (see Figure 7).
Figure 7 Customizing Time and Date Formats
Figure 7 Customizing Time and Date Formats
If you choose to create a custom pattern including text or unique date separators, you will want to make sure that you distinguish between formatting characters (M, d, y) and actual text to be rendered in the presentation of date and time formats. The way to distinguish formatting characters from actual text is by escaping the text to be rendered inside single quotes, as seen in the long date formats for Simplified Chinese:
  • yyyy'年'M'月'd'日'
  • yyyy'年'M'月'd'日',dddd
  • dddd,yyyy'年'M'月'd'日'
Any data within single quotes will be displayed without the quotes. Any sequence requiring a single quote can be escaped by using two single quotes: dd'|'MM'|'yyyy will display in the form 18|08|2006, and hh:mm''''ss'''' will display in the form 12:09'53'' (note that the second quote is a double quote).
It's also important to keep in mind that when it comes to formatting characters, capitalization matters. For example, M is the formatting character representing months in date formats, while m is the formatting character representing minutes in time formats. Things like this may seem small, but if your custom locale gets them wrong, then every application that uses your date and time formats will be broken.
Both short and long date formats should contain content to represent day, month, and year, with short dates typically using abbreviated month names and long dates using the full string.
You can choose to include multiple calendar types in your locale, but only the Gregorian calendar can be localized. Locale Builder allows users to select multiple calendar types for a custom locale, with one calendar designated as the default.
The Gregorian calendar is automatically available for all custom locales, and it is currently the only calendar for which you will need to specify localized day and month names. If you select another calendar type to be included, the data that will be used for month and day names will come from built-in Windows data. If you select a non-Gregorian calendar to be the default for your custom locale, you will want to make sure that the built-in calendar name data work for your intended users.
Separators matter. The Windows Vista Regional and Language Options Control Panel expects only the four separator types shown in Figure 8. If your locale requires some other date separator, you will want to make sure that you escape it with single quotes in the way described previously.
Figure 8 Date Separators

Date Format | Description
dd-mm-YY | Default short date pattern for Kannada (India)
dd.MM.yyyy | Default short date pattern for French (Switzerland)
dd/MM/yyyy | Default short date pattern for Irish (Ireland)
d MMMM yyyy | Default long date pattern for Tajik (Tajikistan)

Numbers, Currency, and Text
The data that you specify for formatting numbers and currencies can be used by a range of applications, including money management tools, spreadsheets, currency converters, and other business applications. It is important that you select your values such that the resulting formatting presents an intuitive user experience across the range of applications that might use the data.
The use of native digits may vary. Not all locales use ASCII 0-9 as the characters that represent their base 10 digits. Even for those locales that do use 0-9, there may be other native forms that are used in special contexts. The Locale Builder allows you to specify the native digits for your locale as needed.
Once you have specified the native digits, you will have to decide where you would like them to be used. If you select National digit substitution, then the native digits will be used everywhere that numbers are needed. If you select Never, then ASCII 0-9 will be used. If you select Context, then the native digits will be used sometimes but not others. For instance, in Windows, Arabic and Thai digits are context-sensitive. Even if your locale does have native digits, you may or may not wish to apply digit substitution across the board. You will have to consider the expectations of your locale's intended audience in choosing the digit substitution setting.
You will be asked to provide both group and decimal separators for your custom locale, where the decimal separator is used to separate whole from fractional digits and the group separator is used to separate strings of whole digits into groups. Be aware that these separators should be distinct characters; if they are the same, then it will not be possible to compute the intended value of a numeric string. For instance, in the en-US locale, the decimal separator is a period and the group separator is a comma, while in the fr-FR locale, it is the reverse:
  • 123,456.7890 (digit frmatting fr en-US users)
  • 123.456,7890 (digit frmatting fr fr-FR users)
You will also be asked to provide the increment at which digits should be grouped. In the examples just shown, the group size is 3, meaning that there must be 3 digits between group separator characters.
In addition to specifying basic text properties such as line directionality and default paper size, you will need to designate the Unicode ranges and other code page assignments for your locale. Windows uses this information to tell applications how users will expect their text to be displayed. Be sure to include exactly those Unicode ranges that represent the character set that is required to support your locale's intended users.
Using Unicode is a requirement for developing really globally extensible applications, but we are aware that some developers are still relying on ANSI and other encoding schemes. Locale Builder lets you specify code pages so that your locale's users can also use non-Unicode applications if they need to.
You will also need to specify the Unicode script name for the writing system that your locale requires. The script name that you select is important in making sure that anti-spoofing mitigation works on your machine, so you will want to make sure that your selection reflects the major writing system used for your locale (see Figure 9).
Figure 9 Selecting Script Name
Figure 9 Selecting Script Name
The most challenging part of number formatting is changing between the decimal and grouping separator. Mistaking a period for a comma can cause values to be incorrect by orders of magnitude. It is recommended that applications persist numbers in a binary or application-specified number format. Of course, when displaying numbers it is important to use the user preferences.
Number representation can also use non-ASCII native digits, so your application may encounter characters other than 0-9 as inputs. Avoid filtering on U+0030 through U+0039 to prevent frustration for users who are trying to enter data using non-ASCII digits.

How Locale Data Affect Your Application
So now there's a custom locale installed on your computer. You've selected it as your default in the Regional and Language Options control panel, allowing applications to use your custom data whenever they ask for the user default locale. The question remains: you want to use a custom locale, but how well do your applications work with it?
The discussion about custom locales can be a little bit scary for the application developer. After all, we generally expect users in the U.S. to use a 12-hour clock with A.M. and P.M., for numbers in the U.S. to have periods before the decimal and commas to mark thousands, and we really expect the spelling of January to stay January. With custom locales it is possible for these values to change, even for en-US.
If a user chooses a 24-hour clock, wants spaces instead of commas in their numbers, or even installs a locale that uses janvier instead of January, she probably has a reason for doing so. It is important to distinguish between data representations that are important to the user and those that are important to the application. The intent of the locale data is for presentation, and applications should respect the user or system administrator's choices whenever possible.
Realize that locale data can change. Microsoft locales themselves have evolved in response to changing cultural preferences, new international standards, or feedback from users. What happens to your application when the date pattern changes from mm/dd/yyyy to dd/mm/yy? How will your application respond if the user's long date pattern causes strings like "The week day of Monday, day 14 of the month August (yes, August), of the year 2006"? What if the locale uses Unicode PUA (private use) characters or characters that don't appear in the computer's ANSI code page?
First, assume that locale data can and will change, even for en-US. Honor the user's selected locale, and allow user overrides. The user wouldn't have specified a preference if she didn't intend for it to be used. Remember that different machines may have different versions of locale data, even for en-US, and that applications cannot specify a specific locale data version.
Allow for unexpected values. Numbers may not be parseable, dates may return longer or shorter strings than expected, or values may be incomplete. When persisting data for future machine reading, use a format that is locale-independent. And finally, use the new named APIs for locale information, such as GetLocaleInfoEx and GetDateFormatEx.
Custom locales have an LCID of 0x1000 (LOCALE_CUSTOM_UNSPECIFIED) or 0x0c00 (LOCALE_CUSTOM_DEFAULT). So you can't tell Hawaiian from Fijian from a custom English from Klingon by using the LCID. Therefore avoid any dependencies on the LCID by calling GetLocaleInfoEx instead of GetLocaleInfo.
Replacement locales will have the same string identifiers as the Microsoft locales on which they are based, such as en-US, zh-CN, and so on. Supplemental locales provided by third parties should use the same format. For example, fj-FJ would be the expected string identifier for Fijian (Fiji). While this naming scheme makes it easy to identify the intended use of the locale, custom locales will differ from existing Microsoft shipped, replacement, or custom locales using the same identifier. Hopefully all locales identified as fj-FJ actually do represent Fijian (Fiji), but there's no way to guarantee that the custom locale was designed correctly, or that identical versions are installed on different machines. Even en-US could be customized with data that differs from the en-US data that we ship with Windows.
Custom locales can specify their display names. Variations of similar locales could cause multiple locales with the same display name. For example an en-Latn-US locale could be created identical to the shipped en-US, including the display name "English (United States)." Does your application need to distinguish between these cases in a list?
Another oddity of custom locales is that the concept of a localized name is impractical. If we take the 6,912 languages mentioned before and localize each language name in the other's languages we end up with 47,775,744 strings that would need to be maintained. For that reason calls for localized names of custom locales generally display the native name of the locale.

Time, Dates, and Calendars
Locales don't always use the Gregorian calendar by default, so your application should make sure it can handle non-Gregorian dates. If you only display formatted dates this may not be a problem, but date entry devices should consider whether a non-Gregorian calendar is a possibility.
Most developers are aware of the difference between a MM/dd/yyyy and dd.mm.yy type string, but more variations are possible, like "day 12, of month 08, of year 2006." In some cultures the inclusion of this sort of rendered text in date patterns is quite common. Date entry can also be tricky. A date format like 04/05/06 doesn't allow easy determination of which value is the month, day, and year. Consider specifying the expected format for the user or using a calendar date picking control. In the data we ship, about half of the long date formats have day of week names by default and half do not. Your application may need to add a day of week if necessary.
Time formats are surprisingly consistent across locales, with the obvious exception of preferences for the 12/24-hour clock and associated A.M./P.M. indicators. Time zones, however, are more challenging. Time zones are not, strictly speaking, part of the locale, but users can specify a time zone for their system. The bigger problem here is that some time zones switch between standard time and daylight savings time according to principles that are not always algorithmic.
Time data should often be stored in Universal Time Coordinate (UTC) and converted to local time for display because this allows the smallest chance of confusion about the time and the error. Some cases don't fit this generalization though. Specific appointments can be stored in UTC, but a recurring appointment (for example, a meeting that occurs every Wednesday at 3:00 P.M.) usually needs to carry the time zone information along with it. The UTC offset is going to be different in the winter than the summer due to daylight savings time.
Some events may need local time. For example, if you are interested in behavioral patterns, it might be interesting to know that your customers access your Web site around noon local time. So this may require storing the time zone as well. Even time zone data that are usually predictable can change, as seen with the recent U.S. government decision to extend daylight savings time into November. In those cases the system rules and any persisted rules may need to be updated.
One very common software problem involves switching to and from daylight savings time. Applications developed in the summer can easily find themselves off by an hour in the winter or vice versa. Other applications may miss events that happen during the transition. Make sure your application's design and test plan consider these scenarios.

Text Rendering in a Locale
Numerous interesting things happen to text rendering in various locales. Users of some locales expect text to be written from left to right (such as English, Russian, and Hindi), while others prefer right to left (such as Arabic and Hebrew). Some locales require unique casing rules, like mapping i to I (English), or i to İ (Turkish). Users also have locale-specific expectations around linguistic sorting behavior, so that German speakers expect ähnelte to sort before plante, but Swedish speakers want it the other way around. Finally, although storing Unicode character data allows applications to support the broadest range of locales, some legacy applications and protocols require code page encodings, and different locales may require different encoding schemes.
The likely behavior of right-to-left (RTL) custom locales can be tested by using a locale like Arabic or Hebrew, or the RTL qps-mirr mirrored pseudo locale. The behavior of custom locales requiring large script size can be tested with an Asian locale like ja-JP or zh-CN or the qps-asia East Asian pseudo locale.
Applications should use Unicode when possible, but some legacy protocols may require using code pages like ASCII, windows 1252, or iso-2022-jp. In those cases using a newer protocol that supports Unicode is a good idea. If Unicode is not possible, then your application may run into difficulties if the character repertoire of a locale does not match that of the computer's system locale. Locale Builder does not allow users to change the code page assignments for replacement locales.
Different locales can follow different casing rules. In English i and I are different cases of the same character. In Turkish there are two pairs: ı and I, and i and İ. Applications that assume that i and I are equal could run into trouble. For some operations other operating system or protocol limitations may require use of a specific casing algorithm to avoid multiple conflicting mappings.
Similarly, sorting and comparison rules can differ across locales. In some languages factors like diacritics and capitalization can radically alter sorting expectations, while in other languages such factors are only considered as tie-breakers to put otherwise identical words into some kind of an order.
Comparisons are also complicated by the indexing that is required for a database application. If comparison rules change as they did between Windows XP and Windows Vista, then databases may need to be reindexed. Such changes can happen between custom locale versions. One fj-FJ locale may have chosen en-US as its sorting locale, but another may have chosen fr-FR. Your application can call GetNLSVersionEx to get the current sort version for a specific locale and determine if re-indexing is necessary.

When Not to Use Locale Data
Not all data manipulation is intended for presentation. If your application wants to save a bunch of numbers to a file so that it can read it in later, or send a bunch of data across the Web to a SOAP server, then formatting it with a locale is not the appropriate choice. Remember that the target machine may have a different locale, so the commas and periods in the numbers might be backwards from your expectations. When sending data to a machine or storing it for later retrieval, use a format that is guaranteed to be read back later. Store binary data, use data standards when appropriate, format data with the invariant locale, or store that data using specific formatting strings (like yyyy/MM/dd) that your application will use to read the data later.

Pseudo-Locales for Testing
Sometimes testing data for unfamiliar locales can be difficult. Windows Vista includes three built-in pseudo-locales for use in testing. The qps-ploc pseudo-locale creates longer than average strings and uses unexpected characters outside of the normal en-US Latin range to imitate situations that may occur in other locales. “[Шěđлеśđαỳ !!!], 8 ōf [Μäŕςћ !!] ōf 2006” is an example long date using the qps-ploc locale. Enabling of pseudo-locales on Windows Vista is discussed on the MSDN® site. You may also build your own pseudo or custom locales for testing. Examples might be locales with really short strings or other data interesting to your app.

What's Next?
Microsoft Locale Builder and other similar tools are the first steps towards creating an extensible model for globalization support that allows customers to create personalized user experiences that are appropriate to a broad range of expectations. Microsoft has also released Keyboard Layout Creator, which allows users to create and share customized keyboard layouts, and Transliteration Utility, a tool to allow users to transliterate text from one writing system to another.
These tools are just the beginning. As more and more customers use the tools to create custom solutions, there will be a need for a centralized portal where people can obtain the tools and share the solutions that they create with others of similar linguistic or cultural background. We also recognize that there are other aspects of globalization support that are just as important. We are seeking feedback from our customers as to which aspects of user experience are crucial for them to be able to customize.

Kieran Snyder is a Program Manager in the Windows Globalization team at Microsoft where she drives core language enablement support and extensibility tools across Microsoft products. Kieran holds a PhD in linguistics from the University of Pennsylvania and blogs at blogs.msdn.com/kierans. She can be reached at kierans@microsoft.com.

Shawn Steele is a Software Design Engineer at Microsoft working on Windows and the .NET Framework. He is primarily responsible for culture/locale data (naDev tlhInganpu' tu'lu', Qapla'!), code pages/encodings, normalization, and IDN. Shawn blogs at blogs.msdn.com/shawnste and can be reached at shawnste@microsoft.com.


Editors Note
A Look Into the Future
Joshua Trupin

It's hard to believe it, but we've reached the end of another year here at MSDN Magazine. The December issue is safely tucked away with visions of sugar plums in its head, and our thoughts turn to what 2007 will bring.
In our case, it's pretty easy to look into the crystal ball. Windows Vista is going to be the big news in 2007 for developers, alongside the .NET Framework 3.0. Imagine getting this month's issue a few years ago—when "security" meant that you'd have a button that popped up a fake spreadsheet when your boss walked past your desk.
Windows Embedded CE 6.0? Windows Workflow Foundation? Windows Vista? You'd be lucky to get the upgrade to 640KB of memory. And keyboards? We had to touch the wires together to make zeroes and ones. Each floppy disk was as big as a Buick, and it could only store 15 characters.. The real hackers would cut an extra notch on the side and turn it over to store another 15 characters. We'd scrounge for months to earn enough to buy a genuine Hercules graphics card, which weighed as much as a Buick with a floppy disk on top of it.
Whoa there! We started to lapse into our beloved "crusty old-timer" editorial character again. Let's step back and look at mobility, which just happens to be the theme of this month's issue.
By the time you read this, Windows Embedded CE 6.0 will have been released to manufacturing, maybe even launched. This new version has a bunch of changes and updates. The kernel and memory model have been overhauled. The Platform Builder is integrated into Visual Studio 2005. A lot of the source code is even available for you to look over. It's a great deal, especially when you think back to when we were just breaking into this business years ago. Why, the only mobile computing we did was typing out 7734 on our calculators, then holding it upside down to read the word it spelled. Aaaah, good times.
In any event, 2007 will be the year of Windows Embedded CE 6.0. After you read Paul Yao's article in this issue, you can check out the virtual launch event at ce6launch.com. They'll have a bunch of drilldowns on features of the new version, available on demand when you visit.
It will also be the year of Windows Vista—you don't need a Tarot deck to figure that one out. Among all the cool features of Windows Vista that we're looking forward to, one is the ability to build applications that can be distributed to a global market. This is cheaper than ever before, since it is so easy to sell software online now. Locale Builder is a tool that lets you more easily develop apps that can spread across the globe, helping you conquer your market. Or just help your international users. Of course, back in the day we didn't have international users. You'd have to put that big floppy disk in a container and ship it overseas, and that was it. You kids don't know just how easy you have things today with your fancy Internets and interruptible downloads and dial-up speeds up to 53Kbps depending on carrier. Or maybe that's just the irascible old-timer in us acting up again.
The future is more than just mobility and Windows Vista, of course. In the next year, we'll be bringing you the most coverage of the .NET Framework 3.0 and beyond. There's so much to know about the next version, from Windows Workflow Foundation and Windows Presentation Foundation to LINQ and more, that we'll need at least a year to get through it all. Hang in there, old-timers!
Speaking of old-timers, the upcoming year also brings some change at MSDN Magazine. Grant Duers, who led us through the launch of TechNet Magazine and our largest peacetime expansion in history, is retiring after this issue. We'll all miss Grant and his service around here, and we wish him well as he takes on the surprisingly large world outside of our magazine group. — J.T.

Thanks to to the following Microsoft technical experts for their help with this issue: Jonathan Caves, Amit Chopra, Matt Gibbs, Mike Hall, Dan Javnozon, David Kelley, David Kline, Jennifer Knecht, Sridhar Mandyam, Michael McKeown, Polita Paulus, Doug Purdy, Ayman Shoukry, Dharma Shukla, Don Smith, Sowmy Srinivasan, and Jason Stolarczyk.

ActiveSync, BizTalk, Internet Explorer, MapPoint, Microsoft, MSDN, Outlook, SharePoint, Virtual Earth, Visual Basic, Visual C++, Visual Studio, Windows, Win32, Windows Media, Windows Mobile, Windows NT, Windows Server, Windows Vista, and Xbox 360 are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN MAGAZINE does not make any representation or warranty, express or implied with respect to any code or other information herein. Msdn magazine disclaims any liability whatsoever for any use of such code or other information. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Technology LLC. CMP Technology LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox
Issue and Defect Tracking, Automate IT Tasks, and More
Scott Mitchell

Contents
Issue and Defect Tracking
Automate IT Tasks
Add Spreadsheet Capabilities to Apps
The Bookshelf

Issue and Defect Tracking
During the lifetime of any software project there are various tasks that need to be recorded, assigned, searched, managed, and analyzed. For large projects with several developers working in concert with other stakeholders-testers, customers, architects, and so on-it's imperative that a project management tool be used.
One such tool worth checking out is OnTime 2006 (version 6.1.2) from axosoft. At the core of any project management tool is issue and defect tracking, which enables you to record any outstanding issues or bugs. OnTime 2006 provides a wide array of properties that can be assigned to an issue or defect.
Tracking Defects with OnTime 2006
Tracking Defects with OnTime 2006
OnTime 2006 also offers task lists, feature and enhancement tracking, timesheet capabilities, and can be extended to provide a customer-facing helpdesk. Each of the systems offers support for notifications and history tracking.
Many project management tools provide only a Web-based interface, but OnTime 2006 offers three interfaces: a standalone Windows®-based application, a Web-based interface, and a Visual Studio®-integrated interface. The Windows-based application might be ideal for the project manager to review the outstanding work items and analyze reports, whereas developers might find the Visual Studio interface more conducive to locating and addressing any defects. The Windows and Web-based interfaces include numerous reports that can help project managers forecast deadlines and better manage resource allocation. If needed, you can customize the built-in reports, creating custom filters, or you can build your own reports from the ground up with the OnTime Report Designer.
If you're in the market for a software project management system, consider OnTime 2006.
Price: $149 (and up) per registered user, depending on features.
www.axosoft.com

Automate IT Tasks
There are a variety of scenarios where automating computer tasks involves system, developer, and end user-level domains, or requires performing some action at the user interface level (such as launching a new window, entering some text into a textbox, and clicking a button). For these advanced automation scenarios, check out Automation Anywhere version 3.0, from Tethys Solutions.
In a nutshell, with Automation Anywhere you can arrange any number of actions into a saved task, which can then be executed on demand (according to a predefined schedule), or in response to a trigger, such as when a specified window is closed, or when a particular file is deleted.
Automation Anywhere Task List
Automation Anywhere Task List
Automation Anywhere includes dozens of built-in actions. These include system-level actions-launching programs, creating, deleting, and moving files, performing Internet-related operations (downloading files from an FTP server, sending an e-mail, and so on)-as well as user interface level actions, like prompting the user for input and injecting keystrokes or mouse movements into a specified window. There are also database-related actions that can be used to select from or modify database data.
Whereas many automation solutions require some level of programming experience, Automation Anywhere's actions can be pieced together through a straightforward point-and-click user interface. It also ships with several templates for performing common automation tasks.
Price: $249.95 per Standard edition.
www.tethyssolutions.com

Add Spreadsheet Capabilities to Apps
If you develop internal support software for a company, chances are you've been asked at some point to create a program that works with or generates Microsoft® Excel® spreadsheets. If so, check out SpreadsheetGear for .NET by SpreadsheetGear LLC.
Generating Excel Spreadsheets
Generating Excel Spreadsheets
The software works with both Windows-based and ASP.NET applications and offers a rich API for creating, reading, and modifying Excel spreadsheets. A new spreadsheet can be created or an existing one opened with just a single line of code. With a few more lines, the spreadsheet's data can be modified, read, copied from one worksheet to another, and so on. Formulas can be specified and charts can be created. Saving the spreadsheet-or writing all or part of its contents into an ADO.NET DataSet-can likewise be accomplished with just one line of code.
SpreadsheetGear for .NET can be used in ASP.NET applications to easily display existing or modified Excel reports in a Web page. Simply read the Excel spreadsheet data into a DataSet and bind it to a GridView or DataGrid Web control. Additionally, with a few lines of code you can export your current Web-based reports as Excel reports.
For developers using Windows with the Microsoft .NET Framework 2.0, the WorkbookView and FormulaBar controls allow a spreadsheet-like UI to be embedded directly within a Windows-based application. With these controls, you can offer a familiar user experience with underlying data that can be modified, charted, saved to disk, or exported to a DataSet through SpreadsheetGear for .NET's straightforward API.
Best of all, SpreadsheetGear for .NET generates native Excel spreadsheets without an Excel installation.
Price: $799 per developer license.

The Bookshelf
AJAX-or Asynchronous JavaScript and XML-describes an approach and underlying technologies for creating responsive Web applications. The challenge of creating an AJAX-enabled application is that it requires expertise in both client-side and server-side technologies. Furthermore, the client-side script used to make the asynchronous calls and to modify the Web page layout can differ subtly between different browsers and browser versions.
This complexity is commonly encapsulated by an AJAX-specific library. Microsoft has released an AJAX-development library called ASP.NET "Atlas" that enables programmers to create interactive Web applications using the familiar ASP.NET Web controls and syntax. (Note that the pieces that make up Atlas have recently been given official names by Microsoft, including the Microsoft AJAX Library, the ASP.NET 2.0 AJAX Extensions, and the ASP.NET AJAX Control Toolkit.) At the time of this writing, Atlas is provided as a Community Technical Preview and has not yet been finalized or officially integrated into the ASP.NET base classes.
For those itching to use Atlas to create responsive ASP.NET applications, check out Foundations of Atlas by Laurence Moroney (Apress, 2006). The book begins with a good overview of AJAX, followed by a look at how Atlas simplifies AJAX development by providing a set of client and server controls that work together to emit the needed client-side script and server-side functionality to participate in asynchronous postbacks.
Tracking Defects with OnTime 2006
Tracking Defects with OnTime 2006
The next several chapters focus on examples and walkthroughs that show how to use the Atlas controls, including an entire chapter on data binding. The last chapter looks at building an AJAX-enabled financial research Web application from the ground up. The chapter does a great job at tying together the lessons learned throughout the book and illustrating a real-world benefit from using AJAX techniques with Atlas.
As with any early adopter book, Foundations of Atlas may contain source code that does not compile in the final version. Fortunately, the author has a section of his blog dedicated to discussing Atlas and addressing any corrections in this book.
Price: $39.99.
www.apress.com

All prices were confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott MitchellScott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


Advanced Basics
TableLayoutPanels
Ken Getz

Code download available at: AdvancedBasics2006_12.exe (169 KB)
Browse the Code Online
Contents
Using the TableLayoutPanel Control
Helper Procedures
The Display Menu Procedure

I've been writing quite a bit of courseware lately, and many of the demos for the courses require the presenter to step through a number of simple procedures, demonstrating various features of Visual Studio® or the Microsoft® .NET Framework. For example, I recently worked on a course in which the demos ran as console applications. The students saw a menu with lettered prompts, and by pressing a letter (A through Z), the demo ran the appropriate sample procedure.
On a long plane trip home one day, I crafted a simple menuing system that allowed the courseware author (generally, me) to supply a list of objects containing prompts and a delegate instance that indicates the procedure to run when the user selects the item. The code lays out the menu, even breaking the list into two columns if it gets too tall. It worked flawlessly, and I never thought about how to display a demo's menu again.
Now that I'm back to creating Windows®-based applications for demos, I thought it might be nice to have the same sort of self-generating demo system in place. The current course (and almost every conference talk I give) requires a large number of demo applications, and not having to lay out the main switchboard form for each application can save time. It's a lot simpler to modify a single line of code than it is to manage laying out the main form.
With this goal, I put together a demo-creation system for Windows-based applications and it's the basis for this month's column. Figure 1 shows sample code that can be used to generate the switchboard in Figure 2.
Figure 1 Creating the Switchboard
Public Sub Main()
  Dim menu As New DemoMenu
  menu.Add("Work with Strings", AddressOf WorkWithStrings)
  menu.Add("String Methods", AddressOf StringMethods)
  menu.Add("StringBuilder Class", AddressOf StringBuilderClass)
  menu.Add("Dates and Times", Address_Of DateAndTimes)
  menu.Add("DateTimeFormatInfo", _
           AddressOf DateTimeFormatInfoClass)

  menu.Add("", Nothing)

  menu.Add("TimeSpan", AddressOf TimeSpanDemo)
  menu.Add("StopWatch Class", AddressOf StopwatchDemo)

  menu.DemoFont = New Font("Verdana", 8, FontStyle.Regular)
  menu.MaxItemsInSingleColumn = 10

  menu.DisplayMenu("Demo Menu")
End Sub
Figure 2 Menu
Figure 2 Menu
If you want your menu to display its items in more than one column, you can change the MaxItemsInSingleColumn property of the DemoMenu class. For example, if you change the property value to 4 (its default value is 10), the same form shown previously displays its buttons as you see in Figure 3.
Figure 3 Menu Items in Columns
Figure 3 Menu Items in Columns
For use in courseware, I include the DemoMenu project along with the course samples, and have each demo refer to the compiled DemoMenu assembly in a known location. You can deploy this assembly as you deploy any other shared assembly-but that's a topic for a different column.
The DemoMenu class creates an internal list of objects that maintain the prompt text and the address of the procedure to be called by each button click. The code creates a new form, lays out the buttons, and hooks up the Click event handlers. My first attempt at writing this code (on the Boston-to-Minneapolis leg of my trip back to the West Coast) simply created buttons on a form. I used an internal offset value to space the controls on the form, and I didn't add support for multiple columns. During the layover, it occurred to me that multiple columns would be nice, and as soon as I got back on the plane again, I realized that laying the controls out manually was too much effort. The answer was obvious-the TableLayoutPanel control, new in the Microsoft® .NET Framework 2.0, makes this sort of thing relatively simple.

Using the TableLayoutPanel Control
The TableLayoutPanel control is the most advanced, and the most complex, of the set of container controls that you'll find in Windows Forms 2.0. (This set also includes the Panel, SplitContainer, TabControl, and FlowLayoutPanel controls.) This control allows you to arrange other controls in a grid-like fashion, with a fixed number of rows and columns. This control is richer than I can begin to demonstrate here, and its capabilities far outstrip the needs of the simple switchboard form. In this example, the TableLayoutPanel control provides one important function: it creates neatly spaced, multi-columned cells in which the code can place buttons. The code doesn't need to worry about aligning cells, or laying out the spacing-the TableLayoutPanel control does that formatting. (You may find it useful, before digging into this column, to create a simple form with a TableLayoutPanel control on it. Try working with the various user interface features of the control. Add rows and columns, place controls within cells, set the height of rows and the widths of columns, and try out the various sizing options.)
Because the sample project doesn't load its own form as it starts up (it calls the DemoMenu class's DisplayMenu procedure to display the main menu), the project doesn't use a startup form, as do most Visual Basic®-based applications. Instead, this application starts from a Sub named Main. In order to make this possible, the sample project, TestDemoMenu, disables the standard application framework, and sets the Startup object option to Sub Main, as shown in Figure 4.
Figure 4 Disable the App Framework to Start from Sub Main
Figure 4 Disable the App Framework to Start from Sub Main
The sample application includes a reference to the DemoMenu project, which includes the DemoMenu class. In order to use the DemoMenu class, you start by creating an instance, and then adding items to the menu by specifying text for each button, along with an instance of the EventHandler delegate to be called by the menu when you click each button:
Dim menu As New DemoMenu
menu.Add("Work with Strings", AddressOf WorkWithStrings)
menu.Add("String Methods", AddressOf StringMethods)
' and so on...
If you want to insert an empty space within the list of demo buttons, you can add an empty item:
menu.Add("", Nothing)
(If you'd like, you can also create your own List of DemoItem objects, and assign this list to the MenuItems property of the DemoMenu instance. Most of the time, however, it's easiest to just call the Add method to create each individual item.)
You can also set any of the public properties of the DemoMenu class, including the Font, MaxItemsInSingleColumn, Text (of the menu form), or Form (this allows you to supply your own form on which to place the buttons, which you'll need to do before you call the DemoMenu.DisplayMenu method). The sample project includes this code to complete laying out the form before it calls the DisplayMenu method (setting the form's Text property in its call to DisplayMenu):
menu.Font = New Font("Verdana", 8, FontStyle.Regular)
menu.MaxItemsInSingleColumn = 4
menu.DisplayMenu("Demo Menu")
The DemoItem class describes the contents of each demo item, as shown here:
Public Class DemoItem
  Public ItemText As String
  Public ItemProc As EventHandler

  Friend Sub New(ByVal ItemText As String, _
      ByVal ItemProc As EventHandler)
    Me.ItemText = ItemText
    Me.ItemProc = ItemProc
  End Sub
End Class
Note that the DemoItem.ItemProc property contains an EventHandler delegate instance. This delegate type corresponds to the type required for the Button class's Click event handler. If you attempt to set any other type of delegate for the ItemProc property, your code won't compile-that's the point.
The DemoMenu class includes two private variables, along with two public fields:
Private Shared formCount As Integer = 0
Private frm As Form

Public MenuItems As New List(Of DemoItem)
Public MaxItemsInSingleColumn As Integer = 10
The shared formCount variable allows the class to provide unique behavior for the first instance of this class within a running application-you'll see more about this when you investigate the code in the class. The MenuItems variable contains the list of menu items (you can set this directly, if you like), and the variable frm maintains a reference to the form containing the buttons. Finally, the public MaxItemsInSingleColumn field allows the caller to override the default behavior, which, as you saw, allows up to 10 items in a vertical column on the form.
The class's constructor simply sets up the default form:
Public Sub New()
  ' Set up the default form
  frm = New Form
  frm.Font = New Font("Verdana", 10)
  frm.Text = "Main Menu"
  frm.FormBorderStyle = FormBorderStyle.FixedDialog
  frm.MaximizeBox = False
  frm.MinimizeBox = False
End Sub
The Add method provides support for adding menu items to the data structure that maintains the menu items:
Public Sub Add(ByVal itemText As String, ByVal itemProc As EventHandler)
  Dim item As New DemoItem(itemText, itemProc)
  MenuItems.Add(item)
End Sub
In addition, the DemoMenu class provides three simple property procedures for the Form, Font, and Text properties, for getting and setting the form reference, as well as for properties of the default form.
In the class, you'll find an override for the DisplayMenu procedure that accepts a caption for the form, and then calls the full DisplayMenu procedure:
Public Sub DisplayMenu(ByVal Text As String)
  Me.Text = Text
  DisplayMenu()
End Sub

Helper Procedures
Before examining the main DisplayMenu procedure, it's important to understand how each of its support procedures works. The CalculateDimensions procedure calculates how to break the demo items into multiple columns, attempting to minimize the number of columns given the value for MaxItemsInSingleColumn. The procedure (shown in Figure 5) allows a maximum of five columns-if it can't fit the number of demo items you've created into five columns, it simply raises an exception and quits. (You'll know the first time you run the demo if it's going to fail, at which time you can rethink the number of items to display on the main menu. Note that you can have a button on the menu display another dynamically generated menu, so the column limitation isn't terribly onerous.)
Figure 5 CalculateDimensions Determines Columns
Private Function CalculateDimensions(ByVal totalCount As Integer) As Size
  ' Calculate the dimensions of the TableLayoutPanel control:
  Dim rows As Integer = totalCount
  Dim cols As Integer = 1

  ' If you can fit all the demos in a single column, you're all set.
  If totalCount > MaxItemsInSingleColumn Then
    cols = CInt(Math.Ceiling(totalCount / MaxItemsInSingleColumn))
    If cols > 5 Then
      ' Won't fit, so get out!
      Throw New InvalidOperationException("You have too many items!")
    End If
    rows = CInt(Math.Ceiling(totalCount / cols))
  End If
  Return New Size(cols, rows)
End Function
The CalculateDimensions procedure is passed an integer containing the total number of items to be displayed. It checks to see if the total number is less than the largest number per column and if so you're finished.
Once the procedure determines that it must calculate the number of columns, it looks for numbers of rows and columns that fit the number of items it must display, minimizing the number of columns. If it is unable to find an appropriate combination of rows and columns, it raises an exception back to the caller. The procedure completes by returning a new Size instance containing the numbers of rows and columns.
Given that the list of items is a one-dimensional array, but you may be working with buttons in a two-dimensional layout, much of the remainder of the code handles moving items into a set of two-dimensional data structures. The ArrangeItems procedure (see the code in Figure 6) takes items from the original list and returns a two-dimensional array of DemoItem instances, ready for use by other procedures.
Figure 6 Create a Two-Dimensional Array
Private Function ArrangeItems(ByVal rows As Integer, _
    ByVal cols As Integer) As DemoItem(,)

  ' Return array of DemoItem instances that matches
  ' the layout of the form:
  Dim items(cols - 1, rows - 1) As DemoItem

  ' Fill in the items array:
  Dim currentItem As Integer = 0
  For col As Integer = 0 To cols - 1
    For row As Integer = 0 To rows - 1
      If currentItem < MenuItems.Count Then
        items(col, row) = MenuItems(currentItem)
        currentItem += 1
      End If
    Next
  Next
  Return items
End Function
The ArrangeItems procedure starts by creating the output array. Note that because the buttons descend columns first, all the array access uses columns first (unlike most array processing, which is row by column). The code keeps track of the current item, loops through all the columns and rows, and copies references from the original List of items into the new array. As it's looping, the code checks to make sure it hasn't run out of items. Finally, the procedure returns the new array.
In order to make sure the buttons are wide enough to display their text, the code must calculate the size of the text on the buttons in the font of the form. In addition, the DemoMenu class attempts to allow different columns of buttons to have different widths. To make this possible, the CalcSizes procedure (shown in Figure 7) creates an array of SizeF structures, one for each column in the form.
Figure 7 Calculate Sizes for Each Column
Private Function CalcSizes( _
    ByVal rows As Integer, ByVal cols As Integer, _
    ByVal items As DemoItem(,)) As SizeF()

  ' Calculate the width of each column:
  Dim sizes(cols - 1) As SizeF

  ' Calculate the width of each column individually.
  ' Just keep track of the maximum width and height,
  ' and use those values:
  Using g As Graphics = frm.CreateGraphics
    For col As Integer = 0 To cols - 1
      Dim maxSize As New SizeF(0, 0)
      For row As Integer = 0 To rows - 1
        ' Get the DemoItem object, and calculate
        ' its font metrics:
        Dim menuItem As DemoItem = items(col, row)
        If menuItem IsNot Nothing Then
          Dim fontSize As SizeF = _
            g.MeasureString(menuItem.ItemText, frm.Font)
          maxSize.Width = Math.Max(maxSize.Width, fontSize.Width)
          maxSize.Height = Math.Max(maxSize.Height, fontSize.Height)
        End If
      Next
      ' Use the maximum size you got for this column:
      sizes(col) = maxSize
    Next
  End Using
  Return sizes
End Function
The CalcSizes method first creates the output array. Next, the code asks the form to create a Graphics object to perform the font size calculations. The code loops through each column, initializing a temporary SizeF structure and then looping through each row in the column. Inside the loop, the code will calculate the size of the text for each button and store the maximum size required in each column. Once it's finished with each column, the code stores the maximum size in the array of sizes.
For each row, the code retrieves the menu item and calculates its size, using the form's font. The Graphics.MeasureString method does the work. Given the maximum size, the outer loop can store the SizeF structure for each column, for later use.
The FillTable procedure (see Figure 8) creates the actual buttons and hooks up their event handlers within the TableLayoutPanel control. This procedure loops through all the columns and rows, and if the item in the array of items isn't Nothing, checks the item's ItemText property. If it's not an empty string, the code creates a button; otherwise, it leaves blank the position within the TableLayoutPanel control. If the code determines that it needs to create a button, it creates the control, sets its properties, hooks up its event handler, and adds the control to the appropriate location within the TableLayoutPanel control.
Figure 8 FillTable Procedure
Private Sub FillTable(ByVal items As DemoItem(,), _
    ByVal rows As Integer, ByVal cols As Integer, _
    ByVal tbl As TableLayoutPanel)

  For col As Integer = 0 To cols - 1
    For row As Integer = 0 To rows - 1
      Dim demo As DemoItem = items(col, row)
      If demo IsNot Nothing Then
        If Not String.IsNullOrEmpty(demo.ItemText) Then
          Dim btn As New Button
          btn.Text = demo.ItemText
          btn.Dock = DockStyle.Fill
          AddHandler btn.Click, demo.ItemProc
          tbl.Controls.Add(btn, col, row)
        End If
      End If
    Next
  Next
End Sub
The SetupPanel procedure creates the TableLayoutPanel control and sets its row heights and column widths (see Figure 9). It starts by creating TableLayoutPanel, docking it, setting the rows and columns, and clearing the default collections of row and column styles. In this control, it's the RowStyles and ColumnStyles properties that determine the behavior of the control. A row or column can have a fixed size, a percent size (that is, a percentage of the available space in the control), or it can be automatically sized (based on the contents of the row or column). For this app, the rows need to be of equal height and each column needs to be a distinct fixed width. In addition, the form must conform to the size of the TableLayoutPanel control's contents, so this procedure needs to calculate the size of the contents of the control.
Figure 9 SetupPanel Creates TableLayoutPanel Control
Private Function SetupPanel( _
    ByVal rows As Integer, ByVal columns As Integer, _
    ByVal sizes As SizeF()) As TableLayoutPanel

  ' Create the TableLayoutPanel and set
  ' its properties:
  Dim tbl As New TableLayoutPanel
  tbl.Dock = DockStyle.Fill

  tbl.RowCount = rows
  tbl.ColumnCount = columns
  tbl.RowStyles.Clear()
  tbl.ColumnStyles.Clear()

  ' Keep track of the total width and height,
  ' so you can set the form's size:
  Dim totalWidth As Single = 0
  Dim totalHeight As Single = 0

  ' Create variables for use within the loops:
  Dim height As Single
  Dim width As Single

  ' Set all rows to the same height.
  For i As Integer = 0 To rows - 1
    ' The 1.8 factor just gives a little space.
    ' It's an arbitrary value.
    height = CSng(Math.Ceiling(sizes(0).Height * 1.8))
    tbl.RowStyles.Add( _
      New RowStyle(SizeType.Absolute, height))
    totalHeight += height
  Next

  ' Set the width of each column appropriately:
  For i As Integer = 0 To columns - 1
    ' The 1.2 factor just gives a little space.
    ' It's an arbitrary value, and seems to work
    ' with most font sizes:
    width = CSng(Math.Ceiling(sizes(i).Width * 1.2))
    tbl.ColumnStyles.Add( _
      New ColumnStyle(SizeType.Absolute, width))
    totalWidth += width
  Next

  ' Set up the form:
  frm.Controls.Clear()
  frm.Controls.Add(tbl)
  frm.ClientSize = _
    New Size(CInt(totalWidth), CInt(totalHeight))

  Return tbl
End Function
The code continues by setting up variables to track the total width and height of the table, along with variables to track the height and width of individual rows and columns. The code loops through all the rows, setting each to the height of the first row of text. Because all the buttons use the same font, they all fit within the same height. This code multiplies the height by a factor of 1.8, allowing room for space above and below the text on each button. Within the loop, the code sums up the heights and stores the values in the totalHeight variable.
The next block of code sets the width of each column, multiplied by an arbitrary factor of 1.2, giving room on either side of the text on the button. This code sums up the widths of all the columns in the totalWidth variable. The procedure finishes up by clearing any existing controls from the form, adding the TableLayoutPanel control to the form, setting the size of the form, and returning the TableLayoutPanel control to the caller.

The Display Menu Procedure
Given all these helper procedures, the DisplayMenu procedure doesn't have much work to do (see Figure 10). This procedure starts by calculating the dimensions of the table, calling the CalculateDimensions procedure shown earlier. The code continues by setting up temporary data structures containing the menu items in a two-dimensional structure and an array of column widths. The code creates, configures, and fills the TableLayoutPanel control. Finally, the code displays the form. If this is the first form in the application, the code must call the Application.Run method, but for further instances of the class the code simply needs to display the form. In order to make this possible, the code maintains a shared variable, formCount, which tracks the number of instances of the form that have been created. If the value is 1, the code calls Application.Run; otherwise, it calls the ShowDialog method of the form.
Figure 10 DisplayMenu Sets Up the Menu and Displays It
Public Sub DisplayMenu()
  ' Calculate the number of rows and columns:
  Dim tableSize As Size = CalculateDimensions(MenuItems.Count)
  Dim rows As Integer = tableSize.Height
  Dim cols As Integer = tableSize.Width

  ' Create temporary data structures to perform
  ' necessary calculations:
  Dim tempItems(,) As DemoItem = ArrangeItems(rows, cols)
  Dim tempSize() As SizeF = CalcSizes(rows, cols, tempItems)

  ' Set up the TableLayoutPanel control:
  Dim tbl As TableLayoutPanel = SetupPanel(rows, cols, tempSize)

  ' Fill in the TableLayoutPanel.
  FillTable(tempItems, rows, cols, tbl)

  ' Show the form
  formCount += 1
  If formCount = 1 Then
    Application.Run(frm)
  Else
    frm.ShowDialog()
  End If
End Sub
Using the TableLayoutPanel control in this way, the DemoMenu class can lay out its buttons in one or more columns without requiring you to perform exacting location calculations. You set the widths and heights of the columns and rows, and the TableLayoutPanel control does the rest. All you need to do is supply the list of DemoItem instances, either by creating the list yourself or (as shown here) calling the DemoMenu.Add method to add each item to the list. You supply the text of the button and the procedure you want to call when you click the button, and the DemoMenu class does the rest.
I can think of several ways in which you could extend this demonstration. You could support reading the menu items from a configuration file so that you could make changes to the application without recompiling it. You could add support for different control types besides buttons, and you could add behavior that would allow the TableLayoutPanel control to display within a container on the menu form. I'm sure I'll continue to enhance this functionality, so stay tuned. In the meantime, if I keep writing courseware and conference demos at the rate I've been going, this assembly will save me a ton of time. I'm sure it will save you time as well.

Send your questions and comments to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies, and a courseware author for AppDev (www.appdev.com). He is coauthor of ASP.NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2002), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


CLR Inside Out
Inside Diagnostic Tools for .NET
Jonathan Keljo

Contents
Performance Profilers
Memory Profilers
Purpose-Built Profilers
Diagnostic Tools
Profiling API Tips and Tricks

Many diagnostic tools use the CLR Profiling API-even those that aren't strictly profilers. So if you've ever wondered how these tools work, a look at the Profiling API is a good start. In this column, you'll see how they work and look at some useful tips and tricks. You'll also find some essential resources in the "Other Profiling Resources" sidebar.
To use the CLR Profiling API, you create a DLL using an unmanaged language-typically C++, then you set some environment variables that instruct the common language runtime (CLR) to load the DLL and allow it to use the Profiling API. When loaded, this DLL effectively becomes an extension of the CLR itself, receiving callbacks, requesting information, and making changes deep within the implementation of the CLR. The Profiling API can provide notification of many activities within the CLR and managed code, including the creation and destruction of appdomains, loading and unloading assemblies, JIT compiling functions, executing functions, throwing and catching exceptions, and doing garbage collections. Using the Profiling API, you can get information about parts of the application, such as names and locations of assemblies, descriptions of types and functions, and locations and layout of objects in memory. Finally, you can use the Profiling API to modify settings, instructions, and the like, including disabling optimizations in the JIT compiler, changing the intermediate language (IL) for a function, or even creating new types and functions.
As you'll see, the combination of these capabilities can power a broad range of diagnostic tools.

Other Profiling Resources
Here are the promised resources. Please note that if you're trying to get samples from the older articles to work with the .NET Framework 2.0, be sure to follow Tip 4. While you're reading articles, also keep in mind that when you first open or create a project in Visual Studio, you should go back and read Tips 1 and 2 .If you can't seem to get your profiler to load, go back and read Tip 4. And if you encounter strange behaviors when you're running your profiler, reread Tip 3.

Overviews of Profiling API functionality with sample code:
  • Under the Hood: The .NET Profiling API and the DNProfiler Tool
  • CLR Profiler: No Code Can Hide from the Profiling API in the .NET Framework 2.0

More detailed walkthroughs:
  • .NET CLR Profiling Services: Track Your Managed Components to Boost Application Performance
  • Design: Inspect and Optimize Your Program's Memory Usage with the .NET Profiler API
  • http://Profiling API official documentation

Changing a function's code with a profiler:
  • .NET Internals: Rewrite MSIL Code on the Fly with the .NET Framework Profiling API

Writing a profiler:
  • David Broman: blogs.msdn.com/davbr
  • Jonathan Keljo: blogs.msdn.com/jkeljo
  • Rico Mariani: blogs.msdn.com/ricom

Sample profilers:
  • CLR Profiler, a memory profiler
  • CoverageEye.NET, a code-coverage tool

Performance Profilers
Performance profilers are diagnostic tools that show how an application spends its time. If some operations in an application are running too slowly, a profiler can identify where that's happening so you can focus the optimization effort.
There are two main categories of performance profilers: tracing and sampling. The basic task of a tracing profiler is to log the time of every function call or return as an application runs. From this log, the profiler knows how control flow moves through the application, and how the time is spent. In the Visual Studio® 2005 Performance Tools, choosing Instrumentation from the Performance Wizard specifies a tracing profiler (see Figure 1).
Figure 1 Performance Profiling is Visual Studio 2005
Figure 1 Performance Profiling is Visual Studio 2005
In practice, especially in a managed code app, there are a lot of function calls. Even if the profiler can very quickly log each function entry and exit, the overhead adds up and can cause the application to run very slowly during profiling. Not only does this make profiling painful for the developer, it also drastically changes the timing of the app, revealing or hiding subtle race conditions or even causing timeouts. As a result, tracing profilers generally allow you to choose which modules or functions are interesting to profile.
At a minimum, a tracing profiler logs the name of the function being entered or exited, and some measure of the time. That might be the clock time (or "wall-clock time") or it might be the actual number of CPU cycles spent. The profiler can also log other information, such as the arguments passed to the function, to help you better understand the application. Of course, the more information it logs, the longer the logging takes, and the greater the overhead.
There are many ways to write a tracing profiler for .NET. Some, which don't use the Profiling API at all, open the assemblies on disk and insert log-writing code at the beginning and end of functions. This approach is analogous to the way tracing profilers were written for unmanaged code.
Even those who do use the Profiling API have options. With a couple of function calls, a profiler can ask for callbacks from the CLR every time a function is called or returns, and every time an exception passes through a function. The profiler can even choose to receive these callbacks only for certain functions, or to obtain information about the functions' arguments and return values. The profiler then does its logging from inside these callbacks.
If the built-in enter/leave callbacks aren't powerful enough, a profiler can also use the profiling API to modify a function's code before it gets JIT-compiled. This approach is similar to modifying the assemblies on disk, except that it occurs at runtime in memory and doesn't require the extra step of creating the instrumented assemblies before the profiling run.
Sampling profilers use simple statistics to enable the profiling of an entire application without the high overhead of a tracing profiler. As an application runs, a sampling profiler periodically checks (or samples) to see what the app is doing. The simplest sample just records which function is executing at the time. At the end, with hundreds or thousands of samples accumulated, the profiler analyzes how many samples appear in each function and extrapolates from that to estimate how much time is being spent in those functions. For example, if 25 percent of the samples occurred in function Foo, it's likely that the application spent 25 percent of its time running Foo. With enough samples, such estimates can be fairly accurate, and if the act of taking a sample is fast enough, this kind of profiling will not slow down the application very much. More advanced sampling profilers also record the call stack at the time of the sample, to give you a better idea of context. Choosing "Sampling" from the Visual Studio 2005 Performance Wizard lets you use a sampling profiler. Sampling profilers use various techniques to actually take the sample. Some create a new thread that sits in a loop sleeping, periodically waking to see what the other threads are doing. Others work with the operating system kernel to receive an interrupt on a regular basis.
Regardless of how the profiler gets control, it finds the current native-code instruction pointer, uses the CLR Profiling API to determine whether that instruction is actually in JIT-compiled managed code and, if so, which function it's in. Profilers that take call stacks for samples also use the Profiling API to find the managed-code functions on the stack.

Memory Profilers
Memory profilers examine an application's memory usage. In unmanaged code, most memory profilers focus on finding instances where developers had forgotten to free memory or where they'd written to memory they hadn't allocated. While the CLR's built-in bounds checking and garbage collector make such problems disappear, an application's memory usage can still lead to performance problems, especially if it forces the garbage collector to work in suboptimal ways. In fact, when investigating a performance problem in a managed application, it is often best to start by looking at its memory usage rather than how it spends its time.
Managed memory profilers typically keep track of memory usage over the lifetime of an application. They track objects from their creation, as they're moved around by various garbage collections, until they are finally released. A developer can use this data to recognize problematic allocation patterns and clean up the application's memory usage. The Visual Studio 2005 Performance Tools can profile memory usage while performing a sampling or tracing profile; in the property pages for your performance project there are checkboxes for enabling memory profiling. CLR Profiler is a dedicated memory profiler tool and it comes with source code, so it's also a great example profiler as seen in Figure 2.
Figure 2 Memory Profiling in Visual Studio 2005
Figure 2 Memory Profiling in Visual Studio 2005
Managed memory profilers lean heavily on the CLR Profiling API, which notifies them of object allocations, enables them to discover the size and layout of objects, and tells them how objects are moved and deleted during garbage collections.

Purpose-Built Profilers
Occasionally you'll need to get a piece of information that's not easily obtainable from commonly available profilers. Sometimes, you can just put some tracing statements into your code, but that can be prohibitively expensive for large applications. If you're familiar with the CLR Profiling API, however, you can often build a custom profiler to answer a specific question. Here are some questions your profiler can answer:
  • Are any objects being finalized after they've already been disposed? Are there disposable objects being finalized without being disposed?
  • What does the call graph of my program look like?
  • Who is throwing all the handled exceptions?

Diagnostic Tools
The power of the Profiling API-particularly its ability to change a function's code or introduce whole new types-is the backbone behind many tools beyond just profilers. In fact, the API should probably have been given a much more generic name. I hear about new types of tools being built with the Profiling API on a regular basis; here are a few of my favorites.
Fault Injectors Much of an application's code is devoted to handling errors and corner cases. Correspondingly, a large part of its testing is devoted to error-handling code, which is difficult because error conditions are often hard to produce on demand. Fault injectors help exercise failure paths in an application by forcing failures.
There are many ways to build a fault injector. One way is using the Profiling API's ability to change the code of a function before it's JIT-compiled. For example, the fault injector can simulate an out-of-memory condition by changing a function to throw an OutOfMemoryException instead of allocating an object, or it can simulate a network outage by replacing a call to a network function with a throw of a TimeoutException. Figure 3 shows an example.
Figure 3 Typical Fault Injection Modification

Original Code
void Foo()
{
    StringBuilder sb;
    sb = new StringBuilder();
    ...
}

After Fault Injection
void Foo()
{
    StringBuilder sb = null;
    throw new OutOfMemoryException();
    ...
}
Code-Coverage Tools Code-coverage tools are similar to tracing profilers, but instead of measuring time, code coverage tools merely record which chunks of code are executed. Since so much of testing is about exercising corner cases, measuring the code coverage of an application's test suite is one way of estimating thoroughness of testing. If a function is never executed during an entire test run, then the function can't have been tested. CoverageEye.Net, available from a GotDotNet Workspace, is a code-coverage tool built on the CLR Profiling API (see Figure 4), and its accompanying source code is another great sample of Profiling API usage.
Some code-coverage tools only want to know which functions were executed, and can thus use the profiling API's built-in enter/leave callbacks. Others go deeper and record which paths have been taken through a function by inserting logging statements into the function's code.
Figure 4 Code Coverage Data from CoverageEye
Figure 4 Code Coverage Data from CoverageEye
Flight Recorders Flight recorders are similar to tracing profilers and coverage tools, but their goal is more far-reaching-to record as much information about the execution of an application as possible, producing a log of activity. You can then peruse this log after a failure occurs, looking for clues as to its cause.
Flight recorders will typically take information from wherever they can get it-tracing infrastructures like Event Tracing for Windows® (ETW), the system event log, window messages, and so on. The Profiling API provides a treasure trove of information about an application's execution.
Aspect Weavers Aspect-oriented programming (AOP) lets you reduce the number of places in your code that you need to repeat the same logging tasks. For example, if many functions in an application need to write log entries, in a traditional programming language you would copy and paste logging code into each of those functions. Even if the logging code makes use of a logging utility class, certain kinds of updates would require you to touch every function with logging code in it.
In an aspect-oriented approach, you can place a logging aspect on any function that needs logging. In a managed language, the logging aspect may be a new attribute type, say LogAttribute. Then you write code just once to implement the aspect. This aspect-implementation code typically works with a tool called an aspect weaver to insert the logging code into any function with a LogAttribute.
One way to build an aspect weaver is using the Profiling API's ability to add new types and to modify the code for functions prior to JITing. I tend not to recommend this approach, because only one DLL can use the profiling API at a time. If that single DLL is an aspect weaver, you can't simultaneously use a performance profiler to measure your aspect-oriented application.

Profiling API Tips and Tricks
1. Have the build register your profiler Before you can run your profiler, you have to put some information about it in the registry. This is easy to forget, so it's best to configure the Visual C++® build process to do it for you. In the property pages of your project, set a Post Build Event to do "regsvr32 /s $(TargetPath)".
2. Start your profiler with F5 in Visual Studio 2005 To start an application under a profiler, you set a couple of environment variables that specify which profiler to use and then run the application in that environment. If you're using Visual C++ 2005 to develop your profiler, you can speed up the build/run/debug cycle by having Visual Studio set these environment variables for you when you launch an application from within the IDE. In the property pages of your C++ project, in the Debugging pane, enter information about the target application and the profiler environment variables. Make sure you set Debugger Type to Native Only.
3. Strange behavior? Stop when Win32 exceptions are thrown When a profiler DLL gets loaded into a process, it effectively becomes an extension of the CLR. While the CLR always erects barriers before transitioning into managed code, it seldom does anything special before calling a function in a profiler DLL-such functions are effectively treated like functions within the CLR!
When an exception-say, an access violation (AV)-comes out of a profiler function, it emerges in the middle of a function within the CLR that may not have been expecting an exception at that point. That can lead to a variety of strange behaviors, including drastic behavior changes in the application being profiled.
So, if you run into strange behavior when you're running an application under your profiler, set the debugger to stop when native exceptions are thrown by opening the Exceptions dialog and clicking the Thrown checkbox next to Win32 Exceptions.
Now when you're running your profiler under the Visual Studio debugger, it will stop as soon as it sees an AV or any other Win32® exception. If you see one of those getting raised inside your profiler, or inside a function called by your profiler, you've probably found the cause of the weird behavior.
In managed code in particular, a Win32 AV doesn't always indicate a big problem. For example, in many cases NullReferenceExceptions begin life as Win32 AVs. Since you're debugging in native-only mode, you won't see the managed exceptions, and it may be difficult to tell whether an AV is coming from a bug in your profiler. As a result, you may want to use this trick only when you run into strange behaviors that might be caused by exceptions in profiler code.
4. Implement ICorProfilerCallback2 If you're writing a profiler for the .NET Framework 2.0 or the .NET Framework 3.0 (which contains the former), make sure your profiler class inherits from ICorProfilerCallback2 (even if all the methods just return E_NOTIMPL), and that the QueryInterface method responds to requests for IID_ICorProfilerCallback2. The CLR 2.0 has a few concepts that didn't exist in the CLR 1.x-such as generics-and the way a profiler signals to the CLR that it can handle those concepts is by responding to a QueryInterface for the new interface.

Send your questions and comments to clrinout@microsoft.com.

Jonathan Keljo is a Program Manager on the CLR team. During his time with the CLR, he's worked on the Debugging API, exception system, threading APIs, Managed Debugging Assistants, and memory dump support. His current responsibilities include error reporting, the Profiling API, and special projects.


Data Points
RSS Feeds on a Smartphone
John Papa

Code download available at: DataPoints2006_12.exe (172 KB)
Browse the Code Online
Contents
Preparing for Mobile Development
The News Reader Application
Asynchronous Feed Requests
Loading a DataSet
Loading the List of Feeds
Sorting with Anonymous Delegates
Displaying with the WebBrowser Control
Deploying to a Smartphone
Conclusion

Mobile applications are a lot like other applications. They rely on data, they often communicate data over a connection, and the data needs to be managed. As mobile applications become more sophisticated, so do the tools we use for developing them. In fact, when developing data-centric mobile apps using the Microsoft® .NET Compact Framework 2.0, you rely on many of the same skills that you use to develop Windows® Forms apps. With the .NET Compact Framework, developers can use ADO.NET and other familiar libraries to manage data in a mobile device.
Developing for a Smartphone introduces yet another level of design consideration, such as accounting for the small screen size. And its built-in Internet connectivity influences application design as the ability to send and receive data lends itself to hosting applications that communicate with Web servers.
This month I will demonstrate how to build a Windows Mobile® 5.0 application that reads RSS feeds and loads them into an ADO.NET DataSet. I will start out by showing how to request an RSS feed from a URL asynchronously and then demonstrate how to read the RSS feed and load it into a DataSet. Next, I will show you how to navigate the DataSet's hierarchy to find the appropriate data containing each feed's items. Then I will discuss how to use anonymous delegates to sort the data, how to check for the existence of a field in the RSS data using ADO.NET, and how to use the WebBrowser control to navigate to a URL or display HTML. Finally, I will demonstrate how to deploy the application to a mobile device using a CAB project. The full source code for this sample application is available from the MSDN®Magazine Web site.

Preparing for Mobile Development
Developing mobile applications with the .NET Compact Framework 2.0 and Visual Studio® 2005 is very similar in many aspects to developing ASP.NET or Windows Forms applications. Before diving into the code, however, you'll need to do some setting up. Assuming you already have Visual Studio 2005 installed, you should make sure you also have the following installed, or download them as necessary from the Microsoft Web site:
  • .NET Compact Framework 2.0 Service Pack 1
  • ActiveSync 4.2
  • Windows Mobile 5.0 SDK for Smartphone
  • Emulator Image for Windows Mobile 5.0 Smartphone with 320×240 (Landscape) Screen
Even if you are not going to use an actual Smartphone device during development and instead will rely on the emulator, you still need to install the latest version of ActiveSync®. This is used for such tasks as deploying the application to the emulator and allowing the emulator to connect to the Internet. The Windows Mobile SDK 5.0 for Smartphone adds a few project types to Visual Studio 2005 that are specific to Smartphone development. (There is also an SDK for Pocket PC development.) Finally, you should install the emulator that most closely resembles the target devices. (My application is targeting Smartphone devices with a 320×240 landscape screen.)
A word about emulators. If you have a mobile device, I highly recommend testing your application on it, rather than relying entirely on the emulator. While an emulator is a great tool, the best test is always to run the application on a physical device. When I develop a mobile application, I use the emulator but also deploy the application to a physical device at certain checkpoints to make sure everything is running smoothly in both environments.
Before running the code, you need to start the Device Emulator Manager from the Tools menu of Visual Studio. A list of available emulators will appear. Choose the 320×240 (Landscape) Windows Mobile 5.0 Smartphone Emulator. If you haven't downloaded this option, you can make do with another Windows Mobile 5.0 Smartphone emulator. The code in this column is written for a 320×240 landscape screen, but it could easily be modified to fit other resolutions.
Once you choose the emulator, right-click on it and select Connect. The emulator will then begin to boot the Smartphone. Once the Smartphone is running in the emulator, right-click the item in the Device Manager again and select Cradle. Cradling the emulator's Smartphone will allow it to talk to ActiveSync, which in turn allows it to connect to your network so it can access the Internet and grab the RSS feeds. Finally, in Visual Studio, select the 320×240 emulator from the dropdown list in the Device toolbar. This tells Visual Studio which emulator image to use when you debug and deploy.

The News Reader Application
Before diving into code, let's look at what the application does from a high level. When the application launches, it requests the RSS feeds asynchronously from their respective Web sites and loads a ListView control that provides the title of each feed and the number of items that each feed returns (see Figure 1).
Figure 1 List of RSS Feeds
Figure 1 List of RSS Feeds
Figure 2 Feed Item List
Figure 2 Feed Item List
Figure 3 Item Description
Figure 3 Item Description
When you select a feed and either click the enter soft key (the middle key on the Smartphone) or choose Menu | Select, the selected feed's list of items is displayed in another form's list view, sorted with the most recent first. If the RSS feed contains the item's publication date (which is an optional element in the RSS schema), it will be displayed in the second column of the list view (see Figure 2). If there is no publication date, then the column is omitted and only the title of the item is displayed.
Once you select an item, its description is displayed in another screen. Any content within the item's description element is displayed in a WebBrowser control (see Figure 3).
The item description can contain hyperlinks or images since it reads the description element from the RSS feed. If it contains a URL, the user can click the URL and navigate to that address. If the user clicks the Select menu item with the soft key while viewing the item's description, the application switches to another screen and navigates to the URL of the selected item, loading the content into a WebBrowser control so the entire post can be viewed. The user can always navigate backwards to a previous screen by clicking the Back soft key. And from the Feed Item List screen, the user can select Done from the menu using the Left soft key to exit the application.
This application demonstrates the basic abilities you need to include in an application of this type. However, there are several features and enhancements you can include to make the app more sophisticated, such as progress bars, icons, and the ability to manage the feed list. Feel free to download the code and add your own enhancements.

Asynchronous Feed Requests
Now that I have shown how the application functions, let's take a look under the hood at the code. When the application loads, it first requests three RSS feeds (for brevity, I have hardcoded these URLs in a List<string>), looping through the registered URLs and requesting the feed for each one by calling the BeginScanFeeds method:
   List<string> urlList = new List<string>();
   urlList.Add(
     "http://msdn.microsoft.com/msdnmag/rss/        rss.aspx?Sub=Data+Points");
   urlList.Add("http://blogs.msdn.com/MainFeed.aspx");
   urlList.Add("http://blogs.technet.com/MainFeed.aspx");

   ClearList();
   foreach (string url in urlList) BeginScanFeeds(url);
The BeginScanFeeds method creates an instance of the HttpWebRequest class for the specified URL. The request is then made asynchronously for the given URL using the HttpWebRequest's BeginGetResponse method. This method accepts an asynchronous callback delegate (which in this case is the EndScanFeeds method) and the request:
   public void BeginScanFeeds(string url)
   {
       HttpWebRequest request =
          (HttpWebRequest)WebRequest.Create(url);
       IAsyncResult asyncResult = request.BeginGetResponse(
           new AsyncCallback(EndScanFeeds), request);
   }
Each request is sent and when the response returns, the EndScanFeeds method is invoked. The EndScanFeeds method has a single argument of type IAsyncResult, which can be used to retrieve the feed. Figure 4 shows a snippet of the code from this method that puts the response from the resulting HttpWebResponse object into a string named feedData using a StreamReader object. The feedData string contains the RSS feed's XML and is passed to the LoadFeedList method, which loads the feed data into a ListView control. Because the feeds are read asynchronously and the LoadFeedList method accesses the ListView control, the method needs to be called using the Control.Invoke method.
Figure 4 Reading the Feeds
public void EndScanFeeds(IAsyncResult result)
{
    string feedData;
    HttpWebRequest request = (HttpWebRequest)result.AsyncState;
    using(HttpWebResponse response =
        (HttpWebResponse)request.EndGetResponse(result))
    using(StreamReader streamReader =
        new StreamReader(response.GetResponseStream()))
    {
        feedData = streamReader.ReadToEnd();
    }

    Invoke(new LoadFeedListDelegate(LoadFeedList), feedData);
}

Loading a DataSet
Once the feed has been retrieved, its contents must be parsed. While there are several ways to read an XML string, I have chosen to load the data into a DataSet, using the following code:
   DataSet ds = new DataSet();
   XmlTextReader xmlRdr =
      new XmlTextReader(new StringReader(rssXmlData));
   ds.ReadXml(xmlRdr);
I've opted for this approach because it allows me to utilize such capabilities as checking for the existence of columns and navigating the RSS feed's hierarchy.
The feed's contents are loaded into a StringReader object, which is then used to create an XmlTextReader. The XmlTextReader can then be read by the DataSet's ReadXml method, which loads the contents of the feed into the DataSet. The RSS schema includes three main elements in the hierarchy: rss, channel, and item. (Some schemas contain more elements, such as an images element.)
The rss element is the root of the RSS feed, which contains one or more channel elements. The channel elements represent the feed and can contain a series of child elements that describe the feed (such as title, link, description, and item). The item element is a required child of the channel element. (This, by the way, is not the same as the main item element.) Item elements represent each item in the feed. For example, in the MSDN Blog, the channel is the MSDN Blog while the items represent the posts.
When an RSS feed is loaded into a DataSet, the main elements-rss, channel, and item-are loaded as tables. Each level of the hierarchy becomes a DataTable and can be read and navigated using the DataTable's features. The simple child elements that describe each main element are loaded as columns in the respective DataTable. For example, the channel DataTable contains a DataColumn for the item, description, and link elements (all three of these elements are required).
Finally, a DataRelation is created between the channel DataTable and the item DataTable representing how a channel contains zero or more items. I exploit this relationship so the application can locate a specific channel's items when a user selects a specific feed.

Loading the List of Feeds
Once the feed is loaded into a DataSet, I locate the channel table and grab the channel's title and the number of items contained in each channel. I store this information in an array and load it into the list view as the feeds are received (see Figure 5).
Figure 5 Loading the List of Feeds
foreach (DataRow channelRow in ds.Tables["channel"].Rows)
{
    string title = channelRow["title"].ToString();
    int itemCount = int.Parse(channelRow.GetChildRows(
        "channel_item").Length.ToString());

    string[] displayInfo = new string[2];
    displayInfo[0] = title;
    displayInfo[1] = itemCount.ToString();

    ListViewItem item = new ListViewItem(displayInfo);
    item.Tag = channelRow;
    lvFeed.Items.Add(item);
}
The title of the channel is accessed using the standard ADO.NET syntax to read the value from a DataRow's column. To get the total number of items in each channel I use the DataRow's GetChildRows method to return an array of DataRow objects for the current channel row's items. The GetChildRows method accepts the name of the DataRelation, which by default is the concatenation of the channel DataTable's name and the item DataTable's name, with an underscore between them. Notice that I store a reference to the channel's DataRow in the tag of the ListViewItem. This makes it easy to grab the feed's information when a user selects a particular feed.

Sorting with Anonymous Delegates
When a user selects a feed, the DataRow is passed to the FeedItemList form, which then loads the list of items into a ListView control. I load the title of the item as well as the publication date, if it is provided. Since the pubDate element is not a required element of the RSS 2.0 schema, I first check to see if the column exists in the item DataTable. Since I have an array of DataRow objects that represent the feed's items, I can use that to find the DataRows' DataTable object and then use the DataTable.Columns.Contains method to check whether the pubDate column exists. It may sound like a lot to do, but the code is quite simple, as shown here:
bool containsPubDate = (itemRows[0].Table.Columns.Contains("pubDate"));
Once I determine whether the pubDate exists, I can then determine how to sort and display the items in the list view. If the pubDate exists, I sort the items by date in descending order. If there is no pubDate, I sort the items by title and hide the publication date column in the list view.
The following code executes if the pubDate exists:
colNewsItemName.Width = 170;
colPubDate.Width = 140;

Array.Sort(itemRows, delegate(DataRow row1, DataRow row2)
{
    DateTime pubDate1 = ParseDateTime(row1["pubDate"].ToString());
    DateTime pubDate2 = ParseDateTime(row2["pubDate"].ToString());
    return pubDate2.CompareTo(pubDate1);
});
It sets the width of the columns so the app will display both the item's title and publication date. It then uses the Array.Sort method to sort the array of DataRow objects. Before I can sort the dates, I have to convert them to a valid DateTime. Once this conversion is done I use a custom comparer delegate to sort the DataRow array.
The custom comparer delegate is a nice, short way to code the sorting routine. I could have created an explicit routine to handle the sorting, but since I wanted to minimize the lines of code, the custom comparer delegate was a good option. The comparison of the dates is executed for each element comparison in the array. In this example I create an anonymous delegate that accepts two DataRow objects and compares them by evaluating the values in their pubDate columns.
If the pubDate column does not exist, I hide the second column in the list view and expand the first column to fill the screen width. Then I sort the DataRow array by the item title in ascending order. I use the same technique in this case, except this time I compare string values:
colNewsItemName.Width = 310;
colPubDate.Width = 0;

Array.Sort(itemRows, delegate(DataRow row1, DataRow row2)
{
    return row1["title"].ToString().CompareTo(row2["title"].ToString());
});
The anonymous method provides a great way to help reduce code when you do not need to reuse the comparer functionality.
Once the DataRow array is sorted, I loop through the items and grab the title and the pubDate (if it exists), as shown in Figure 6. If the pubDate exists, I format it accordingly and load the title and pubDate into a string array, which I then use to load the list view. At this point, the list looks like the one shown in Figure 2.
Figure 6 Loading the Item List
foreach (DataRow itemRow in itemRows)
{
    string title = itemRow["title"].ToString();
    string[] displayInfo = new string[2];
    displayInfo[0] = title;
    if (containsPubDate)
    {
        string pubDateString =
            itemRow.Table.Columns.Contains("pubDate") ?
                itemRow["pubDate"].ToString() : string.Empty;
        DateTime pubDate = ParseDateTime(pubDateString);
        displayInfo[1] = pubDate.ToString("MM/dd/yyyy HH:mm");
    }

    ListViewItem item = new ListViewItem(displayInfo);
    item.Tag = itemRow;
    lvFeed.Items.Add(item);
}

Displaying with the WebBrowser Control
When the user selects an item, the selected ListViewItem is examined and the DataRow of the selected item is retrieved from its Tag property. The DataRow that contains the selected item is passed to the constructor for the ItemSummary form, which displays the description of the selected item. Note that I did not use a label control to display the description because the descriptions of an item in an RSS feed often contains embedded HTML. Thus I used the WebBrowser control to display the description.
Figure 3 shows the summary of an item's description in the WebBrowser control. You can see that the summary description includes hyperlinks and other HTML elements. To render the HTML, I set the WebBrowser control's DocumentText property to the item's description and the Form's Text property to the title of the item:
this.Text = itemRow["title"].ToString();
webSummary.DocumentText = itemRow["description"].ToString();
In this application, the WebBrowser control serves two purposes. It is used by the ItemSummary form to display the description of the item, which may or may not contain HTML. It is also used by the Item form, which navigates to the link specified in the item's DataRow. This is the more conventional way to use the WebBrowser control:
this.Text = itemRow["title"].ToString();
webItem.Navigate(new Uri(itemRow["link"].ToString()));
Here, I have set the title of the Item form to the item's title, and I navigate to the item's URL in the WebBrowser control using the Navigate method. The link is then located and the page is loaded into the WebBrowser control.

Deploying to a Smartphone
Once the application is built, it is time to deploy it to an actual Smartphone device. At this point, I usually shut down the emulator window and the Device Emulator Manager so I can fully test the deployment to a Smartphone.
Some notes about deploying the application. First, you should create a Smartphone Device CAB project and add it to the current solution. Also, to run an application built using the .NET Compact Framework 2.0, the Framework must be installed on the mobile device. Once these two steps have been taken care of, you can copy the CAB file to the mobile device and install the application directly.
I created the CAB project, named MSDN200611, and added content files and primary output. The content files pick up the icon file I used in the Smartphone device application. I created a shortcut to the primary output, added the Start Menu folder to the CAB file project, and dragged the shortcut to the Start Menu folder-this puts the application in the Smartphone's list of applications in its Start Menu. I also renamed the shortcut to a friendlier and shorter name.

Conclusion
When developing datacentric mobile applications with the .NET Compact Framework 2.0, you'll use many of the same skills that you use to develop Windows Forms applications. This month, I have demonstrated how ADO.NET can be used to load RSS feeds, navigate the feeds and their items, and check for the existence of RSS information. Anonymous methods and asynchronous invocation assist in the development, but the core of the application is still the data itself-whether it be the RSS feed, the DataSet, an array of DataRow objects, or the HTML that is rendered in the WebBrowser control for the item description.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences such as VSLive or blogging at codebetter.com/blogs/john.papa.


Cutting Edge
The Client Side of ASP.NET Pages
Dino Esposito

Code download available at: CuttingEdge2006_12.exe (153 KB)
Browse the Code Online
Contents
Analysis of the ASPX Code
Analysis of the HTML Client Code
The View State Field
The PostBack Mechanism
Analysis of Class Code

There's a trend in the software industry towards moving much of the burden of code writing to the infrastructure of the underlying platform. A variety of development platforms ask developers to provide a high-level description of the information they need in a relatively loose syntax, instead of hard-coding every single byte of it according to a strict set of syntax rules. It is now common for developers to use an XML dialect to describe the desired result and have a compiler or runtime engine parse and process the contents into traditional and executable code.
For example, Windows® Presentation Foundation, one of the pillars of the .NET Framework 3.0, uses XAML as the XML-based presentation language to describe the user interface of the form. The Microsoft AJAX Library (part of the system formerly code-named ASP.NET "Atlas") applies the same principle to rich Web pages with its XML-Script metalanguage (although, technically, XML-Script is not part of the core release, but rather it's being shared as an unofficial sample technology). A declarative layout language, XML-Script wires up HTML elements and script together and forms virtual client-side controls. In the end, XML-Script injects logic and functionality in client pages.
There are a few advantages to using a declarative language to author Web pages and forms. In this way, server-side components can more easily generate pages and forms than if they had to emit actual Visual Basic®, C#, or JavaScript code. Furthermore, declarative markup is inherently easier to devise and design for authoring tools such as Visual Studio®. From an architectural standpoint, by using a declarative approach you indicate what page elements will do, but not how they will do it. In this way, you create an additional abstraction layer.
The first concrete programming environment to take advantage of such a model was ASP.NET, starting with version 1.0. As most Web developers should know by now, an ASP.NET page is typically written in one or two files: an .aspx markup file and, optionally, a code-behind file. The code-behind contains a class file written in any supported programming language, though typically Visual Basic or C#. The .aspx markup file contains HTML tags, ASP.NET control tags, and literals that form the structure of the page (it can also contain code). This text is parsed at run time and transformed into a page class. Such a page class, combined with the code-behind class and some system-generated code, comprise the executable code that processes any posted data, generates the response, and sends it back to the client.
While the overall model is known to the vast majority of ASP.NET developers, a number of black holes exist that only a small group of developers understand with much depth. MSDN®, books, and online articles explain single aspects of the page machinery, but an overall and unified coverage of the page internals is still lacking. If you take a look at the HTML source code of an ASP.NET page you see a number of hidden fields and automatically injected blocks of JavaScript code that you may hardly make sense of. However, these fields and blocks contribute to make the Web page work. In this column I'll analyze the client-side source code that ASP.NET pages generate. I'll cover hidden fields such as the well-known view state, but also little known ones such as control state, event validation, event target, and argument and system-provided script code.
Much of the implementation details I cover here are specific to the current version of ASP.NET. These details could change in the future (they have changed in the past), and you shouldn't build any production code that depends on any undocumented details.

Analysis of the ASPX Code
Figure 1 shows a minimal but working ASP.NET page. Despite its extreme simplicity, this is a good sample as it includes typical elements of a real-world ASP.NET page-input fields, clickable postback elements, and read-only elements.
Figure 1 Sample ASP.NET Page Ready for Dissection
<%@ Page Language="C#" AutoEventWireup="true" CodeFile="Test.aspx.cs"
    Inherits="Test" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server"><title>Test page</title></head>
<body>
  <form id="form1" runat="server">
    <div>
      <asp:TextBox ID="TextBox1" runat="server"></asp:TextBox>
      <asp:Button ID="Button1" runat="server"
        OnClick="Button1_Click" Text="Click me" />
    </div>
    <hr />
    <h1><asp:Label ID="Msg" runat="server" Text=""></asp:Label></h1>
  </form>
</body>
</html>
The .aspx page contains three server controls: a textbox to capture data, a Submit button to start a post operation, and a label to display read-only data. On top of the .aspx file, the Page directive defines some global attributes for the individual page. Let's take a look at the most commonly used attributes of the Page directive, such as those that you saw in Figure 1.
<%@ Page Language="C#"
         AutoEventWireup="true"
         CodeFile="Test.aspx.cs"
         Inherits="Test"
%>
Most of the Page directive attributes have limited effect on the page markup, the HTML code that the browser receives with the HTTP response. Rather, most Page attributes affect the code of the dynamically generated page that the system builds on top of the .aspx markup and code-behind class. The Language attribute designates the language used to author the code-behind in Visual Studio. The system will use the same language to generate the dynamic page class to serve the browser request for the .aspx resource. The CodeFile attribute indicates the source file where the code-behind class is stored. The Inherits attribute indicates the name of the code-behind class in the code file that should be used as the parent of the dynamically generated page class. Finally, the AutoEventWireup attribute indicates whether a default naming convention should be used to map handling code to Page events. When AutoEventWireup is set to true, you can add a Page_Load method to the code file to handle the page Load event, and it will automatically be registered with the Page's Load event. The implicit naming convention dictates that the event handler will take the form of Page_XXX, where XXX can be the name of any public events defined on the Page class. If AutoEventWireup is set to false, you must explicitly bind the Page class event with its handler. You can do that in a made-to-measure class constructor:
public partial class Test : System.Web.UI.Page
{
  public Test()
  {
    this.Load += new EventHandler(Page_Load);
  }
  ...
}
When the Web server receives an HTTP request for a given .aspx resource, it forwards the request to the ASP.NET worker process. The process hosts the CLR, inside of which a runtime environment is created to process ASP.NET requests. The ultimate goal of the ASP.NET HTTP runtime environment is serving the request-that is, obtaining the markup (HTML, WML, XHTML, and whatever else the app is supposed to return) which will be embedded in the HTTP response. In charge of returning the markup for the request is a special system component known as the HTTP handler.
The HTTP handler is an instance of a class that implements the IHttpHandler interface. The ASP.NET framework comes with a few predefined HTTP handlers to serve particular situations or to act as a base class for other and more specialized requests. The System.Web.UI.Page class is one of the most complex and sophisticated built-in HTTP handlers in ASP.NET.
Each ASP.NET request is mapped to an HTTP handler. Suppose that a client browser places a request for a page named test.aspx. The request is passed to ASP.NET and processed by the HTTP runtime. The runtime determines the HTTP handler class to serve the request through a page handler factory. If is the correct handler is not yet available in the AppDomain, it is created dynamically and stored in the ASP.NET temporary folder on the Web server machine. For a page named test.aspx, the HTTP handler is created as a class named ASP.text_aspx.
The dynamic creation of the HTTP handler class for a given request is a process that takes place only once per page, the first time that page is requested in the application lifetime (although when batch compilation is used, the handler can be generated on the first request for any page in the application). The dynamically created assembly is invalidated and replaced if the application is restarted or if the page source is modified on the Web server. Figure 2 shows the hierarchy of page classes from the base Page class down to the dynamically generated class to serve the user request.
Figure 2 Hierarchy of Page Classes
Figure 2 Hierarchy of Page Classes
The ASP.NET runtime creates the Visual Basic or C# source code of the dynamic page class by parsing the source code of the corresponding .aspx file. Each tag with runat="server" is mapped to a server control instance. Any other text is mapped to a literal control and emitted verbatim. The Register directive, if any, helps to resolve tags pointing to non-standard controls. The markup returned to the client browser is composed by accumulating the markup that each server control in the page emits. Note that each page generally emits markup, usually HTML markup. However, this is not a requirement, and an ASP.NET page can output any data it wants.

Analysis of the HTML Client Code
Figure 3 shows the HTML output for the sample page in Figure 1. In the HTML there's no clue that a Page directive existed in the server-side .aspx page. Instead, the !DOCTYPE directive is copied verbatim. The first runat="server" block in Figure 1 is the <form> tag. This means that any text in between Page and <form> is emitted verbatim. In the source code of the dynamically created page class on the server, this text is converted into a single instance of the LiteralControl class. The <form> tag is emitted like this:
<form name="form1" method="post" action="Test.aspx" id="form1">
Figure 3 HTML Markup for a Test Page
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head><title>Test page</title></head>
<body>
  <form name="form1" method="post" action="Test.aspx" id="form1">
    <div>
      <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE"
        value="/wEPDw ... eeY=" />
    </div>
    <div>
      <input name="TextBox1" type="text" id="TextBox1" />
      <input type="submit" name="Button1"
        value="Click me" id="Button1" />
    </div>
    <hr />
    <h1><span id="Msg"></span></h1>
    <div>
      <input type="hidden" name="__EVENTVALIDATION" id="__
      EVENTVALIDATION"
        value="/wE ... u7" />
    </div>
  </form>
</body>
</html>
The <form runat="server" ...> tag is rendered through an instance of the HtmlForm class. The control class has no property to let you set the action attribute on the output markup. The action attribute is hardcoded to the URL of the current page. This behavior is at the foundation of the ASP.NET platform. Note that the ID attribute is partnered with an identical name attribute.
The <asp:textbox> tag is rendered in HTML through an <input type="text"> element. In this case, a name attribute is added to match the original ID attribute. Note that if you omit the ID attribute you may receive a warning from Visual Studio 2005, but ASP.NET will still compile the page successfully. If the ID attribute is missing, a random string is generated and bound to the name attribute. The <asp:Button> tag is rendered through an <input type="submit"> button. An <asp:Label> tag will render the HTML <span> tag to the client browser.
In most cases (though not in all), each tag decorated with the runat="server" attribute generates a corresponding block of HTML markup. The ID string guarantees a persistent match between the two blocks-one on the client side and one on the server side. As you can see in Figure 3, a couple of hidden fields complete the HTML markup: __VIEWSTATE and __EVENTVALIDATION.

The View State Field
The contents of the __VIEWSTATE field represent the state of the page when it was last processed on the server. Although sent to the client, the view state doesn't contain any information that should be consumed by the client. The information stored in view state is pertinent only to the server page and some of its child controls and is exclusively read, consumed, and modified by the server.
Implemented in this way, the view state doesn't consume any critical server resources and is fast to retrieve and use. On the other hand, just because the view state is packed with the page, it inevitably increases the size of the HTTP request and response by a few kilobytes. Note that a realistic page padded with a grid of data can easily reach a view state size of 20KB. This extra stuff is uploaded and downloaded each and every time. The view state is one of the most important features of ASP.NET because it enables stateful programming over a stateless protocol such as HTTP. Used without strict criteria, though, the view state can easily become a burden for pages.
By overriding a couple of methods on the code file class, you can leave the contents of the view state field on the server, stored in a database, in the Cache or in the Session object. However, note that leaving the view state information on the server is not the obvious workaround it first appears. It's not by chance, in fact, that the ASP.NET team opted for a page-based view state. A server-based view state is fine as long as user navigates from one page to the next following the links in the application. Remember that ASP.NET applications work by posting repeatedly over the same page. But what if the user clicks the Back button? To be safe, you should maintain view state on a per-request basis rather than on a per-page basis. And the chain of tracked requests should be as long as the requests the user can reach through the Back and Forward buttons. View state stored on the client may not be perfect, but neither is view state stored on the server. The one that's preferable for your application depends on the expectations you have for it.
In ASP.NET 2.0, the __VIEWSTATE hidden field contains two types of information-view state and control state. Developers can disable view state altogether and operate their applications in a pure stateless manner. This is not an issue as long as you use built-in controls and controls that you wrote yourself, or at least controls for which you have access to the source code. What if you use a custom control that assumes an enabled view state? Some controls-typically, rich third-party and custom controls-need to persist private information across postbacks. This information is not public and not designed to be exposed to the application level-for example, the collapsed/expanded status of a dropdown panel. This information can only be persisted to the view state. If the view state is disabled, the control may inadvertently fail.
To alleviate this issue, ASP.NET 2.0 introduces the notion of the control state. Each server control can pack any critical properties to a collection and store it to the page's control state. The control state is saved to the __VIEWSTATE field but, unlike the traditional view state, can't be disabled and is always available. Developers manage the control state through a pair of new overridable methods on the Page class: LoadControlState and SaveControlState. Speaking of the view state in ASP.NET 2.0, though, it is also worth noticing that a new and more effective serialization algorithm is employed to streamline the state of individual controls to a hidden field. As a result, the overall size of the __VIEWSTATE hidden field in most cases is as small as half the size of the corresponding field in ASP.NET 1.x.
As mentioned, the view state is stored in a hidden field to associate it unambiguously with a particular page request. When any of the HTML elements in a given page instance post back, the dynamically generated page class starts working on the server and uses the data stored in the view state to recreate the last known good state for the controls in the page. What if the view state is tampered with on the client? Is that ever possible? By default, the view state is encoded using the Base64 schema and hashed, and the resulting hash value is also stored with the view state. The hash value is calculated from the contents of the view state plus a server key. Whenever the page posts back, the code in the page class separates the contents and hash value of the view state. Next, it recalculates the hash value based on the retrieved view state contents and server key. If the two hash values don't match, a security exception is thrown (see Figure 4).
Figure 4 Page View Can't Be Altered on the Client
Figure 4 Page View Can't Be Altered on the Client
What if a malicious user attempts to post a fake request with a modified view state? The malicious user would need to know the server key in order to generate a hash value on the modified view state contents that can be matched on the server. The server key, though, is made of server-only information and is not included in the view state field. The tweakviewstate.aspx page in the companion code contains script code to modify the view state and practice with the exception shown in Figure 4.
Although the view state can hardly be used to plan an attack, it doesn't guarantee data confidentiality unless encryption is used. The contents of the view state, in fact, can be decoded and examined on the client, but not successfully modified to serve an altered page state to the server environment.
The __EVENTVALIDATION hidden field is a security measure new to ASP.NET 2.0. The feature prevents unauthorized requests sent by potentially malicious users from the client. To ensure that each and every postback and callback event originates from the expected user interface elements, the page adds an extra layer of validation on events. The page basically matches the contents of the request with the information in the __EVENTVALIDATION field to verify that no extra input field has been added on the client and that value is selected on a list that was already known on the server. The page generates the event validation field during rendering-that is at the last possible moment when the information is available. Like the view state, the event validation field contains a hash value to prevent client-side tampering.
Controls use the RegisterEventForValidation method on the ClientScriptManager object to store their own information for safe postbacks. At a very minimum, each control registers its own unique ID. List controls also store all the values in the list. Server controls that support event validation typically call the ValidateEvent method in their implementation of the IPostBackDataHandler interface. If the validation fails, a security exception would be thrown.
You can enable and disable event validation on a per-page basis; each control class enables event validation through the SupportsEventValidation attribute. Currently, there's no way to enable or disable event validation on a particular control instance.
Event validation is a defense barrier aimed at limiting input to a known set of values. It simply raises the security bar higher and doesn't stop script injection attacks by itself.
Event validation may pose issues if used in the context of AJAX-enabled applications. In such applications, some client work can create new input elements on the fly, thus making the next postback fail because of unknown elements. The best workaround is to render any user interface on the server whenever possible, and hide it on the client using the cascading style sheets display attribute. In this way, any user interface you're going to use is registered with the event validation field. If you write custom controls, you should decorate it with the SupportsEventValidation attribute to enable this feature.

The PostBack Mechanism
The ASP.NET page in Figure 1 posts back as the user clicks the button. This is because the <asp:Button> tag renders as an HTML submit <input> element. When a submit input field is clicked, the browser fires the onsubmit HTML client event and then prepares the new request to the server based on the contents of the submitted form. The HTTP request being sent includes an additional piece of information that evaluates to the ID of the button.
The page class scans the body of the HTTP request to see if any of the posted fields matches the ID of a button control in the ASP.NET page. If the match is found, that button control is called to run any code associated with its Click event. More precisely, the page class checks to see if the matching button control implements the IPostBackEventHandler interface. If so, it invokes the RaisePostbackEvent method on the interface. For a button control, the method raises the server-side Click event.
So far, so good. But what if the page contains a LinkButton control instead? Figure 5 shows the markup for an ASP.NET page that is identical to the page in Figure 1 except that a LinkButton is used instead of the Submit button. As you can see, the markup includes two more hidden fields, __EVENTTARGET and __EVENTARGUMENT, and a bit of JavaScript code. The href target of the link button is bound to the __doPostback script function meaning that the function will be invoked whenever a client click on the element is detected. The __doPostback function is emitted in the page by the rendering code of the LinkButton control. It populates the __EVENTTARGET and __EVENTARGUMENT fields with proper information and then triggers the postback via script. In this case, the body of the HTTP postback request simply contains the input fields in the page and no posted data references the Submit button.
Figure 5 HTML Markup Using a LinkButton Control
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head><title>Test page</title></head>
<body>
  <form name="form1" method="post" action="LinkBtn.aspx" id="form1">
    <div>
      <input type="hidden" name="__EVENTTARGET"
        id="__EVENTTARGET" value="" />
      <input type="hidden" name="__EVENTARGUMENT"
        id="__EVENTARGUMENT" value="" />
      <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE"
        value="/wEP ... Eag==" />
    </div>

    <script type="text/javascript">
    <!--
      var theForm = document.forms['form1'];
      if (!theForm) {
        theForm = document.form1;
      }
      function __doPostBack(eventTarget, eventArgument) {
        if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
          theForm.__EVENTTARGET.value = eventTarget;
          theForm.__EVENTARGUMENT.value = eventArgument;
          theForm.submit();
        }
      }
    // -->
    </script>

    <div>
      <input name="TextBox1" type="text" value="" id="TextBox1" />
      <a id="LinkButton1"
        href="http://javascript:__doPostBack('LinkButton1','')">Click me</a>
      <hr />
      <h1></h1>
    </div>

    <div>
      <input type="hidden" name="__EVENTVALIDATION" id=
         "__EVENTVALIDATION"
        value="/wEW ... JbJ" />
    </div>
  </form>
</body>
</html>
How does ASP.NET recognize the control responsible for handling the postback? When no controls referenced in the request body implement the IPostBackEventHandler interface, the page class looks for the __EVENTTARGET hidden field, if any. The contents of the field is assumed to be the ID of the control that caused the postback. If this control implements the IPostBackEventHandler interface, the RaisePostbackEvent method is invoked. For a LinkButton control, this results in the invocation of the Click server event.

Analysis of Class Code
The .aspx markup defines the layout of an ASP.NET page and determines size, style and position of constituent controls. It contains no logic, however, except perhaps for some client script code and any Visual Basic or C# inline code you may have. Initialization code, event handlers, and any helper routines typically go in a separate companion file, known as the code-behind file:
public partial class Test : System.Web.UI.Page
{
  protected void Page_Load(object sender, EventArgs e)
  {
    ...
  }
  protected void Button1_Click(object sender, EventArgs e)
  {
    ...
  }
}
The class in the code file inherits, directly or indirectly, from System.Web.UI.Page. The code file and markup represent required but distinct pieces of information. To fully represent the ASP.NET page, they must be combined to form a page class that incorporates the logic of the code file and the layout data of the markup file. The code file class is already a page class, but it lacks two key pieces of information: the list of child server controls to populate the user interface and the declaration of class members that identify the various server controls.
In ASP.NET 1.x, each time the page author drops a control onto the Web Form, Visual Studio .NET 2003 automatically adds a new line to the code file to create a class member that handles the just dropped server control. This does a pretty good job of keeping everything in sync, but often developers run into compile errors due to the lack of a class member or existence of useless class members.
In ASP.NET 2.0 the issue is fixed in an elegant way. Enter partial classes, a source-level, assembly-limited, non-object-oriented way to extend the behavior of a class. In the .NET Framework 2.0, a class definition can span over two or more files. Each file contains a fragment of the final class definition and the compiler takes care of merging the various partial definitions to form a single, unified class. All fragments must have the same signature and the final class definition must be syntactically correct.
Next, a second partial class is generated dynamically to list all control members. The two partial classes are merged at compile-time. When the .aspx markup file is parsed to create the temporary ASP.test_aspx class, this class inherits from the combined code file in its final version. If the ASP.NET page is not bound to a code file but contains its code inline, then the dynamic page class inherits from System.Web.UI.Page and includes any inline code in its body.
There's a lot more to learn about the dynamic page compilation machinery, but this provides fodder for a future column.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Test Run
String Permutations
Dr. James McCaffrey

Code download available at: TestRun2006_12.exe (161 KB)
Browse the Code Online
Contents
A String Permutation Class
Generating All Permutations of Order n
Determining a Specific Permutation Element
The Number of Permutation Elements of Order n
Wrapping It Up

The ability to programmatically create and use string permutations is an essential software testing skill. A string permutation is a rearrangement of a set of strings. For example, if you have an initial set of three strings—apple, banana, cherry—then there are a total of six permutations:
{ "apple", "banana", "cherry" }
{ "apple", "cherry", "banana" }
{ "banana", "apple", "cherry" }
{ "banana", "cherry", "apple" }
{ "cherry", "apple", "banana" }
{ "cherry", "banana", "apple" }
A typical use of permutations in software testing is generating test case data for unit, API, and module tests. Suppose you have a method that accepts three strings, and one of your test case inputs is "apple", "banana", "cherry". In most situations you want to create five additional test cases using the other input permutations. There are many other uses of permutations in software testing. In fact, permutations are so important and prevalent in software engineering that questions about permutations are among the most common type of interview questions for testing jobs at Microsoft.
A good way to see where I'm headed in this month's column is by way of a screenshot. Figure 1 shows a run of a demo program. (The complete source code used to generate the demo is available in the download for this column.) As the output in Figure 1 points out, the three fundamental skills related to string permutations are the ability to calculate exactly how many permutations there are for a given set of strings, the ability to generate all permutations, and the ability to generate a specified permutation.
Figure 1 String Combinations Demonstration
Figure 1 String Combinations Demonstration
Because permutation terminology varies somewhat between problem domains, let's establish a few key vocabulary terms that I'll be using throughout this column. The term permutation element refers to an ordered set of strings from some initial set. The term atom means any one of the individual strings in a permutation element. The order of the permutation is the number of atoms in any element. I will be using lexicographical permutations. This means the initial permutation element, also called the identity element, has atoms listed in dictionary order. Each permutation element is also ordered relative to the other elements. This means that there is a distinct identity permutation element, a distinct last element, and each element has a successor and predecessor element (except for the first element which has no predecessor, and the last element which has no successor). Finally, the term string permutation just means a permutation element where the atoms are strings. I use this term because mathematical permutations of order n have integer atoms from 0 to n-1, such as { 0, 1, 2 }. The total number of permutation elements of order n is n! (read as "n factorial"). For example, with four atoms, the total number of permutation elements is:
   4! = 4 * 3 * 2 * 1 = 24
The reason for this should be easy to see. For the first atom in the element, any of the n atoms can be chosen. For the next, any of the n-1 remaining atoms can be chosen. And so on.
Permutations are closely related to, and sometimes confused with, combinations. String combinations are subsets of an initial set of strings, where order does not matter. For example, consider my initial set of strings that comprise apple, banana, and cherry. There are three combination elements of subset size k = 1:
    { "apple" }, { "banana" }, {"cherry" }
There are three combination elements of subset size k = 2:
{ "apple", "banana" }
{ "apple", "cherry" }
{ "banana", "cherry" }
And there is one combination element with subset size k = 3:
{ "apple", "banana", "cherry" }
The July 2004 issue of MSDN®Magazine includes a column on combinations, "Using Combinations to Improve Your Software Test Case Generation".
In the remainder of this column I will present the overall design of a string permutation class, including a basic constructor, and I will show you how to generate all possible permutation elements by using a neat algorithm to generate the successor to a given permutation element. Next, I'll show you how to generate a specific permutation element using a cool technique that employs one of the most elegant algorithms I've ever seen. I'll then present techniques to determine how many permutation elements there are for an initial set of strings. I'll conclude with a brief discussion of how you can adapt and extend the code and algorithms presented here to meet your own needs. I'm confident that after you've read this column you will have added several useful new tools to your personal skill set.

A String Permutation Class
String permutations are perfectly suited for implementation using an object-oriented approach because permutations contain closely related data (the string atoms) and code. The overall structure of my class library is presented in Figure 2.
Figure 2 StringPerm Class Library Structure
using System;
using System.Text;

namespace StringPermLib
{
  public class StringPerm
  {
    private string[] element;
    private int order;

    public StringPerm(string[] atoms) { ... }
    public StringPerm(string[] atoms, int k) { ... }

    public override string ToString() { ... }

    public static bool IsValid(string[] e) { ... }
    public StringPerm Successor() { ... }

    public static ulong FactorialCompute(int n) { ... }
    public static ulong FactorialLookup(int n) { ... }
    public static ulong FactorialRecursive(int n) { ... }
  }
}
I've named my single class StringPerm; you can choose a different name of course. The StringPerm class has only two data fields, both private: a string array named element to hold the individual atom strings for a particular string permutation element, and an integer field to store the order of the permutation class. I do not implement a default constructor; instead I implement a constructor that accepts a string array as a single input parameter and creates the identity permutation element:
public StringPerm(string[] atoms)
{
    if (atoms == null) throw new ArgumentNullException("atoms");
    if (!IsValid(atoms)) throw new ArgumentException("atoms");
    this.element = new string[atoms.Length];
    atoms.CopyTo(this.element, 0);
    this.order = atoms.Length;
}
I simply allocate memory space for the class array field, use the Array.CopyTo method to assign each value, and then use the input argument's Length property to assign a value to the order field—clean and simple, which is usually a good sign when designing a class. The following code snippet shows that class constructor being called:
string[] atoms = new string[] { "ant", "bat", "cow", "dog" };
StringPerm p = new StringPerm(atoms);
Because I am creating a lexicographical string permutation class, my constructor must accept a string array that is sorted. For that task, I've added a static method IsValid to my class, which I call from my constructor. Here is one way to implement IsValid:
public static bool IsValid(string[] e)
{
    if (e == null) return false;
    if (e.Length < 2) return false;
    for (int i = 0; i < e.Length - 1; ++i)
    {
        if (e[i].CompareTo(e[i + 1]) >= 0) return false;
    }
    return true;
}
My method accepts a string array and returns true if the array is valid as an input argument to my StringPerm constructor, false otherwise. I check to make sure that my input array is not null and that it has at least two cells. You may want to modify this part of the code to allow the possibility of a string permutation of order 1. Next I use the String.CompareTo method to make sure that each string atom in the input array is strictly less than its next neighbor or, in other words, I check to make sure that the string atoms are in order, and I do not allow duplicate atoms. Notice that I use reverse logic and return false if the string atom at index position i is greater than or equal to the string atom at index i+1. You could also allow for duplicates, such as in the following array:
{ "ant", "bat",
 "bat", "cow" }
To do so, just change the >= operator in the check to >.
It is convenient to have a small ToString helper method. Here is one way:
public override string ToString()
{
    StringBuilder result = new StringBuilder();
    result.Append("{ ");
    for (int i = 0; i < this.order; ++i)
    {
        result.Append(this.element[i]);
        result.Append(" ");
    }
    result.Append("}");
    return result;
}
All I'm doing here is simply building up a result string by concatenating each string atom separated by a blank space and surrounded by curly brace delimiters.

Generating All Permutations of Order n
One approach to generating all string permutations is to write a Successor method that returns the next lexicographical string permutation element for a given element. This is an interesting problem and has a pretty solution. Briefly, the algorithm to find the successor to a permutation element determines the locations of two atoms, swaps the atoms at those two positions, and then reverses the order of atoms on the right tail. It's easiest to see what I mean with an example. Suppose you are working with a string permutation of digits (this will make my explanation easier to follow) and the current permutation element is:
{ "1" "3" "5" "4" "2" "0" }
The successor element is:
{ "1" "4" "0" "2" "3" "5" }
First you find a left index and a right index, which point to two atoms that will be swapped. To find index left you start at the next to last atom (2 in this example), and move left until you find a condition where the atom on the left is less than the current atom. So, in this example I'd move left until reaching atom 3 because 3 is less than 5. So index left is [1]. To find index right, you start at the right-most atom (in this case 0) and move left until you find an atom that is greater than the atom at index left. Here I'd move left until hitting atom 4, so index right is [3]. Now swap the atoms at indexes left and right ([1] and [3]), yielding an intermediate result of:
{ "1" "4" "5" "3" "2" "0" }
I finish by reversing all atoms to the right of index left, in this case all atoms to the right of [1], which includes 5, 3, 2, and 0, giving us the final successor result:
{ "1" "4" "0" "2" "3" "5" }
The code to implement this algorithm is presented in Figure 3.
Figure 3 Implementing the Successor Algorithm
public StringPerm Successor()
{
  StringPerm result = new StringPerm(this.element);
  int left, right;

  // Step #1 - Find left value
  left = result.order - 2;
  while ((result.element[left].CompareTo(result.element[left + 1])) >= 0
         && (left >= 1))
  {
    --left;
  }
  if ((left == 0) &&
      (this.element[left].CompareTo(this.element[left + 1])) >= 0)
  {
    return null;
  }

  // Step #2 - find right; first value > left
  right = result.order - 1;
  while (result.element[left].CompareTo(result.element[right]) >= 0)
  {
    --right;
  }

  // Step #3 - swap [left] and [right]
  string temp = result.element[left];
  result.element[left] = result.element[right];
  result.element[right] = temp;

  // Step #4 - reverse order the tail
  int i = left + 1;
  int j = result.order - 1;
  while (i < j)
  {
    temp = result.element[i];
    result.element[i++] = result.element[j];
    result.element[j--] = temp;
  }

  return result;
}
This Successor method only works for permutations with no duplicate atoms. A mildly tricky part of the Successor method is the part that determines what to do on the last permutation element. Suppose you are working with this element:
{ "ant", "bat", "cow", "dog" }
The last permutation element is:
{ "dog", "cow", "bat", "ant" }
There are several ways to identify this situation. The approach I take uses the fact that when determining the value of index left in the Successor code, if index left has value 0 and the atom at [0] is greater than the atom at [1], I must be at the last permutation element. In this case I return null. Alternatives to returning null that you might want to consider are returning the identity (first) element (thus wrapping around to the start of the permutation sequence), or throwing an exception.
With this Successor method, you can easily generate a single successor permutation to any given permutation element, or you can generate all permutation elements (if you do not have any duplicate atoms) like this:
string[] atoms = new string[] { "ant", "bat", "cow", "dog" };
Console.WriteLine("In lexicographical order, all permutations are:\n");
for(StringPerm p = new StringPerm(atoms); p != null; p = p.Successor())
{
  Console.WriteLine(p.ToString());
}
A subtle part of this algorithm concerns the issue of duplicate string atoms. As presented, my code does not allow duplicates. But if you want to allow duplicate atoms in your permutation elements, you can first modify the IsValid helper method described in the previous section. Then, to generate all permutation elements with duplicates, you must use a different technique such as the one I present in the next section.

Determining a Specific Permutation Element
Suppose you want to generate just one particular string permutation element. For example, suppose you are working with strings "ant", "bat", "cow", and "dog" and you want just element [13], which is:
{ "cow" "ant" "dog" "bat" }
At first I thought this would be an easy problem; just start with the identity permutation element and then call the Successor method from the previous section 13 times. However, instead of working with just 4 string atoms, suppose you are working with 20 string atoms. In this case there are 2,432,902,008,176,640,000 permutation elements! So, suppose you want just element [999,999,999,999]. Using the looping technique on my desktop computer, it took well over 24 hours to compute the answer. Furthermore, using the looping technique to generate all permutation elements won't work at all if you allow duplicate string atoms. But there is an elegant algorithm you can use that will compute the answer in well under one second and that allows duplicate atoms.
The basis of this graceful algorithm that directly computes a specific permutation element is a neat mathematical construct called the factoradic. A factoradic is an alternate representation of an integer. The trick is to compute the factoradic of the desired index value, then convert the factoradic into a permutation element.
Before I show you the details, let me explain a factoradic. Consider the number 859. If you set the order to n = 7, then the number 859 can be represented as:
(6! * 1) + (5! * 1) + (4! * 0) +
(3! * 3) + (2! * 0) + (1! * 1) + (0! * 0)

= 720 + 120 + 0 + 18 + 0 + 1

= 859
So, the factoradic of 859 is [1 1 0 3 0 1 0]. It turns out that any integer can be uniquely represented as a factoradic with a specified order. What makes the factoradic useful is that there is a one-to-one mapping between factoradics and permutation elements. For example, if you are working with permutations of order 4, the table in Figure 4 shows the relationship between an integer, its factoradic, and corresponding mathematical permutation.
Figure 4 Relationships
index  factoradic    permutation
=================================
0      [ 0 0 0 0 ]   { 0 1 2 3 }
1      [ 0 0 1 0 ]   { 0 1 3 2 }
2      [ 0 1 0 0 ]   { 0 2 1 3 }
3      [ 0 1 1 0 ]   { 0 2 3 1 }
4      [ 0 2 0 0 ]   { 0 3 1 2 }
5      [ 0 2 1 0 ]   { 0 3 2 1 }
6      [ 1 0 0 0 ]   { 1 0 2 3 }
7      [ 1 0 1 0 ]   { 1 0 3 2 }
8      [ 1 1 0 0 ]   { 1 2 0 3 }
9      [ 1 1 1 0 ]   { 1 2 3 0 }
10     [ 1 2 0 0 ]   { 1 3 0 2 }
11     [ 1 2 1 0 ]   { 1 3 2 0 }
12     [ 2 0 0 0 ]   { 2 0 1 3 }
13     [ 2 0 1 0 ]   { 2 0 3 1 }
14     [ 2 1 0 0 ]   { 2 1 0 3 }
15     [ 2 1 1 0 ]   { 2 1 3 0 }
16     [ 2 2 0 0 ]   { 2 3 0 1 }
17     [ 2 2 1 0 ]   { 2 3 1 0 }
18     [ 3 0 0 0 ]   { 3 0 1 2 }
19     [ 3 0 1 0 ]   { 3 0 2 1 }
20     [ 3 1 0 0 ]   { 3 1 0 2 }
21     [ 3 1 1 0 ]   { 3 1 2 0 }
22     [ 3 2 0 0 ]   { 3 2 0 1 }
23     [ 3 2 1 0 ]   { 3 2 1 0 }
Here's an example of what I'm getting at. Suppose you are dealing with the four string atoms "ant", "bat", "cow", and "dog", and you want to compute permutation [13] directly. First determine the factoradic of 13, which is [2 0 1 0], then map that factoradic to its corresponding mathematical permutation, which is { 2 0 3 1 }, and then map that permutation to the initial string atoms:
{ "cow", "ant", "dog", "bat" }
A detailed description of factoradics is outside the scope of this column. However, I dedicate an entire section of my book, .NET Test Automation Recipes (Apress, 2006), to the topic, and you can also find online information.
I decided to implement the subroutine of determining a specific permutation element as an auxiliary constructor. The code is presented in Figure 5. With the constructor shown in Figure 5, you can make calls like this:
string[] atoms = new string[] { "ant", "bat", "cow", "dog" };
Console.WriteLine("\nJust element [13] computed directly is:");
p = new StringPerm(atoms, 13);
Console.WriteLine("     " + p.ToString());
Figure 5 Determining a Specific Permutation Element
public StringPerm(string[] atoms, int k)
{
  this.element = new string[atoms.Length];
  this.order = atoms.Length;

  // Step #1 - Find factoradic of k
  int[] factoradic = new int[this.order];
  for (int j = 1; j <= this.order; ++j)
  {
    factoradic[this.order - j] = k % j;
    k /= j;
  }

  // Step #2 - Convert factoradic[] to numeric permuatation in perm[]
  int[] temp = new int[this.order];
  int[] perm = new int[this.order];
  for (int i = 0; i < this.order; ++i)
  {
    temp[i] = ++factoradic[i];
  }
  perm[this.order - 1] = 1;  // right-most value is set to 1.
  for (int i = this.order - 2; i >= 0; --i)
  {
    perm[i] = temp[i];
    for (int j = i + 1; j < this.order; ++j)
    {
      if (perm[j] >= perm[i]) ++perm[j];
    }
  }
  for (int i = 0; i < this.order; ++i)  // put in 0-based form
    --perm[i];

  // Step #3 - map numeric permutation to string permutation
  for (int i = 0; i < this.order; ++i)
    this.element[i] = atoms[perm[i]];

}
This constructor also allows you to generate all permutation elements regardless of whether you have duplicate atoms or not. For example, see the following code:
string[] atoms = new string[] { "ant", "bat", "cow", "cow" };
StringPerm p = null;
Console.WriteLine("All permutations are:\n");
for (int k = 0; (ulong)k < StringPerm.FactorialLookup(4); ++k)
{
  p = new StringPerm(atoms, k);
  Console.WriteLine(k + " " + p.ToString());
}
This technique will generate duplicate permutation elements. If for some reason you want just unique elements, there are several approaches you can use. For example, you can load all the elements into a hashtable, checking duplicates before each Add operation.

The Number of Permutation Elements of Order n
Calculating the total number of string permutation elements for a given set of strings is paradoxically easy and hard at the same time. Suppose you have four strings, "ant", "bat", "cow", and "dog", and so the order is n = 4. There are a total of 24 permutation elements, as shown in Figure 1. Any element can have any one of the four string atoms in the first (left-most) position, then any one of the remaining three atoms in the next position, then any one of the remaining two atoms in the next position, and finally the sole remaining atom in the last position. In other words, for n = 4, there are 4 * 3 * 2 * 1 = 24 permutation elements. You probably recognize this as the formula for n factorial (often written as n!). Therefore, the total number of string permutation elements of order n is just n factorial. A very easy Computer Science 101 problem, yes? Well, not necessarily. Now let's examine three different implementations of a factorial method.
One approach, often taken by recent college graduates, is to write a factorial method using recursion. For example:
public static ulong FactorialRecursive(int n)
{
  if (n < 0) throw new ArgumentOutOfRangeException("n");
  if (n == 0 || n == 1) return 1;
  else return (ulong)n * FactorialRecursive(n - 1);
}
This is a weak approach in most situations, and if you're not careful it can easily give you the wrong answer. The value of n! gets very large very quickly. For example, the value of 64! is roughly:
126,886,932,100,000,000,000,000,000,000,000,000,000,
000,000,000,000,000,000,000,000,000,000,000,000,000
In fact, the largest value that can be stored by a variable of type ulong is only 20! which is:
20 * 19 * 18 *. .. * 2 * 1 = 2,432,902,008,176,640,000
What do you suppose would happen if you called
ulong result = FactorialRecursive(21)
in a program? You'd expect to throw an arithmetic overflow of some sort, but that is not what happens by default. The compiler will reach ulong.MaxValue (which is 18,446,744,073,709,551,615) and then wrap around, back to 0, and continue computing without any warnings. Note that the common language runtime (CLR) is capable of overflow and underflow checking on numeric operations, a feature you can enable at compile time. On top of this problem, there's really no compelling reason to use a recursive solution here. A second try at a factorial method is to compute the answer iteratively, something like this:
public static ulong FactorialCompute(int n)
{
  if (n < 0 || n > 20)
      throw new Exception("Input argument must be between 0 and 20");
  ulong answer = 1;
  for (int i = 1; i <= n; ++i) answer = checked(answer * (ulong)i);
  return answer;
}
This is certainly better in most cases. The method checks its input parameter and uses the C# checked keyword, which will cause a runtime exception if arithmetic overflow occurs. But, because the input argument set is so small—0 to 20—you might just as well store the 21 possible results in a lookup table, giving you a solution like:
public static ulong FactorialLookup(int n)
{
  if (n < 0 || n > 20)
    throw new Exception("Input argument must be between 0 and 20");
  ulong[] answers = new ulong[] { 1, 1, 2, 6, 24, 120, 720, 5040, 40320,
    362880, 3628800, 39916800, 479001600, 6227020800, 87178291200,
    1307674368000, 20922789888000, 355687428096000, 6402373705728000,
    121645100408832000, 2432902008176640000 };
  return answers[n];
}
In most software testing scenarios, the lookup approach is the most efficient. Obviously, my discussion of different ways to implement a factorial method just scratches the surface of this interesting subroutine.

Wrapping It Up
You can use string permutations in many ways. For example, suppose you are testing some software setup program and you are considering the effect of the order of installation of three additional but related programs. Let's call your software program "A", and the other three programs "B", "C", and "D". Using the techniques in this column you can easily determine that there are 24 installation orders, and easily generate those permutations. But suppose you are considering 19 other related programs. You will conclude that the 20! installation orders are just too many to test exhaustively so you'll have to prioritize your effort.
To summarize, string permutations are rearrangements of a set of strings. The ability to understand and use string permutations is a fundamental skill for software testers. In particular all testers should know how to calculate how many permutation elements there are for a given set of initial strings, produce all possible permutation elements, and efficiently generate a specified permutation element. This month's column showed you exactly how to accomplish those tasks. The code presented here can be used as is or can be modified to meet your own needs.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006). Reach him at jmccaffrey@volt.com or v-jammc@microsoft.com.


Service Station
Web Service Software Factory
Aaron Skonnard

Contents
About Software Factories
Web Services Software Factory
Getting Started with Service Factory
Prescriptive Architecture
Data Access Guidance
Generating Business Layer Code
Service Layer Tasks
Clients and Testing
Customizing the Solution
Summary

As architects and developers, we always want to be confident that we're solving problems in the right way. For this reason, there has always been a demand for good guidance in the developer world. Often, this guidance has come from the community of developers, sharing and learning from each others' experiences. But nowadays, with so many emerging technologies, there's often very little communal experience to build on. This makes it increasingly important for software vendors to provide valuable guidance. After all, these vendors have designed and created the tools and technologies, so they should have good advice as to how to properly use these tools.
Guidance comes in many forms, but it can generally be divided into two types: written guidance and code-driven guidance. Written guidance is the most common type. It includes architectural overviews and whitepapers, general guides to patterns and practices (which are typically technology-agnostic), and guidance tailored to specific technologies in the form of primers and "how to" descriptions.
Code-driven guidance, on the other hand, takes the form of source code. It can be sample code or perhaps a complete reference implementation that demonstrates how numerous patterns come together in a real-world solution. Code-driven guidance can be a reusable library or an automated code recipe. Code recipes automate a particular development task, typically using designers or wizards to gather information from the developer for customizing the code.
To help developers, the patterns & practices (p&p) team at Microsoft works with product groups to develop official Microsoft guidance for various technologies. The p&p team publishes all of these forms of guidance, including complex reference applications and code libraries, which are known as application blocks (such as the Enterprise Library for the Microsoft® .NET Framework 2.0).

About Software Factories
The p&p team recently began publishing software factories, which take a more holistic approach to guidance. These are collections of guidance assets-tools, reusable code, documentation, reference implementations, and the like. They help automate the process of building software according to accepted patterns and predefined standards.
Generally speaking, a software factory is installed to extend your development environment, adding guidance-related tools and resources. For example, a software factory might include solution templates that make it easier to start a new application. And it might provide wizards and designers that you can use throughout the entire development cycle.
Software factories enhance the relationship between architects and developers. Architects can customize the various code recipes that are available with a software factory and then they can redeploy the customized software factory to development teams. This gives architects a practical mechanism for distributing their own guidance to developers.
A key aspect of software factories is automated tool support. While Visual Studio® has long been extensible via the Visual Studio Industry Partners (VSIP) program, this framework has been far from easy to use. So the p&p team created the Guidance Automation eXtensions (GAX) and the Guidance Automation Toolkit (GAT) to allow architects to more easily apply custom guidance to the Visual Studio experience.
GAX is a runtime built on VSIP and is used in extending Visual Studio with guidance packages. Meanwhile, GAT is a toolkit used for implementing guidance packages that run on GAX within Visual Studio. GAX must be installed if you want to use an existing guidance package, such as those found in the software factories. GAT, on the other hand, is only required if you are going to build guidance packages. (It's important to note that anyone can create new guidance packages with GAT. You can even start with an existing software factory, customize its guidance packages, and then rebuild them.) For more information on GAT, check out the "Introduction to the Guidance Automation Toolkit".

Web Services Software Factory
The p&p team has recently released a few new software factories, including the Web Service Software Factory (sometimes referred to as the Service Factory), which I'll take a closer look at here. This software factory is designed to help developers build Web service solutions that consistently follow known architecture and design patterns.
There are two flavors of Service Factory: one for ASP.NET Web services (ASMX) and another for Windows® Communication Foundation (which will ship with the .NET Framework 3.0). The ASMX version is officially supported and available today. The official release of the Windows Communication Foundation version will coincide with the official release of the Windows Communication Foundation. However, the Windows Communication Foundation version of the Service Factory is currently available as a community release on the Web Service Software Factory community workspace found on GotDotNet.
Both versions install documentation and a complete reference application called the Global Bank Reference Application. Each also installs several guidance packages that provide Visual Studio automation. For example, the ASMX version installs two guidance packages-one for ASMX tasks and another for data access tasks. The Windows Communication Foundation version installs one for the Windows Communication Foundation tasks and another for security-specific tasks. The Guidance Package Manager (see Figure 1), which is available from the Tools menu in Visual Studio 2005, lets you enable or disable the available guidance packages.
To simplify this discussion, the remainder of this column will focus on using the ASMX Guidance Package. In a future column, I will cover the unique aspects of the Windows Communication Foundation Guidance Package.
Figure 1 Guidance Package Manager
Figure 1 Guidance Package Manager

Getting Started with Service Factory
When installed, the Service Factory creates a Microsoft patterns & practices | Web Service Software Factory item on the Start menu. Here, you'll find links to the various Service Factory assets. When you open Visual Studio 2005 and create a new project, you will find a project type called Guidance Packages. Each installed software factory provides templates that show up here. And if you have GAT installed, you'll see a package for creating new guidance packages (called Guidance Package Development).
Select Web Service Software Factory (ASMX) and one of its recipes is displayed as a template on the right. This recipe creates the initial solution structure according to the architecture prescribed by Service Factory. When you select the template and press OK, the recipe is invoked. This recipe first asks you for a prefix that it will use when naming each of the projects it's about to create (I've specified Coho.ClubServices.Membership as a prefix). When you press Finish, the recipe creates all of the projects; the generated solution is your starting point (see Figure 2).
Figure 2 Generated Solution Structure
Figure 2 Generated Solution Structure
When you use a guidance package, Visual Studio provides a Guidance Navigator (see Figure 3). This allows you to navigate the various guidance assets made available by each guidance package you're using.
Figure 3 Guidance Navigator
Figure 3 Guidance Navigator

Prescriptive Architecture
The structure of the initial solution created by the guidance package complies with one of the prescriptive architectures defined by Service Factory. This architecture is primarily targeted at enterprise-level Web services. But remember, if the specifics don't match your needs, you can always customize the guidance package and modify the initial solution template.
The prescriptive architecture is divided into three layers: the service interface layer, the business layer, and the resource access layer. Note how these layers map to the generated solution structure shown in Figure 2. This architecture embodies the Layered Application pattern (described on the patterns & practices site), which greatly simplifies dependency management in large systems.
Each layer consists of different software components that are closely related to one another. The architecture also defines some concerns that apply to all of the layers-notably security, operational management, and communication. The overall architecture is illustrated in Figure 4.
Figure 4 High-Level Service Factory Architecture
Figure 4 High-Level Service Factory Architecture
The components of the service interface layer include the service contract and the service adapter. The service contract defines what operations the service can perform but it does not include any behavior, such as how the operation is actually implemented. This requires defining interfaces, which are groups of operations defined in terms of message exchange patterns. For each operation, you define the message types used in the exchange, and you define each message type in terms of the composable data types. The service adapter implements the service contract that is exposed on an endpoint (commonly referred to as the service host) and is responsible for adapting the endpoint to the underlying business layer.
The business layer consists of three components: business entities, business logic, and business workflows. Business entities are classes that model real-world, domain-specific objects. They are different than the data types used in the service contract because they include behavior and perhaps state. The architecture encapsulates business entities so that they are not exposed across a service boundary. This ensures more flexibility within each layer. It also gives you the opportunity to format the data in different ways, catering them to specific integration scenarios. However, this also means entity translation is necessary for moving between the layers.
The business logic implements the actual business behavior. These classes operate on the business entities to perform a desired action. Some business entities are quite simple while others take advantage of more complex logic.
Business workflows handle long-running processes that require sophisticated message correlation and state management. They are typically implemented with a business process management product, such as BizTalk® Server 2006 or Windows Workflow Foundation.
The business layer operates on the underlying resource access layer, which provides access to both data access logic and service agents. The data access logic provides everything you need for interacting with the underlying data store while the service agents provide everything you need to interact with external Web services. Figure 5 provides a more detailed view of how these layers and software components are related to one another. Each of these components maps to one of the projects found in your initial solution. As you can see, the scope of Service Factory is very comprehensive. It provides guidance that begins with the consumer and continues all the way through to the back-end data sources.
Figure 5 Layer Components and Relationships
Figure 5 Layer Components and Relationships

Data Access Guidance
Service Factory includes the data access guidance package, which provides numerous recipes for automating common tasks in the data access layer. You typically launch recipes by right-clicking on a project and selecting the recipe from the Service Factory context menu. (There is one menu for each package enabled in a particular context.) You can also invoke recipes directly from the Guidance Navigator.
There are several common tasks that developers must perform to implement a data access layer. These tasks are often tedious and prone to error. One of the first recipes you'll want to use here is the Add database connection recipe. You can launch it by selecting it from the context menu in the Host project. This recipe asks you for a connection name and then has you specify the database connection details. Once you press finish, the recipe adds the specified connection string to the Host project's configuration file.
Then, assuming you already have your database model in place, you can generate a set of stored procedures that encapsulate typical operations (like select, insert, update, and delete) on a particular table in your database model. The recipe lets you choose the table and then you specify the name for each stored procedure operation (see Figure 6).
Figure 6 Generating Stored Procedures
Figure 6 Generating Stored Procedures
These simple recipes take much of the pain out of generating a lot of the code that's required for integrating your services with an existing database model. Beyond merely generating code, however, they also provide you with confidence that the implementation is done in accordance with proven best practices. And, handily, you can always add more recipes to fulfill your specific needs.

Generating Business Layer Code
The data access guidance package also provides a recipe for generating the business entities that correspond to the underlying database model. You can invoke this recipe from the context menu on the business entities project. The recipe asks you to choose a database connection and then it lists all of the available tables and views. You then specify the tables and views you want to use and then, for each one, you can customize the names of each business entity class and the fields they contain.
The generated business entities classes model the design of the original table. The code that is generated contains a constructor for populating a new instance with supplied data, using properties to access the underlying fields representing the table columns. Although you can generate most of your business entity layer using this technique, you will surely need to extend or customize the generated code. The recipe generates these entities as partial classes, giving you a mechanism for extending or regenerating them safely without modifying the generated code directly.
The business logic project is where you have to manually write most of the code for this layer-your business logic classes should operate on your business entity classes. Since business logic is very specific to any particular application, there's not much that can be automated in this area.
You can, however, generate the data repository classes that know how to integrate between the generated business entities and the underlying stored procedures you generated. You use the Create data repository classes from business entities recipe for doing this. This recipe asks you to define a mapping between a particular business entity and a stored procedure, and then you define the name and type of each method you want to generate for the repository class. Once you've completed this, you have the classes necessary for moving back and forth between the data and business layers.

Service Layer Tasks
The ASMX guidance package provides several recipes that automate common service layer development tasks. Since the prescriptive architecture doesn't allow business entities to be exposed across service boundaries, the first thing you need to do in the service layer is define the various data types you use to construct messages.
The Create data type recipe allows you to define an XmlSerializer-compatible class annotated with the appropriate serialization attributes. And you can do this without touching any code. The recipe provides a heuristic (based on the project namespace, the year, and the month) for choosing XML namespaces when defining these types. Then it asks you to specify the data members that will make up the structure of the type.
If you already have existing XML Schema definitions, you can use them to define data types using a different recipe. Simply add the XML Schema file to the data type project, right-click on it, and select Create data type from the schema. This recipe gives you support for schema-first design scenarios in your development process.
Once you have all of your service layer data types in place, you can define your message types using the Create message type recipe. This is available in the service contracts project. The recipe asks you to name the classes to represent the request and response messages and then to specify the data types that will make up each message structure (see Figure 7). It then generates more XmlSerializer-compatible classes to represent the messages.
Figure 7 Creating Message Type
Figure 7 Creating Message Type
With all the message types in place, you're ready to design your service contract definitions with the Create service contract recipe. This recipe first asks you to specify the name and namespace of the service contract. Then you list the operations you want to support on the service contract. For each one, you specify the method name (the Action) followed by the corresponding request/response messages the operation maps to.
Once the service contract details are defined, you can instruct the recipe to generate the service implementation class (see Figure 8). The generated ASMX class will implement the service contract you defined earlier in the recipe.
Figure 8 Generating Service Implementation Class
Figure 8 Generating Service Implementation Class
The implementation of each ASMX WebMethod should invoke the business logic classes defined in the business layer. However, in order to do this, the WebMethods must first perform a translation between the service contract types and the business entity types used in the business logic. Since this is a common and tedious task, there is a recipe available for generating the translation code. This recipe, called the Create service contract translator recipe, is available on the service implementation project by default. It provides a user interface for defining the mapping between the two classes and it generates the code in a helper translator class. All you have to do then is use the generated translator classes within your WebMethod implementations before invoking the appropriate business logic.
After you're finished implementing the service and you're ready to expose it to the outside world, you run the Expose service recipe to generate the .asmx endpoint that references your ASMX implementation class. Now you're ready to test it.

Clients and Testing
You may have noticed that the original recipe that generated the solution structure automatically produced a client application that can be used as a test harness for the service layer. To make this client useful for your particular service scenarios, you may need to write a little code.
There are several Service Factory recipes for building and testing clients. For example, Service Factory provides an Add service reference recipe, a Debug client recipe, and a Run client recipe. There are also some testing-related recipes available on the service host, such as the View in browser and Debug host recipes.

Customizing the Solution
You might decide that the solution generated by Service Factory is overkill for your particular project. Or it simply might not be suited exactly to your organization's needs or standards. To address this, you can customize the solution structure by simply modifying what's generated by the original recipe and then reassigning project responsibility for the different layers in the prescribed architecture.
In order to reassign project responsibility, you must unlock the solution by selecting Unlock solution on the Service Factory menu. Then, right-click on service contract project and select Specify project responsibility. The resulting dialog allows you to easily assign various responsibilities (see Figure 9).
Figure 9 Specifying Project Responsibility
Figure 9 Specifying Project Responsibility
Once you're finished making changes to the solution, you relock it. You can also export the modified solution template so that you can potentially rebuild a customized version of the guidance package based on your modified solution.

Summary
The Web Service Software Factory provides valuable guidance assets that help increase speed, quality, and consistency when developing Web service solutions. It includes useful written guidance and a complete reference application that illustrates the prescribed architecture, patterns, and practices in action. In addition, the Service Factory provides several guidance packages that automate many common development tasks.
I recommend downloading the Web Service Software Factory and checking out its many features. And check out the suite of hands-on labs on MSDN® (available on the GotDotNet site) for a step-by-step walkthrough.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a cofounder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services 2.0, Applied BizTalk Server 2006, and Introducing Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


.NET Matters
Deserialization Progress, and More
Stephen Toub

Code download available at: NETMatters2006_12.exe (161 KB)
Browse the Code Online

Q In my Windows® Forms application, I use a BinaryFormatter to save a large amount of application state, sometimes serializing to disk very large objects. At a later point, this state file needs to be loaded and deserialized by the application, and this deserialization process can take a measurable amount of time. Currently for both serialization and deserialization, I show a marquee progress bar, but that only provides an indication to the user that something is happening. For deserialization, I'd really like to show a standard progress bar that indicates how much of the deserialization has completed and how much still remains. BinaryFormatter doesn't provide any such feature, and I'm at a bit of a loss for how to extend the class to provide the behavior I need. Do you have any suggestions?
Q In my Windows® Forms application, I use a BinaryFormatter to save a large amount of application state, sometimes serializing to disk very large objects. At a later point, this state file needs to be loaded and deserialized by the application, and this deserialization process can take a measurable amount of time. Currently for both serialization and deserialization, I show a marquee progress bar, but that only provides an indication to the user that something is happening. For deserialization, I'd really like to show a standard progress bar that indicates how much of the deserialization has completed and how much still remains. BinaryFormatter doesn't provide any such feature, and I'm at a bit of a loss for how to extend the class to provide the behavior I need. Do you have any suggestions?
A BinaryFormatter is a difficult beast to extend, as it has a very simple interface. It's sealed with no virtual members, so you can't derive from it to override some key member like you might to add functionality to another class. And it doesn't expose any events you can hook to listen for progress notifications and the like. Instead, you'll need to get a bit creative.
A BinaryFormatter is a difficult beast to extend, as it has a very simple interface. It's sealed with no virtual members, so you can't derive from it to override some key member like you might to add functionality to another class. And it doesn't expose any events you can hook to listen for progress notifications and the like. Instead, you'll need to get a bit creative.
Getting quickly to the punch line, BinaryFormatter consumes the stream to be deserialized as is needed to parse it; this means that if we can figure out how much of the stream the BinaryFormatter has read, we can create a good approximation for how much of the deserialization process has been completed.
Figure 1 shows a first attempt at creating a utility method that allows you to track the progress of deserialization. The generic method accepts the stream to be deserialized and a callback delegate that will be invoked by the method to alert the caller of progress updates. Internally, it creates a BinaryFormatter and deserializes the stream, but before doing so, it first creates a Timer set to expire each second. Thus, after every second the Timer will invoke the callback delegate on a background thread, examining the current position of the stream and the length of the stream, and using that information to supply the callback with a number representing the percentage of deserialization completed. Unfortunately, as you can probably glean from the figure title, this method is rife with bugs. Most importantly, instance methods on Stream are not guaranteed to be thread-safe, so we shouldn't be accessing the Position and Length properties from a secondary thread as we're doing here. Additionally, there are several race conditions that will cause exceptions to be thrown, such as if deserialization finishes and the stream is closed just as the Timer's callback is being raised (you can't access the properties of a disposed FileStream). In short, this is a bad solution.
Figure 1 Buggy Deserialization Progress Tracking
public static T Deserialize<T>(
    Stream stream, ProgressChangedEventHandler callback)
{
    if (stream == null) throw new ArgumentNullException("stream");
    if (!stream.CanSeek) throw new ArgumentException("stream");

    Timer timer = null;
    if (callback != null)
    {
        timer = new Timer(delegate
        {
           callback(null, new ProgressChangedEventArgs(
               (int)(stream.Position * 100.0 / stream.Length), null));
        },
        null, 0, 1000);
    }

    BinaryFormatter formatter = new BinaryFormatter();
    T graph = (T)formatter.Deserialize(stream);

    if (timer != null) timer.Dispose();

    return graph;
}
A much better solution involves the creation of a custom decorator stream. I discussed the decorator design pattern and its application to streams in my September 2005 column, but in short, a decorator is an object that has the same interface as another object it contains (in object-oriented terms, it is an object that has both an "is-a" and a "has-a" relationship with a specific type). So, for example, System.IO.Compression.GZipStream is a decorator, as it both derives from Stream and contains a Stream. A user constructs a GZipStream by supplying to its constructor a Stream that should be compressed or decompressed, and calls made to GZipStream's Stream implementation delegate to the underlying Stream's implementation, doing additional work before and after.
Instead of supplying to BinaryFormatter the actual Stream to be deserialized, we can supply to it a surrogate decorator Stream that contains the Stream to be deserialized. This container stream can implement its Read method to retrieve the data from the contained Stream, but also to raise an event that contains information about the current position and length of the stream.
Whenever I set out to implement a new decorator stream, I typically start with a helper base class, like the one shown in Figure 2. This ContainerStream class provides a constructor that accepts and stores the Stream to be contained, a protected property exposing to derived types the underlying Stream, and then implements all of the methods and properties on Stream to delegate to the contained Stream. This is useful since most of the methods and properties on Stream are abstract, and this helper lets you avoid implementing every method in your own derived type if all you're interested in doing is implementing one or two specific methods.
Figure 2 ContainerStream
public abstract class ContainerStream : Stream
{
    private Stream _stream;

    protected ContainerStream(Stream stream)
    {
        if (stream == null) throw new ArgumentNullException("stream");
        _stream = stream;
    }

    protected Stream ContainedStream { get { return _stream; } }

    public override bool CanRead { get { return _stream.CanRead; } }

    public override bool CanSeek { get { return _stream.CanSeek; } }

    public override bool CanWrite { get { return _stream.CanWrite; } }

    public override void Flush() { _stream.Flush(); }

    public override long Length { get { return _stream.Length; } }

    public override long Position
    {
       get { return _stream.Position; }
       set { _stream.Position = value; }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        return _stream.Read(buffer, offset, count);
    }

    ...
}
My ReadProgressStream implementation is shown in Figure 3. It derives from ContainerStream and overrides the Read method. The Read override calls to the base class's Read method (which in turn calls the contained Stream's Read), and then if anyone has registered with ReadProgressStream's ProgressChanged event, it raises the event with the current percentage of progress made reading through the stream (to improve performance, it only raises the event if the integral percentage has changed in value since the last call to Read).
Figure 3 ReadProgressStream
public class ReadProgressStream : ContainerStream
{
    private int _lastProgress = 0;

    public ReadProgressStream(Stream stream) : base(stream)
    {
        if (!stream.CanRead || !stream.CanSeek || stream.Length <= 0)
            throw new ArgumentException("stream");
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        int amountRead = base.Read(buffer, offset, count);
        if (ProgressChanged != null)
        {
            int newProgress = (int)(Position * 100.0 / Length);
            if (newProgress > _lastProgress)
            {
                _lastProgress = newProgress;
                ProgressChanged(this,
                    new ProgressChangedEventArgs(_lastProgress, null));
            }
        }
        return amountRead;
    }

    public event ProgressChangedEventHandler ProgressChanged;
}
With this class in place, tracking the progress of deserialization is straightforward, and can be done simply by passing a ReadProgressStream rather than the actual stream to be deserialized to the BinaryFormatter:
public static T Deserialize<T>(
  Stream stream, ProgressChangedEventHandler callback)
{
  using (ReadProgressStream rps = new ReadProgressStream(stream))
  {
    rps.ProgressChanged += callback;
    BinaryFormatter formatter = new BinaryFormatter();
    return (T)formatter.Deserialize(rps);
  }
}
This works, accomplishing the goal of being able to generally track the progress of deserialization. Unfortunately, as the Deserialize<T> method is currently written, it has a major performance problem. The BinaryFormatter makes many, many requests to the Stream's Read method, frequently for only a few bytes at a time. While we haven't added much code to the Read method, it's enough to create a significant bottleneck.
To improve performance, we need a way to throttle the number of times the ReadProgressStream's Read method is actually called. To solve this, it's important to realize that this problem isn't new. (Very few problems actually are, and life gets a lot easier if one accepts that most problems have already been solved one way or another.)
For example, FileStream is typically used to read and write data from disk, and disk is relatively slow. If every request to a FileStream required disk access, applications that relied on reading and writing files with FileStream would slow to a crawl. Instead, FileStream buffers data from the disk, reading more than it needs to on small read requests such that future requests for that data may be satisfied out of the buffer cache rather than requiring additional accesses to the disk.
That same solution of buffering data can be used to fix our performance problem. We want to limit the number of times ReadProgressStream's Read method is actually invoked, and one way to do that is by forcing the caller to make several large requests rather than many small requests. And doing that requires adding just one more line of code:
public static T Deserialize<T>(
  Stream stream, ProgressChangedEventHandler callback)
{
  using (ReadProgressStream rps = new ReadProgressStream(stream))
  using (BufferedStream bs = new BufferedStream(rps))
  {
    rps.ProgressChanged += callback;
    BinaryFormatter formatter = new BinaryFormatter();
    return (T)formatter.Deserialize(bs);
  }
}
Now instead of the BinaryFormatter directly accessing the Read ProgressStream, it goes through a System.IO.BufferedStream. This dramatically improves the performance of the solution to the point where it's actually usable. However, it introduces a new minor issue, albeit one we can overcome with a few more lines of code. The constructor for BufferedStream used in the previous example defaults the buffer size to 4KB. This means if we're dealing with a relatively small stream, we're not going to make very many calls to the underlying ReadProgressStream's Read method, and we won't get many progress notifications. Given that it's a small stream, and thus one that should deserialize very quickly, this might not be an issue for you. If, however, it is a problem and you need more frequent updates, you can change the size of the buffer using a second constructor on BufferedStream that accepts the buffer size to use in addition to the stream to wrap. An easy way to determine the buffer size is to take the length of the stream and divide it by a hundred (since the ProgressChangedEventArgs uses an integral value for percentage completed, thus resulting in typical values falling between 0 and 100, inclusive), but for extremely large streams, you might end up with huge buffer sizes. To counteract that, I've chosen to go with whatever's smaller, 1/100th of the stream's length, or 4096 bytes, and my solution that includes that calculation is shown in Figure 4. You should of course test this formula in your own scenarios, and make changes if this is an inappropriate choice for your system.
Figure 4 Deserialization with ProgressChanged Callbacks
public static T Deserialize<T>(
    Stream stream, ProgressChangedEventHandler callback)
{
    if (stream == null) throw new ArgumentNullException("stream");

    using (ReadProgressStream cs = new ReadProgressStream(stream))
    {
        cs.ProgressChanged += callback;

        const int defaultBufferSize = 4096;
        int onePercentSize = (int)Math.Ceiling(stream.Length / 100.0);

        using (BufferedStream bs = new BufferedStream(cs,
            onePercentSize > defaultBufferSize ?
            defaultBufferSize : onePercentSize))
        {
            BinaryFormatter formatter = new BinaryFormatter();
            return (T)formatter.Deserialize(bs);
        }
    }
}

Q I'm using the SortedList<TKey, TValue> collection from the System.Collection.Generics namespace, and the performance is not what I'd expect. What sort of algorithms are used under the covers for this collection?
Q I'm using the SortedList<TKey, TValue> collection from the System.Collection.Generics namespace, and the performance is not what I'd expect. What sort of algorithms are used under the covers for this collection?
A Like List<T>, SortedList<TKey, TValue> wraps an underlying array. (SortedList actually wraps two arrays, one for keys and one for values). When you add an item to SortedList, it uses a binary search to find the correct place to insert an item into array, ensures that the array is large enough to hold the existing items plus the new item, shifts the existing items to make room for the new item, and adds the new item to the array in the correct location. Depending on where you're inserting into the list, that shift operation can be expensive. Take the following two loops as an example. The first inserts the numbers from 1 to 10,000 into the list, whereas the second inserts the same numbers but in the opposite order, starting with 10,000 and going down to 1:
for (int i = 1; i <= 10000; ++i) list.Add(i, i); // loop #1

for (int i = 10000; i >= 1; --i) list.Add(i, i); // loop #2
A Like List<T>, SortedList<TKey, TValue> wraps an underlying array. (SortedList actually wraps two arrays, one for keys and one for values). When you add an item to SortedList, it uses a binary search to find the correct place to insert an item into array, ensures that the array is large enough to hold the existing items plus the new item, shifts the existing items to make room for the new item, and adds the new item to the array in the correct location. Depending on where you're inserting into the list, that shift operation can be expensive. Take the following two loops as an example. The first inserts the numbers from 1 to 10,000 into the list, whereas the second inserts the same numbers but in the opposite order, starting with 10,000 and going down to 1:
for (int i = 1; i <= 10000; ++i) list.Add(i, i); // loop #1

for (int i = 10000; i >= 1; --i) list.Add(i, i); // loop #2
On my machine, the first loop is more than 30 times faster than the second, and that gap will continue to grow as the number of elements increases. What's happening here is that in the first loop I'm always adding to the end of the array, which means no elements need to be shifted on an insertion. In the second loop, I'm always adding to the beginning, which means all elements need to be shifted on every insertion. This degrades the algorithm to an O(n2) algorithm, the worst case complexity for an insertion sort of this nature.
The moral of the story is that you need to pick your data structures carefully based on the scenarios in which you plan to use them. Moreover, once you've chosen a data structure, make sure you're using it in a manner that provides optimal performance. As I've just shown, simply changing the order in which you perform certain operations can have a significant impact.

Q I need to convert a standard file path to an 8.3 file path. Does any support for this exist in the .NET Framework?
Q I need to convert a standard file path to an 8.3 file path. Does any support for this exist in the .NET Framework?
A Sure, but only in the sense that through P/Invoke you have access to the entire suite of Win32® APIs. This conversion can be accomplished easily using the GetShortPathName function exposed from kernel32.dll, as shown in Figure 5. It accepts three parameters: the string containing the path to be converted, an output memory buffer to contain the converted string, and an integer containing the size of that buffer in characters. If GetShortPathName is called with a buffer that's too small to receive the converted string (or, more specifically, if the value in cchBuffer is too small), GetShortPathName will return the size of the buffer required to store the string.
A Sure, but only in the sense that through P/Invoke you have access to the entire suite of Win32® APIs. This conversion can be accomplished easily using the GetShortPathName function exposed from kernel32.dll, as shown in Figure 5. It accepts three parameters: the string containing the path to be converted, an output memory buffer to contain the converted string, and an integer containing the size of that buffer in characters. If GetShortPathName is called with a buffer that's too small to receive the converted string (or, more specifically, if the value in cchBuffer is too small), GetShortPathName will return the size of the buffer required to store the string.
Figure 5 GetShortPathName
[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
public static extern uint GetShortPathName(
    string lpszLongPath, StringBuilder lpszShortPath, uint cchBuffer);

private static string GetShortPathName(FileInfo file)
{
    StringBuilder buffer = new StringBuilder();
    string path = file.FullName;
    uint length = GetShortPathName(path, buffer, 0);
    if (length > 0)
    {
        buffer.Capacity = (int)length;
        GetShortPathName(path, buffer, length);
        return buffer.ToString();
    }
    return null;
}
You can take advantage of this by making two calls to GetShortPathName; the first passes in a buffer size of 0 in order to force GetShortPathName to return the needed buffer size. A second call to GetShortPathName is then made with the appropriately sized buffer. When calling through P/Invoke Win32 functions that expect a pointer to a string buffer to contain the output value, you typically use a StringBuffer, preallocated through its Capacity property with a buffer big enough to hold the output string. After the P/Invoke, the result can be retrieved easily using StringBuffer's ToString method.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


Netting C++
The Design Space of the Common Type System
Stanley B. Lippman

Contents
Native Enum vs. Managed Enum
Single Class Solution
Five-Class Solution
Six-Class Solution

Back in the June 2006 installment of Netting C++, I began a series of columns that delve into moving a working native C++ application onto the Microsoft® .NET Framework using the C++/CLI language extensions in Visual C++® 2005. The sample application I am moving is a Text Query Language (TQL), which was developed back in 1996 for the third edition of my book, C++ Primer.
In June, I started the series by showing you how to wrap the native TQL application, and then in August 2006 I went through the code and cleaned it up. Last month I gave an overview of using regular expressions in the .NET Framework. You can find these previous columns at Netting C++ (June 2006) and Netting C++ (August 2006).
TQL is designed to support two operations:
  • Normalization of an arbitrary text file specified by the user into a map of line number occurrences for each unique word.
  • Handling of user queries against that map to display matching lines of text.
The purpose for designing the text normalization is to illustrate nontrivial uses of collection classes. In the native implementation, it uses the Standard Template Library (STL). In a subsequent column, I will look at this design to explore the generic collections namespace of the .NET Base Class Library (BCL). The purpose for designing the text query support is to illustrate a nontrivial but basic-level class design. And it is this aspect of C++/CLI programming design that I look at in this column.
TQL allows the user to query a text file using a set of logical tokens that represent relationships-&& for And, || for Or, and ! for Not. So, to find all occurrences of Holmes, you would simply type "Holmes" (not including the quotes). If you want to find all the text lines in which Holmes does not occur, you would simply prepend Holmes with the Not operator (!Holmes). If your goal is to find all occurrences of either Sherlock or Holmes, you would connect the two words with the Or operator (Sherlock || Holmes). Finally, to find any occurrences in which Sherlock immediately precedes Holmes, you would connect the two words with the And operator (Sherlock && Holmes).
Admittedly, in today's world of search engines, this seems quite paltry. But for examining various class design strategies, it ably fits the bill. And remember, the native implementation is over a decade old-what I'm doing here is adapting it to the .NET Framework using C++/CLI.
The candidate abstractions of the design are the four types of queries a user can make. Let's represent them initially as a set of enumerations, as follows:
enum EQueryType { // native enum
    qWord = 1, qNot, qOr, qAnd
};
I've chosen to do this for two reasons. First, you'll need this enum for one of my design solutions, and second, I wanted to provide a brief tutorial on enum support with C++/CLI.

Native Enum vs. Managed Enum
The first question programmers new to C++/CLI ask is what are the differences between the native enum I just showed you and the equivalent managed enum, which looks like this:
class enum EQueryType : Byte { // managed enum
    qWord = 1, qNot, qOr, qAnd
};
There are three primary differences.
The first difference is the granularity of size management. In this example, I specify that a byte of storage is sufficient to represent each of the four enumerator values. This feature has been extended to native support of C++ as well, but only in Visual C++ (it is non-standard).
A second difference is the scope of the enumerators. In a managed environment, an enum maintains its own scope, thereby encapsulating the visibility of its enumerators in the same way a class encapsulates the visibility of its members. This is, of course, not true of the native enum, in which the enumerators all spill into the enclosing scope. This can become a problem when combining modules from multiple sites. To prevent enumerators from polluting the global namespace, a design idiom of native C++ is to nest enums within a class, like so:
class ios {
public:
  enum iostates { read, write, append };
  // ...
}
To access any of these enumerators, you prepend the enumerator with the class scope operator, such as ios::read. This is not a good design idiom for managed enums since it will result in two layers of encapsulation, such as ios::iostates::read. Therefore, a managed enum is typically defined within the shared scope of the class it is designed to support rather than within that class, as with native enums.
The third difference is that a managed enum is more like an Object than an integer. This permits the correct resolution of overloaded functions, as in the following:
void f( int );
void f( Object^ );

EQueryType et;

f( et ); // which one?
If managed enums were treated as an integral type-as are native enums-this invocation would have to match f(int). However, given that the backing Enum class within the Common Type System (CTS) is a Value type derived from Object, its type must clearly bind to Object, not int. So it was decided that a managed enum is a kind of Object and not a kind of integer.
I hope this is clear, as I don't have the space to argue this decision more forcefully. While this is a reasonable decision, there is an unfortunate side effect: there is no implicit conversion of a managed enum or one of its enumerators to int, and so each use as an arithmetic value requires an explicit cast, which is a real bear.
So the TQL application has four concrete types that it needs to manipulate: Word, And, Or, and Not. But how many classes do you have? My design solutions yield 1, 5, or 6 classes. Let's step through the design math and see where that leads you.

Single Class Solution
When I start a project, my first design effort is always minimalist. I try to reduce everything to a single class abstraction. That inevitably requires a space trade-off. For example, a query is either unary or binary. A single representation, then, would need to support a binary query and for each unary query you swallow the additional unnecessary operand. Similarly, to squish four abstractions into one general representation, you need a discriminant by which to identify it. For example:
public ref class Query sealed {
 EQueryType discriminant;
 String^  op1; // always used
 String^  op2; // maybe used
 // ...
If you're like me, you'll likely squint at the explicit use of string instances as operands within the Query, but that's an implementation issue rather than a design one and I'll look at that in a future column. I'm using real code just so you can all become familiar with the C++/CLI syntax.
This design is adequate for representing simple queries, such as rain || snow. But it does not work for compound queries, where either operand might be a subquery rather than a single word, as in wind && ( rain || snow ).
The operand fields are the sticking point. The operands need to be able to support substitutability. A query can be viewed as having a tree structure-each leaf node is a word while each interior node represents a compound operation. The representational problem is finding a type for your operands that can reference both a leaf and an interior node type. That common type provides the requisite substitutability and is referred to as polymorphism.
In my one class solution, I again trade space for flexibility, as shown here:
public ref class Query sealed {
 EQueryType isA;
 String^    word;
 Query^     op1;
 Query^     op2;
 // ...
While this is not necessarily the preferable design, it is a legitimate solution, which requires one class to represent the four candidate abstractions.
Because I am programming under .NET, there is a system-defined solution to the problem of substitutability: Object. In .NET, Object represents a common base type between any two .NET types. That's a fundamental aspect of a unified type system, such as the UTS. Therefore, I could revise my native-minded design as follows:
public ref class Query sealed {
 EQueryType isA;
 Object^    op1;
 Object^    op2;
 // ...
In this manner, op1 and op2 can be either a String or Query, and so I no longer need to throw in the entire kitchen sink.

Five-Class Solution
A four-class design solution can also be devised through the use of Object for each requisite operand, as follows:
public ref class Word sealed { String^ word; ...
public ref class And sealed  { Object^ op1; Object^ op2; ...
public ref class Or sealed   { Object^ op1; Object^ op2; ...
public ref class Not sealed  { Object^ op; ...

void resolveQuery { Object^ query );
However, by substituting a domain-specific fifth class in place of Object, I gain significantly both in elegance and performance. This five-class solution is the canonical object-oriented design.
The fifth class is a common Query abstraction from which each of the actual Query types are derived. Simply substitute this abstract Query class in place of the universal abstract Object, as follows:
public ref class Query abstract { ...

public ref class Word : Query { String^ word; ...
public ref class And : Query { Query^ op1; Query^ op2; ...
public ref class Or : Query { Query^ op1; Query^ op2; ...
public ref class Not : Query { Query^ op; ...

void resolveQuery { Query^ query );
Again, this is the canonical object-oriented solution in which you define a shared interface within the abstract Query class and override type-specific operations in each derived class. Query is preferable to Object because it retains and restricts each referenced object to be a kind of Query. Thus you can directly manipulate the object rather than having to downcast it to its explicit type.
The Query class itself can either be a pure abstract class in which all implementation is pushed down into the derived classes or it can be a hybrid class in which shared implementation is managed within the Query class itself. But again, this is an implementation issue rather than a design issue.

Six-Class Solution
The six-class solution makes use of the generic facility of the common Language Runtime (CLR) 2.0. In this scenario, the mapping of a class hierarchy to a parameterized design is a three-step process.
First, the abstract Query class introduced into your object-oriented design is turned into a parameterized class (that is one of the six classes), like so:
generic <typename queryType>
ref class Query { ...
This class provides an equivalent shared interface to that of the abstract Query base class it replaces.
In order to allow that interface to be recognized by the CLR, I need to publish and constrain an interface. This is the second of the six classes. The interface class represents the set of operations each parameterized type bound to Query must implement and therefore can safely be invoked from within the generic class. Not only do I need to define the interface class, but I must also modify the generic declaration to specify it within the constraint clause introduced by the where keyword, as follows:
interface class IQuery { ...

generic <typename queryType>
 where queryType : IQuery
ref class Query { ...
(For a full discussion of the constraint or where clause, see my Pure C++ column from June 2005.)
Finally, I need to define the four explicit query class types. Each of these can be independent and sealed, but must implement the IQuery interface in order to be bound legally to the generic Query class. This looks something like the following:
ref class Word sealed : IQuery { String^ word; ...
ref class And sealed  : IQuery { IQuery^ op1; IQuery^ op2; ...
ref class Or sealed   : IQuery { IQuery^ op1; IQuery^ op2; ...
ref class Not sealed  : IQuery { IQuery^ op; ...
Note that, to save space, I have left off the public keyword.
There are no doubt several alternative designs you might hit upon. But this column pretty much traverses the abstraction facilities of the .NET Common Type System using C++/CLI. In my next column, I'll choose one of these designs and walk you through the implementation. Until then, may all your exceptions be handled gracefully.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. Later, Stan worked in feature animation at both Disney and DreamWorks and served as a Software Technical Director on Fantasia 2000. He has since served as Distinguished Consultant with JPL, and an Architect with the Visual C++ team at Microsoft.


{End Bracket}
And Along Came 10...
Duncan Mackenzie

If you are a developer focused on Microsoft products, you've probably been to Channel 9, where you can chat with fellow developers and watch videos that take you behind the scenes at Microsoft. Well, not content to offer just one massive online video project, the team behind Channel 9 decided to build a similar site targeted at the technology enthusiast who isn't necessarily a programmer. The new site, on10.net, covers a wide variety of topics, from the latest in medical technology to the work that goes into developing an Xbox 360™ game, all with a focus on how things work.
When we started this project in January 2006, we had only three programmers and approximately five weeks to get the site up and running. Our first big architectural decision was that, instead of building on top of an existing code base such as Channel 9 or Community Server, we would build the site from scratch. We had a very clear idea of the look and functionality we wanted to have on the site, and one of the essential features was the ability to let users comment on videos while watching them. This meant that we couldn't require postbacks to log in, to comment, or to page through comments, all of which pointed us at an AJAX implementation.With an AJAX system, info is sent back and forth from the browser through Web requests initiated from a client-side script, allowing asynchronous updating of page content without necessitating a full page refresh. One of the key requirements of an AJAX system, though, is a fairly complete Web service layer, and that was something that none of our current systems offered. As a result, instead of retrofitting a service-style architecture on top of an existing code base, we built a simple blog/forum engine, implementing only those features that were critical to the site.
We had another reason for treating on10.net as a completely new system: we were after a minimalist design and didn't want anything in the code base that wasn't part of our intended feature set. We identified a handful of key elements: Entries, Comments, Tags, and Media files. Every type of posted information would be an Entry, thus putting comments, threads, forum posts, and blog posts all into a single table and object. Tags were added as a list of IDs and names, along with the concept of Tag Groups, allowing us to classify a tag as "Content" or "Location" or even "Appearance." Media files were handled through associating a set of links with each entry, with the link describing the type of file and a URL.
Another reason to start fresh was our desire to make the site as compliant as possible with the current crop of Web standards. Producing valid XHTML output was one goal, as was making sure the site and the videos worked in as many browsers (and on as many platforms) as possible. In order to reach the Apple iPod and Sony PSP audience, we output our content in both Windows Media® and MPEG4 formats, which may be a first for a Microsoft-run Web site. We use the Windows® Media player when running inside Internet Explorer®, playing back the WMV video file, but switch to the QuickTime embedded player and a MP4 file when running inside any other browser. Being a blog, we also needed to properly support RSS along with Pingbacks and Trackbacks and some features specific to a video blog such as special iTunes extensions.
We tried to build on top of existing technology wherever possible. For instance, we took advantage of ASP.NET 2.0 for all the required authentication plumbing, including the membership tracking of Users, Roles, and Profiles, allowing us to avoid writing much of that code. All of the underlying page generation is handled through the use of the Virtual Path Provider, another new feature in ASP.NET 2.0. We are also using the Microsoft® ASP.NET "Atlas" framework primarily to provide a way to call our Web services from client-side JavaScript and handle moving our data around as easy-to-use objects. We also utilized other services such as MapPoint® and Virtual Earth™ for mapping and plotting location coordinates.
All in all, there are quite a few requirements to put up a working blog, but we ended up with a great foundation on which to build new features as on10.net evolves and as we move Channel 9's massive body of content onto this new code base. In addition to on10.net and Channel 9, the team has also built a smaller single-person blog engine that we have currently deployed to our personal sites (see duncanmackenzie.net as an example). Once we have worked out the kinks in this code base, code-named "Oxite," we'll be shipping that code out (in C# and the .NET Framework 2.0). Check out Welcome to 10 for more information on our team and for links to two videos where we describe the system in juicy geeky detail.

Duncan Mackenzie is a developer for on10.net and Channel 9. He can be reached through his personal site.