 MSDN Magazine August 2004

MSDN Magazine August 2004

Reporting: Deliver User-Friendly Reports from Your Application with SQL Server Reporting Services

SQL Server 2000 Reporting Services--a server-based reporting platform built on the .NET Framework and integrated with SQL Server 2000--lets you easily add reporting from diverse data sources. Using Visual Studio and Reporting Services you can integrate reports from any data source that has an OLE DB, ODBC, or ADO.NET provider into your Web applications. This article explains how it's done.

John C. Hancock GridView: Move Over DataGrid, There's a New Grid in Town!

When incorporating the ASP.NET DataGrid control into your Web apps, common operations such as paging, sorting, editing, and deleting data require more effort than you might like to expend. But all that is about to change. The GridView control--the successor to the DataGrid-- extends the DataGrid's functionality it in a number of ways. First, it fully supports data source components and can automatically handle data operations, such as paging, sorting, and editing, as long as its bound data source object supports these capabilities. In addition, the GridView control offers some functional improvements over the DataGrid. Here DataGrid expert Dino Esposito introduces the GridView and explains all its long-awaited features.

Dino Esposito Genetic Algorithms: Survival of the Fittest: Natural Selection with Windows Forms

Genetic Programming is an evolutionary algorithm that employs reproduction and natural selection to breed better and better executable computer programs. It can create programs that implement subtle, non-intuitive solutions to complex problems. By taking a well-known example from the Genetic Programming community and implementing it with the .NET Framework, this article demonstrates that CodeDOM and Reflection provide all the facilities that are needed to do Genetic Programming effectively.

Brian Connolly Web Parts: Use Windows SharePoint Services as a Platform for Building Collaborative Apps, Part 2

Microsoft Office SharePoint Portal Server 2003, which is part of the Office System, lets you create and manage virtual servers, site collections, sites, workspaces, and users. You can also use the Windows SharePoint Services object model to design and implement user-targeted applications. In the second part of a two part series, the authors take a look at the WSS and SPS object models, Web Part Page anatomy, creating and deploying Web Parts, and Web Part security. They also discuss Web Part infrastructure and how to create custom Web Parts.

Jason Masterman and Ted Pattison Virtual Server 2005: Program Customized Testing Environments Without Trashing Your Machine

Efficient testing can require many different machine and server configurations but the effort needed to manage these test environments can take a toll. Microsoft Virtual Server 2005 simplifies the provisioning and management of testing environments. This article demonstrates how the extensive Virtual Server COM API can be used to create an automated application testing environment to make your testing chores a whole lot easier.

Ben Waldron Editor's Note: New Technologies and a New Magazine

Hey, remember the Web? Back around 1996 it was big news. People suddenly realized that with just a telephone line and Internet Explorer, they could go shopping without ever leaving the comfort of their home! Over the Web you could buy furniture, art, books, and appliances, all of which would make your home comfortable enough to stay there and do some more online shopping, accelerating the vicious cycle of consumerism.

New Stuff: Resources for Your Developer Toolbox

Blue Marble Geographics has released its developer toolkit, GeoTransform 5. 0. GeoTransform is a DLL for programmers using Visual Basic®, C++, Delphi, PowerBuilder, C++ Builder, and other Windows®-based development tools.

Theresa W. Carey Bulletin Board: News from Online Communities

This month's column introduces you to Code Project, a vast site of resources for Web development. Code Project is a member of the Microsoft Code Wise Community, a source for tips, code samples, news, and advice from experts outside of Microsoft.

Web Q&A: Page Ready State, DataGrid Row Deletion, and More

Edited by Nancy Michell Data Points: Migrating from ADO to ADO.NET, Part 2

As you move forward with your use of ADO. NET, you'll need to know how to approach situations that you previously learned to handle in ADO and now have to tackle with ADO. NET. Just as n-tiered solutions developed using Visual Basic®, C++, and ASP often rely on ADO for their data access needs, Windows® Forms, Web Forms, and Web services rely on ADO.

John Papa Cutting Edge: Script Callbacks in ASP.NET

If you're involved in Web development you may have faced a problem that you couldn't find a good solution for—making client-to-server calls outside the current page. For example, you might want to validate the content of a textbox against data stored on the server asynchronously, without interrupting the continuity of the work or without forcing a full page refresh, which is particularly heavy for UI-rich pages.

Dino Esposito The XML Files: What's New in WSE 2.0

Microsoft has recently released Web Services Enhancements for Microsoft® . NET (WSE) 2. 0. WSE 2. 0 provides extensions to the existing ASP. NET Web services framework (. asmx) as well as a standalone messaging framework that's completely transport independent.

Aaron Skonnard Test Run: Test Automation for ASP.NET Web Apps with SSL

If you're encrypting user data with Secure Sockets Layer (SSL) over HTTP and you want to test your Web applications programmatically you'll find that the techniques are not widely known. In this month's column I'll show you how to set up a test SSL server and write test automation that verifies the functionality of a simple but representative Web application.

James McCaffrey Wicked Code: Foiling Session Hijacking Attempts

Let's face it: every minute of every day, someone, somewhere, is patrolling the Web looking for sites to hack. ASP. NET developers must constantly be on their guard to ensure attempted hacks can't be successful.

Jeff Prosise Advanced Basics: P2P Comm Using Web Services

Iwanted to use my first Advanced Basics column as an opportunity to strike out into new territory, to do something I haven't seen extolled much in the literature, so I've built a Windows® Forms chat program that uses Web services to communicate with other peers.

Carl Franklin .NET Matters: Debugger Visualizations, Garbage Collection

Stephen Toub C++ Q&A: GetKeyState, the STL String Class, Exposing C++ Objects, and More

Paul DiLascia { End Bracket }: C# to Visual Basic Translation Tool

Having talked to thousands of developers who use the Microsoft . NET Framework, I've heard one consistent complaint: "I really wish all the samples were written in my programming language. " Nothing is more frustrating than having braved the wilds of Internet searches for a snippet of code that does exactly what you want but is written in a language you don't use.

John Robbins


Reporting: Deliver User-Friendly Reports from Your Application with SQL Server Reporting Services

Reporting

Deliver User-Friendly Reports from Your Application with SQL Server Reporting Services

John C. Hancock
This article discusses:
  • An introduction to Reporting Services
  • Designing and deploying reports
  • Using the Reporting Services Web service
  • Securing reports
This article uses the following technologies:
SQL Server, ASP.NET, Visual Basic .NET
Code download available at: SQLServerReportingServices.exe (222KB)
Contents
An Overview of Reporting Services
Designing Your First Report
Deploying and Testing
Adding Reports to a Web App
Embedding Reports into Your Web App
Securing Your Reports
Using the Reporting Services Web Service
Subscribing to a Report
What's Next?

F
lexible reporting capabilities are a requirement for most business applications and their integration into Web apps makes them more versatile than ever. With the recent release of SQL Server™ 2000 Reporting Services, you can easily add reporting from diverse data sources. In this article I will introduce report authoring using Visual Studio® and Reporting Services and demonstrate how to integrate reports into your Web applications.

Reporting Services is a server-based reporting platform that is built on the .NET Framework and integrated with SQL Server 2000, so you can integrate rich reporting features into your apps using an extensive Web services-based API. Although the report server uses SQL Server as the repository for reports, any data source with an OLE DB, ODBC, or ADO.NET provider can be used to supply data to the reports, making Reporting Services a great choice for reporting in diverse enterprise environments.

Reporting Services is licensed as part of SQL Server 2000, so if you already have a licensed copy of SQL Server, you can run Reporting Services on the same server for no additional license fee. If you decide to run Reporting Services on a separate machine that is not yet licensed for SQL Server, such as a Web server, you will need to purchase an additional SQL Server license.

To get started with Reporting Services, you can download the 120-day trial software at SQL Server 2000 Reporting Services, which also includes directions on how to obtain the installation media for the product. Make sure to check the option to include the sample reports when you install the product since I will be integrating these reports into a Web application in this article.

An Overview of Reporting Services

Reporting Services reports are designed using a new Visual Studio .NET 2003 report designer, which is available when you install Reporting Services on the same machine as Visual Studio. The designer outputs a description of the report layout and data access in the new XML-based Report Definition Language (RDL), and the RDL files are published to a report server. Report designers from third parties are also available if you want to provide authoring capabilities to your users.

Administrators can manage the published reports using the Web-based Report Manager, and perform tasks such as securing reports for certain groups of users or altering the data source connection strings for deployed reports. Users can also use Report Manager to browse and view reports; however, many companies will prefer to integrate report viewing directly into their existing internal Web sites or applications, using either URL requests or the Reporting Services Web service.

Figure 1 Reporting Services Architecture
Figure 1 Reporting Services Architecture

Reports can be parameterized so that users can pick from a list of selections when viewing the report; they can also be exported to a wide range of formats including Microsoft® Excel, PDF, and XML. While live reports provide up-to-the-minute data, reports can also be cached for a period of time to improve performance and to reduce the load on data sources. For business intelligence applications, reports can access Analysis Services OLAP cubes, and Reporting Services can even import existing Microsoft Access reports, although not all Access features are supported due to differences between the two technologies. See Figure 1 for a high-level look at the Reporting Services architecture.

Designing Your First Report

Although I will focus on integrating reports into applications rather than report authoring, the best way to familiarize yourself with Reporting Services is to dive right in and create a new report project. To do this, open Visual Studio and create a new project using the Report Project Wizard, located in the new Business Intelligence Projects section. Specify a name for your project, click OK, and the report wizard dialog will be displayed. During the step when you select the data source, click the Edit button to specify a connection to AdventureWorks2000 on your database server, click OK, and then check the option to make it a shared data source.

Click Next to move to the Design the Query step, specify a simple SQL statement such as "SELECT * FROM Product", and click Next. Reporting Services supports standard Tabular reports which show a fixed set of columns, as well as Matrix reports which use a crosstab-style view with dynamic columns, so choose Tabular for this report and click Next.

Drag some columns, such as ProductID and Name, to the Details section and click Next. Choose the default Bold style, click Next, then specify the URL of the virtual directory of your report server—for example, http://MyServer/ReportServer. Now, click Next. On the final step, you can specify a name for your first report such as Products Report; then click Finish.

The wizard will create the project, along with an AdventureWorks2000 data source that can be shared by multiple reports, and a report on the Products data which is opened in the designer (see Figure 2). The Layout tab, which allows you to modify the report design, is displayed by default.

Figure 2 Visual Studio Report Designer
Figure 2 Visual Studio Report Designer

The Data tab is used to specify the queries for your report, including setting up parameters so that the user can be prompted to supply values when the report is executed. Click the Preview tab to test how your report will appear to a user.

You can now add additional reports to the project or modify the Products report by adding additional columns, grouping the data and adding subtotals, or even adding additional tables which display data from other queries. Unlike many report designers, Reporting Services uses an out-of-band design which makes it easy to create reports that include data from a variety of different sources all brought together in one place.

The report project that you have just created consists of a set of files which currently reside on your development machine. In order to publish these reports to your users, you will need to deploy the project to the report server.

Deploying and Testing

Before deploying the report project, you will need to specify the server to which you are deploying your project. If you initially created your report project using the wizard, then you would have already specified a target server.

To check this setting or specify a new server, select Project | Properties to display the Project Properties dialog. Setting this allows you to specify both a folder name for your project and the URL of the Reporting Services Web service. The Build | Deploy Solution menu selection will build your report project and then deploy it to the server.

Now that the report project has been deployed, you can take it for a test drive. Every report that has been deployed to a server has a unique path that can be used to display the report in a browser. For example, to display the first report you created, open a browser and navigate to http://MyServer/ReportServer?/Report+Project1/Products+Report. This will render the report in HTML and display a toolbar at the top of the page to allow you to page through the report, change the zoom, or even view the report in a variety of other formats, such as a PDF.

You can also browse a list of all the report projects that have been deployed to the server by navigating to the following URL on your server: http://MyServer/ReportServer. The report server provides a minimal user interface that allows you to see what reports have been deployed and view the reports.

The Report Manager tool, which is friendlier and more feature-rich, can be accessed by simply navigating to the following URL on your local server: http://MyServer/Reports.

Adding Reports to a Web App

Now that you know how to design, deploy, and test reports, let's integrate these reports into an application. Many Web applications include a customized home page for every user, often referred to as a dashboard. This page usually presents summarized information that is tailored to the user and serves as a jumping off point to access the other functionality in the application.

I will be building a dashboard to demonstrate report integration, shown in Figure 3, using the sample reports that ship with Reporting Services. If you installed the sample reports during setup, you can open the report project (installed to \Reporting Services\Samples\Reports) and deploy it to your test server. You can test the sample reports using the Report Manager tool by browsing to the SampleReports folder.

Figure 3 Dashboard Web App
Figure 3 Dashboard Web App

The simplest way to provide users with access to reports from your application is to use a hyperlink. As you have seen, every report has a unique path that can be used to display it in a browser. For a dashboard application, the report should be opened in a separate window, so I would specify a target for the hyperlink. That's very simple so far, but what about reports that have parameters, such as the Employee Sales Summary report? To really integrate a report into an application, you often need to specify some values for the parameters so that the user is not always prompted.

Reporting Services makes this task very easy: values for report parameters can be specified as part of the URL. For example, the Employee Sales Summary report has ReportYear, ReportMonth, and EmpID parameters, so the hyperlink to display November sales for employee 24 would be:

<a href="http://MyServer/ReportServer?/SampleReports/Employee Sales
   Summary&ReportMonth=11&EmpID=24" target="_blank">November Sales Summary
</a>

Parameters that don't have a value specified in the URL will use the default value that is set in the report designer, so the sample sales summary report just shown uses a ReportYear value of 2003. Reporting Services also has a set of built-in parameters that can be used to control how the report is displayed. These are distinguished from your own report parameters by prefixing them with "rs:".

One of the most useful is the rs:Format parameter, which is used to specify the format in which to render the report. This allows you to include reports that don't necessarily get displayed in HTML, but use formats such as PDF or even XML. The other commonly used parameter is rs:Command, which indicates the action to apply to the path that you have specified. For example, rs:Command=Render will render a report, while rs:Command=ListChildren will list all of the items in a folder.

If you don't specify a command, Reporting Services will look at the path you have specified and work out the appropriate action to take, such as rendering a report. Figure 4 shows a list of rs: parameters.

Figure 4 Report Server rs: Parameters (Close Figure 4)

Parameter | Description | Values
rs:Command | The action to apply to the path, such as render a report or list all of the items in a folder. Defaults to Render for reports. | Render, ListChildren, GetResourceContents, GetDataSourceContents
rs:Format | The format in which to render the report, such as HTML4.0 or PDF. Defaults to the appropriate HTML supported by the browser. | HTML3.2, HTML4.0, HTMLOWC, MHTML, IMAGE, EXCEL, CSV, PDF, XML (can also include custom rendering extensions if available)
rs:Snapshot | Renders a report snapshot, which uses report data captured at a specific point in time and stored in the reporting server database. | Timestamp of snapshot—for example, 2004-03-30T15:45:02
rs:SessionID | Identifies an established active session between the browser and the report server. This is used when the report server has been configured to not use session cookies. | Existing session id
rs:ClearSession | Forces the client browser to clear its cached copy of the report and retrieve a new copy. | True

Another set of built-in parameters controls the behavior of the various output formats and uses the rc: prefix. Each report format has its own specific set of parameters. For example, to render a report in comma-separated values (CSV) format with no header row, you should use the rc:NoHeader parameter:

<a href="http://MyServer/ReportServer?/SampleReports/Company Sales
   &rs:Command=Render&rs:Format=CSV&rc:NoHeader=true">Company Sales
</a>

The HTML format has the widest range of rc: parameters, which gives you a lot of flexibility when integrating reports into a Web application. If your application passes report parameter selections to the report, you may want to avoid prompting the user by setting rc:Parameters to false, or you can even turn off the entire toolbar by setting rc:Toolbar to false. See Figure 5 for a list of commonly used HTML rc: parameters.

Figure 5 Commonly Used rc: Parameters for HTML Format (Close Figure 5)

Parameter | Description | Values
rc:Toolbar | Shows or hides the toolbar, including the parameters area. Defaults to true. | true, false
rc:Parameters | Shows or hides the parameters area of the toolbar. Defaults to true. | true, false
rc:Zoom | Zoom factor to zoom in or out on a report. Defaults to 100. (Note that this setting only works with Internet Explorer 5.0 and above) | Page width, whole page, or an integer percentage such as 75.
rc:Section | The page number to render. The default value is 1. | Integer
rc:LinkTarget | The target name for hyperlinks in the report. | The name of a window or another valid target name such as _blank, _self, _parent, or _top.

If your Web application uses frames to display reports, let's say with a list of reports in a left pane and the report content in the right pane, you will need to use the rc:LinkTarget parameter to specify the name of your content frame. Otherwise, when the user clicks any embedded links in the report, the browser will redisplay the report using the whole window rather than preserving your frames for the navigation pane and the content pane.

You can use the URL parameters I've described to add the Quick Launch area, which contains a list of hyperlinks to allow the user to open common reports (see Figure 3). Each hyperlink has its target set to _blank to open the report in a separate browser window and uses parameters such as rs:Format=EXCEL to control how the report is displayed in the browser.

Embedding Reports into Your Web App

Rather than popping up a new browser window whenever a user views a report, you may want to actually embed the reports into your own Web pages. The simplest way to do this is to use an IFRAME on your Web page and set the SRC attribute using the same URL-based technique I described. If you use this technique, remember to set the rc:LinkTarget parameter to the name of your IFRAME to avoid popping out of the frame when a user clicks a link in the report, as with any frame.

If you have previously designed server controls for ASP.NET, you may be thinking that the IFRAME technique is not particularly elegant since it requires Web developers to understand the URL parameter syntax for Reporting Services in order to build the appropriate SRC string. Wouldn't it be a better idea to build a server control that encapsulated all the parameters and allowed developers to simply set properties such as ReportPath and Zoom?

Reporting Services conveniently ships with a sample called ReportViewer (installed to the \Reporting Services\Samples\Applications\ReportViewer directory) which does exactly that. The control wraps the IFRAME and URL access parameter logic and provides a simpler method of embedding reports into your application. To use the control, open and build the ReportViewer solution, then switch over to your Web application and add the control to the toolbox by browsing to ReportViewer.dll in the bin directory of the ReportViewer solution.

You can use the control to add embedded reports to the dashboard application. Add the control to the Web page, set the height and width properties, and add the following code to display a report in your page:

Private Sub Page_Load(ByVal sender As System.Object, ByVal _
                      e As System.EventArgs) Handles MyBase.Load
    If Not IsPostBack Then
        ReportViewer1.ServerUrl = "http://MyServer/ReportServer"
        ReportViewer1.ReportPath = "/SampleReports/Sales Order" & _
            "Detail&SalesOrderNumber=so8153"
        ReportViewer1.Toolbar = ReportViewer.multiState.False
        ReportViewer1.Zoom = "75"
    End If
End Sub

Securing Your Reports

Reporting Services security uses a familiar role-based model. Users and groups can be assigned to roles such as System Administrator or Browser, and all members of a particular role will be allowed to perform operations that are defined for that role.

Report Manager is used to perform security-related tasks such as assigning users to roles or modifying the permissions for an item such as a report. By default, the Administrators windows group is a member of both System Administrator and Content Manager for all folders. In order to extend report access to other users, the user account or group must be added to a role such as Browser. This is usually set up at the folder level, but also can be overridden for individual reports.

In addition to securing access to reports, administrators must also decide how to connect reports to source databases. Data sources can be configured to use integrated security so that database connections are made using the current user's account. Alternatively, a specific Windows® or SQL Server user name and password can be defined and stored securely in the report server database. This approach is required for reports that will be run without user interactivity, such as those involved in subscriptions.

Using the Reporting Services Web Service

The URL access approach that I have been using so far provides a quick and easy way to view reports and specify parameters, but it doesn't provide access to all of the report management functionality that is available. Reporting Services also comes with a full-featured API, which gives you access to the advanced functions through a Web service. For example, the Web service includes methods to manage data source information for your reports or to enumerate all of the items in a folder.

I will be using the Web service to add a dynamic list of reports to the dashboard application. To get started, you will need to add a Web service reference to your application using Add Web Reference. Specify the path to the Web service, like so:

http://MyServer/ReportServer/ReportService.asmx
Once you have added the reference, you can create an instance of the ReportingService proxy class and invoke a method on the service, like I've done here:
Private Sub LoadSampleReports()
    Dim rs As New ReportingService
    rs.Credentials = System.Net.CredentialCache.DefaultCredentials

    Dim item As CatalogItem
    For Each item In rs.ListChildren("/SampleReports", False)
        If item.Type = ItemTypeEnum.Report Then
            DropDownList1.Items.Add(New ListItem(item.Name, item.Path))
        End If
    Next
End Sub
If you are deploying your application in an intranet environment, you will want to enable impersonation by adding <identity impersonate="true"/> to your application's web.config file. You will also want to pass the client's login credentials to the service, as shown in the previous example.

The ListChildren method returns an array of CatalogItem objects representing the items that the user has permission to view, including reports, folders, and data sources. The Path property gives the unique path to the item, such as /SampleReports/Sales Order Detail, and the Type property allows you to distinguish between reports and other types of items.

Now that I've loaded the report names into a dropdown list, I have to decide how to display the reports within the dashboard application. I decided to extend the ReportViewer code that I wrote earlier and change the ReportPath property whenever the user selects a report and clicks the Go button. You could also use an IFRAME and set the SRC property to the appropriate URL (and prefix the report name with "http://MyServer/ReportServer?").

In addition to report management functions, the other main feature that the Web service provides is report rendering. Most applications (including the Report Manager application) use the Web service features to list reports or alter settings, and use the URL approach I've described to display reports. However, if you want full control over how the report content is handled in your application, you can use the Render method of the ReportingService class. This method returns a byte array which you can then save to an output file, such as an image file or an Excel spreadsheet, depending on the requested format. If you want to render a report in HTML, things are a bit more complicated because you will need to use the RenderStream method to separately render any images or other resources.

Although this article has focused on Web applications, you can use the same techniques to add reporting to Windows-based applications. The easiest approach is still to use URL access and either launch the user's browser from your application or include the Microsoft WebBrowser ActiveX® control and use the Navigate method to load the appropriate URL. The Render method of the Web service is a good alternative when you want more control, for operations such as displaying the results in a picture box or saving the file directly to disk.

Information workers generally have to deal with two types of reports: the type they seek out when they need to gather information and the type that shows up on their desk or in their e-mail every week. So far I've concentrated on the process by which users fetch their own reports, but Reporting Services also includes a powerful feature called subscriptions that allows you to "push" reports to users.

Subscribing to a Report

The Reporting Services subscription feature lets users receive reports on a scheduled basis. Reports are generally e-mailed to the user, but Reporting Services also lets you deliver reports to a file share or even write your own delivery extension. This allows you to easily accomplish dissemination tasks like e-mailing a weekly sales total report to every sales rep, or setting up the monthly financial report to run on the last day of every month and write it out as a PDF to a company file share.

Before setting up a subscription, you need to think about how the report will connect to the database to retrieve its data. Since the subscribed report will not be executed directly by a user, you cannot use integrated security for the data source, but must specify a user name and password to be used when the report is run. If you try to set up a subscription for a report that uses integrated security, Reporting Services will return an error.

To change the data source for the sample reports, open Report Manager | Sample Reports, and click the AdventureWorks data source. This is shared by all the sample reports, so any changes made to this data source will affect all of the reports. Select the "Credentials stored securely in the report server" option, specify a valid domain user name and password of an account that has access to the AdventureWorks database, check the "Use as Windows credentials when connecting to the data source option," and click the Apply button. Note that this change means that the specified account will always be used to connect to the database, regardless of the user who actually runs the report.

So now you're ready to dive in and take a look at the Web service API that provides access to subscriptions. This part of the Web service is a little more challenging to use than the List or Render methods, mainly due to the number of options that are available. The method that I will be using is CreateSubscription, which also belongs to the ReportingService class. The parameters for this method are described in detail online (see ReportingService Class), but essentially you will need to specify which report you are subscribing to, the delivery schedule (such as every Monday morning or the last Friday of the month), the e-mail options including e-mail address, and any report parameter values.

I have included a utility class called EmailSubscriber in the sample code for this article, which abstracts some of the complexity. The dashboard application uses this class to allow the user to specify their e-mail address and to have the Employee Sales Summary report e-mailed to them every Monday. In a production application, you could tailor the report to each user, for example, by retrieving their employee ID based on their Windows user account and then setting the report parameters appropriately when you create the subscription.

If you don't want to wait until next Monday to test your subscription, it helps to know a little more about how Reporting Services handles subscription scheduling. When you create a subscription, Reporting Services creates a SQL Server Agent job which is executed according to the required schedule. You can see this job by opening Enterprise Manager and expanding the Management, SQL Server Agent, and Jobs folders. The subscription job will have a category of "Report Server" and is named with a GUID which is used by Reporting Services to keep track of the job. Right-click the job and choose Start Job, and if you have set everything up correctly your report will be e-mailed to you.

What's Next?

Two other Reporting Services features that you can use to add great reporting capabilities to your application include Data Driven Subscriptions, which allows you to set up a mailing list of users and e-mail specific parameterized reports to them, and Snapshots, which are views of the report at scheduled points in time that can also provide historical views of reports. If you need even more power, Reporting Services is built with a modular architecture, which allows you to add powerful new extensions using your favorite .NET-targeted language.

John C. Hancock is a senior consultant for Microsoft, specializing in business intelligence and .NET development. He is currently based in Toronto, and can be contacted at www.johnchancock.net.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


GridView: Move Over DataGrid, There's a New Grid in Town!

GridView

Move Over DataGrid, There's a New Grid in Town!

Dino Esposito
This article is based on the May 2004 Technology Preview of ASP.NET 2.0. All information herein is subject to change.
This article discusses:
  • The ASP.NET 2.0 GridView, FormView, and DetailsView
  • Differences between the DataGrid and the GridView
  • The programming interfaces of these controls
  • How to achieve master/detail views
This article uses the following technologies:
ASP.NET, ASP.NET 2.0, C#
Code download available at: GridView.exe (124KB)
Contents
GridView Versus DataGrid
GridView and Data Source Controls
The GridView Object Model
GridView Events
Displaying, Sorting, and Paging
Editing Data
The DetailsView Control
The FormView Control
Conclusion

D
espite the richness and versatility of its programming interface, the ASP.NET 1.x DataGrid control requires you to write a lot of custom code to handle common operations such as paging, sorting, editing, and deleting data. For example, while the DataGrid control can raise events when the user clicks to save or cancel changes, it doesn't offer much more than that. If you want to store changes to a persistent medium, such as a database, you have to handle the UpdateCommand event yourself, retrieve changed values, prepare a SQL command, and then proceed from there to commit the update.

The reason the DataGrid control limits the raising of events for common data operations is that it's a data source-agnostic control that can be bound to any data object that is enumerable. Implementing data operations such as update or delete would require a direct link with one particular data source. In ASP.NET 1.x, you work around this limitation by writing ADO.NET code that is specific to your application.

ASP.NET 2.0 enhances the data-binding architecture, introducing a new family of components—the data source objects—which act as a bridge between data-bound controls and ADO.NET objects. These source objects promote a slightly different programming model and provide for new features and members. For data reporting purposes, your ASP.NET 2.0 applications should use the newest grid control—the GridView. The familiar DataGrid control is still supported, but it doesn't take full advantage of the specific capabilities of data source components.

The GridView control is the successor to the DataGrid and extends it in a number of ways. First, it fully supports data source components and can automatically handle data operations, such as paging, sorting, and editing, provided its bound data source object supports these capabilities. In addition, the GridView control offers some functional improvements over the DataGrid. In particular, it supports multiple primary key fields and exposes some user interface enhancements and a new model for handling and canceling events.

The GridView comes with a pair of complementary view controls: DetailsView and FormView. By combining these controls, you can easily set up master/detail views using very little code and sometimes no code at all.

GridView Versus DataGrid

In ASP.NET 2.0, the class hierarchy of data-bound controls is more consistent than in ASP.NET 1.x. In version 2.0, all controls derive from the same base class (the BaseDataBoundControl class), regardless of the actual implementation or user interface characteristics. Figure 1 shows the new class diagram. The DataGrid and other version 1.x controls like Repeater and DataList are not included in the diagram. The inheritance tree for these existing controls is the same as ASP.NET 1.x. In particular, the Repeater inherits WebControl, whereas DataList and DataGrid inherit BaseDataList. As you can see in Figure 1, GridView is a composite data-bound control and shares a common set of methods and properties with all other data-bound controls, including DropDownList, DetailsView, and ListBox.

Figure 1 ASP.NET Class Diagram
Figure 1 ASP.NET Class Diagram

Although similar in their high-level functionality, the GridView and the DataGrid controls have a different foundation. To enable easy migration from existing pages, the GridView preserves the DataGrid object model as much as possible. However, you shouldn't expect 100 percent compatibility between DataGrid-based code and new GridView-based code.

Another key difference between the DataGrid and GridView controls lies in the adaptive user interface. Unlike the version 1.x DataGrid, the GridView can render on mobile devices, too. In other words, to build reports on mobile devices, you can use the same grid control you would use for desktop pages. The DataGrid in version 2.0 can also render adaptively, but its UI capabilities are not quite as rich as those of the GridView.

In ASP.NET 2.0, the DataGrid control has been enhanced to support common control features such as themes and personalization. In addition, the new DataGrid control can be populated by a data source control. Remember, though, that a DataGrid can only bind to a data source object for the purpose of reading data. To actually modify the underlying data source, some user-defined code is still required. In contrast, the GridView control takes advantage of the capabilities of the underlying data source and automatically deletes or updates records. Note that the GridView control also supports the classic binding mechanism based on the DataSource property and the DataBind method. Although fully supported, this programming practice is discouraged.

GridView and Data Source Controls

So, what's a data source control? I covered this hot new feature of ASP.NET 2.0 in great detail in the June 2004 issue of MSDN®Magazine. In summary, a data source control is a set of Microsoft® .NET Framework classes that facilitate two-way binding between data stores and data-bound controls. Existing controls like the DataGrid and new data-bound controls like the GridView can be bound to a data source, albeit with different capabilities.

A data source control represents the key functions of a data source: select, insert, update, and delete. Data source controls can represent any data source from relational databases to XML files, from streams to business objects. If this brief description makes you think of .NET managed providers, take a look at Figure 2.

Figure 2 Data Source Controls, GridView, and Data Sources
Figure 2 Data Source Controls, GridView, and Data Sources

Data source controls may sit on top of some .NET data providers and form an intermediate layer between data-bound controls and data sources. Data source controls also expose a common interface for a few basic operations. Some of the data-bound controls—specifically, the GridView control—bind to these commands to automate in-place editing along with other data-related operations.

A data source control uses its properties and methods to expose the bound content as a set of named views. The IDataSource interface that all data source controls implement provides the basic set of capabilities to retrieve views of data from a data source. ASP.NET 2.0 supplies a handful of built-in data source controls, as listed in Figure 3. The data source controls in Figure 3 belong to two categories: tabular and hierarchical components. The SiteMapDataSource and XmlDataSource components are hierarchical data source controls and can be used by hierarchical components like the TreeView and Menu control. The various other components manage tabular data.

Figure 3 ASP.NET 2.0 Data Source Controls (Close Figure 3)

Tabular Controls | Description
AccessDataSource | Represents a connection to an Access database. Inherits from the SqlDataSource control but ignores the ConnectionString and ProviderName properties in favor of a simpler DataFile property that will point to the MDB file. The control uses the Jet 4.0 OLE DB provider to connect to the database. If required for the MDB file, you can specify a user name and password. A ShareMode property allows you to specify whether the MDB file is read-only or read/write.
DataSetDataSource | Works with the XML representation of a DataSet object. The XML data can be specified as a string or through a file name. You can't bind this control to a DataSet object, not even programmatically. The class features methods to retrieve the corresponding DataSet object and set schema information. Although XML-based, the control supports only the tabular interface and can only be bound to list controls. Mainly used to display XML data in read-only scenarios, the control also supports editing of the underlying XML data.
ObjectDataSource | Allows binding to a custom .NET business object that returns data. The class is specified by name through the TypeName property. The control allows developers to structure applications using a three-tier architecture and still take advantage of the ASP.NET 2.0 declarative data-binding model. The class is expected to follow a specific design pattern and include, for example, a parameterless constructor and methods with a well-known behavior.
SqlDataSource | Represents a connection to an ADO.NET data provider that returns SQL data, including data sources accessible through OLE DB and ODBC. The name of the provider and the connection string are specified through properties. Do not use this class to connect to an Access database.
Hierarchical Controls | Description
SiteMapDataSource | Gets data from an ASP.NET 2.0 sitemap source. The actual source depends on the site provider that the application uses. In the default case, the data source is an XML file with a .sitemap extension. The control pumps site map information into hierarchical data-bound controls such as the new TreeView control.
XmlDataSource | Loads XML input data from a file, URL, or string containing the XML content. If the XML data does not also contain schema information, the XML schema may be specified as a separate file or string. The control can be used to populate hierarchical controls such as TreeView or Menu.

The code in Figure 4 shows a sample page that binds a GridView and a DataGrid to the same data source control. This is the recommended data-binding approach in ASP.NET 2.0. The SqlDataSource control is characterized by a ConnectionString property plus any combination of SelectCommand, UpdateCommand, InsertCommand, and DeleteCommand properties. All of them are string properties and reference a command text, optionally with parameters:

<asp:SqlDataSource runat="server"
  ID="MySource"
  ConnectionString="SERVER=(local);
  DATABASE=northwind;Integrated Security=SSPI;"
  SelectCommand="SELECT * FROM employees WHERE employeeid > @MinID">
  <SelectParameters>
     <asp:ControlParameter Name="MinID"
          ControlId="EmpID"
          PropertyName="Text" />
  </SelectParameters>
</asp:SqlDataSource>

Figure 4 Binding GridView and DataGrid (Close Figure 4)
<%@ Page theme="SmokeAndGlass" %>
<html>
<head runat="server" />
<body>
  <form runat="server">
    <asp:TextBox runat="server" ID="EmpID" Text="3" />
    <asp:button runat="server" Text="Refresh" />

    <asp:SqlDataSource runat="server" ID="MySource"
      ConnectionString="SERVER=(local);DATABASE=northwind;Integrated
          Security=SSPI;"
      DataSourceMode="DataSet"
      SelectCommand="SELECT firstname, lastname FROM employees WHERE
          employeeid > @MinID">
      <SelectParameters>
            <asp:ControlParameter Name="MinID" ControlId="EmpID"
                PropertyName="Text" />
      </SelectParameters>
    </asp:SqlDataSource>

    <asp:GridView runat="server" ID="grid"
         DataSourceId="MySource"
         AutoGenerateColumns="true">
    </asp:GridView>
  </form>
</body>
</html>

A data source control is characterized by a unique ID. This ID is the link that ties data-bound and data source controls together. You bind a GridView to a data source control using the DataSourceId property. For example, whenever the grid needs to fetch data, the SelectCommand SQL command of the associated SQLDataSource control executes. When the grid needs to update or delete a record, the corresponding UpdateCommand or DeleteCommand SQL commands execute. If no such commands exist, an exception is thrown. Internally, the GridView raises events like the DataGrid version 1.x does when the user deletes or updates a record. Unlike the DataGrid, though, the GridView defines internal handlers for these events. The default handlers retrieve the commands defined on the bound data source to handle these operations and execute them. As Figure 4 demonstrates, no code is needed behind the markup to keep the grid displaying or updating data. In more complex and sophisticated scenarios, you may need to write some code.

Figure 5 GridView and DataGrid
Figure 5 GridView and DataGrid

In general, though, data source controls and the GridView control provide for codeless data binding. Figure 5 shows the output generated by the code shown in Figure 4.

In ASP.NET 2.0, the DataGrid exposes the DataSourceId property in addition to DataSource and DataMember. The DataSourceId property connects the DataGrid to a valid data source object defined on the same page. However, the DataGrid doesn't provide the same level of automation the GridView provides. When the user clicks to update a record, the DataGrid raises the UpdateCommand event, whereas the GridView retrieves the data source update command and runs it in addition to raising the Updating and Updated events to allow the user to customize the information sent to the data source control.

The GridView Object Model

The overall structure of a GridView looks similar to that of a DataGrid. Some of the common elements have been renamed and some of the common features now require a different syntax. All in all, though, if you're familiar with the DataGrid control, you'll immediately feel at home with the GridView. Figure 6 details the new elements that form a GridView (note that some of them, like DetailLinkStyle, are available only when the grid is rendered to mobile devices). Row elements are exposed through instances of the GridViewRow class gathered in the Rows collection. The GridViewRow class maps to the DataGridItem class whereas Rows clearly replaces the DataGrid's Items collection. Represented by the DataControlRowType enumeration, the type of the row indicates the position and the role (for example, footer, header, pager, and data row). The GridView also introduces a new concept—the state of a row element. This row state is represented by the values in the DataControlRowState flags enumeration—Normal, Edit, Alternate, Insert, and Selected. Interestingly enough, these two enum types happen to be shared by all of the data view controls—GridView, DetailsView, and FormView.

Figure 6 GridView Properties (Close Figure 6)

Style Property | Description
AlternatingRowStyle | Every other row displayed in the table
DetailLinkStyle | Links in the details view of the control when rendered on mobile devices
DetailTitleStyle | Title in the details view of the control when rendered on mobile devices
EditRowStyle | Row being edited
FooterStyle | Grid's footer
HeaderStyle | Grid's header
EmptyDataRowStyle | Default row rendered when the GridView is bound to an empty data source; can be templatized using the EmptyDataTemplate property
PagerStyle | Grid's pager; can be templatized using the PagerTemplate property
RowStyle | Single row displayed in the table
SelectedRowStyle | Row currently selected
SummaryTitleStyle | Title in the summary view of the control when rendered on mobile devices

Aside from the elements introduced to comply with adaptive rendering, the GridView has only one other new element—the empty data row. When bound to an empty data source, the GridView can optionally display some default content to give feedback to the user. What is displayed on this occasion depends on the contents of the new empty data row element. You can set this row's content using either a property (EmptyDataText) or a template (EmptyDataTemplate).

The properties of the GridView control fall into three main categories: behavior, visual settings, and state. Figure 7 lists some of the GridView properties. Take a look at the new properties including EnableSortingAndPagingCallbacks, EmptyDataText, and UseAccessibleHeader, plus properties that implement functions already supported by the DataGrid that have been renamed or reworked.

Figure 7 GridView Properties (Close Figure 7)

Property | Description
AutoGenerateColumns | Indicates whether bound fields are automatically created for each field in the data source.
AutoGenerateDeleteButton | Indicates whether the control includes a button column to let users delete the record mapped to the clicked row.
AutoGenerateEditButton | Indicates whether the control includes a button column to let users edit the record mapped to the clicked row.
AutoGenerateSelectButton | Indicates whether the control includes a button column to let users select the record mapped to the clicked row.
BottomPagerRow | Returns a GridViewRow object that represents the bottom pager of the grid.
Columns | Gets a collection of objects that represent the columns in the grid. Note that this collection is always empty if columns are autogenerated.
DataKeyNames | Gets and sets an array that contains the names of the primary key fields for the currently displayed items. Extends and replaces DataKeyField.
DataKeys | Gets a collection of DataKey objects that represent the values of the primary key fields set in DataKeyNames for the currently displayed records.
DetailNextRowText | Indicates the text for the link to the next row in the details view on mobile devices.
DetailPreviousRowText | Indicates the text for the link to the previous row in the details view on mobile devices.
DetailSummaryText | Indicates the text for the link to the summary view when the control is rendering on a mobile device.
EmptyDataText | Indicates the text to render in the control when bound to an empty data source.
EnableSortingAndPagingCallbacks | Indicates whether paging and sorting should be accomplished using script callback functions. Disabled by default.
PagerSettings | Gets a reference to the PagerSettings object that lets you set the properties of the pager buttons. The PagerSettings object groups all the pager-specific properties.
Rows | Gets a collection of GridViewRow objects that represent the data rows currently displayed in the control. Replaces the Items property of the DataGrid.
SelectedDataKey | Returns the DataKey object for the currently selected record.
SelectedRow | Returns a GridViewRow object that represents the currently selected row. Replaces the SelectedItem property of the DataGrid.
SelectedValue | Similar to SelectedDataKey; returns the explicit value of the key as stored in the DataKey object.
SortDirection | Read-only property; gets the direction of the column current sort.
SortExpression | Read-only property; gets the current sort expression.
SummaryViewColumn | Indicates the name of the data column used to provide the summary of the record when the control renders on mobile devices.
TopPagerRow | Returns a GridViewRow object that represents the top pager of the grid.
UseAccessibleHeader | Determines whether to render <TH> tags for the column headers instead of default <TD> tags.

The programming model is a bit different for button columns. In an ASP.NET 1.x DataGrid, you had to add a special type of column to create an Edit button—EditCommandColumn. To create a Delete or a Select button column, you had to add a generic button column and give it a predefined command name. The GridView object model is more consistent and simpler. It is based on three new Boolean properties: AutoGenerateEditButton, AutoGenerateDeleteButton, and AutoGenerateSelectButton. When any of these properties are set to true, an Edit, Delete, or Select command button column is added to the grid, respectively. For example, when the AutoGenerateEditButton property is set to true, a column with an Edit button for each data row is automatically added to the grid. These buttons can also be added manually by adding a CommandField object to your set of columns. The Columns property lists the column objects much like the Columns property of the DataGrid does. A few helper properties have also been added based on customer feedback. In particular, you can now store multiple key values for each displayed row. Basically, the DataGrid's DataKeyField string property has been expanded to an array of field names. The new property is named DataKeyNames and stores an array of strings made up of the field names that uniquely identify a data row. DataKeys is the corresponding array of values for a particular row. It returns a collection of DataKey objects. Each DataKey object contains the values for each key name, and there are as many DataKey objects in DataKeys as there are visible rows of the GridView.

SortDirection and SortExpression track the current sorting of the grid. These properties are used internally to implement auto-reverse sorting and to enhance the grid with a glyph that indicates the current sort order. PagerSettings groups in a single object all the properties that configure the user interface, behavior, and position of the pager. The pager now supports a navigation style that also includes first and last rows along with the next and previous rows.

The GridView control can also use a lighter-weight, callback-based mechanism for sorting and paging. You turn this feature on and off by setting the EnableSortingAndPagingCallbacks Boolean property. When a sorting or paging link is clicked and callbacks are enabled, the GridView requests the sorted data or next page with no observable page postback. (A round-trip happens, but there is no page refresh so you don't notice.) Note that this feature comes with a caveat: when selection is enabled in the GridView, the currently selected index is maintained on the new page. If there's an associated details page, the selection appears out of sync. Handling events like PageIndexChanging doesn't help because these events are never fired if callbacks are enabled. Finally, you should keep in mind that the callback-driven paging and sorting mechanism requires Microsoft Internet Explorer 5.0 or higher.

GridView Events

The GridView control doesn't feature methods other than the well-known DataBind method. In ASP.NET 2.0, many controls, and the Page class itself, feature pairs of Pre-load/Post-load events. Key operations in the control lifecycle are wrapped by a pair of events, one firing before the operation takes place and one firing immediately after the operation has completed. This is true of the GridView class as well. The list of new events is shown in Figure 8. Events that announce operations significantly enhance your programming power. For example, by hooking the RowUpdating event you can check what is being updated against the new values. You might want to handle the RowUpdating event by HTML-encoding the values supplied by the client before they are persisted to the underlying data store. This simple trick helps you fend off malicious script injections.

Figure 8 GridView Events (Close Figure 8)

Event | Description
PageIndexChanging | Both events fire when one of the
and PageIndexChanged | pager buttons is clicked. They fire before and after the grid control handles the paging operation.
RowCancellingEdit | Fires when the Cancel button of a row in edit mode is clicked, but before the row exits edit mode.
RowCommand | Fires when any button in a button column is clicked. Replaces the DataGrid's ItemCommand event.
RowCreated | Fires when a row is created. Replaces the DataGrid's ItemCreated event.
RowDataBound | Fires when a data row is bound to data. Replaces the DataGrid's ItemDataBound event.
RowDeleting and | Both events fire when a row's
RowDeleted | Delete button is clicked. They fire before and after the grid control deletes the row.
RowEditing | Fires when a row's Edit button is clicked, but before the control enters edit mode.
RowUpdating and | Both events fire when a row's
RowUpdated | Update button is clicked. They fire before and after the grid control updates the row.
SelectedIndexChanging | Both events fire when a row's
and SelectedIndexChanged | Select button is clicked. The two events occur before and after the grid control handles the select operation.
Sorting and Sorted | Both events fire when the hyperlink to sort a column is clicked. They fire before and after the grid control handles the sort operation.

The availability of a pre/post pair of events lets you cancel an ongoing event based on runtime conditions. Take a look at the following code snippet:

void PageIndexChanging(object sender, GridViewPageEventArgs e)
{
   // Is this the sensitive page? (> 4)
   bool isSensitivePage = (e.NewPageIndex > 4);
   if (isSensitivePage && (User.Identity.Name != "username"))
      e.Cancel = true;
   return;
}
Cancel is a read/write Boolean property present in all event argument classes derived from CancelEventArgs. Many of the GridView's event argument classes inherit CancelEventArgs, meaning that all those events can be canceled. The value of the Cancel property is always set to false when the "pre" event fires. By handling the event, you can check some conditions and optionally cancel the event by setting the Cancel property to true. For example, the code snippet just shown cancels the transition to the new page if the current user is not authorized to view pages with an index greater than 4.

Displaying, Sorting, and Paging

A grid is often used to display the results of a database query. This is easier than ever with the GridView control. You set up a data source object, provide the connection string and query text, and assign the ID of the data source to the DataSourceId property of the GridView. At run time, the GridView automatically binds to the source and generates appropriate columns of data. By default, all the columns in the query are displayed in the grid.

Like the DataGrid control, the GridView supports custom column fields through the Columns collection. If you want to display only a subset of the retrieved data fields, or if you just want to customize their appearance, you can populate the Columns collection with objects that represent columns of data to display. The GridView supports a variety of column types, including the new checkbox and image column types:

<columns>
  <asp:boundfield datafield="productname" headertext="Product" />
  <asp:checkboxfield datafield="discontinued"
     headertext="Discontinued" />
  <asp:buttonfield buttontype="Button" text="Buy" />
  <asp:hyperlinkfield text="More Info..."
     datanavigateurlfields="productid,discontinued"
     datanavigateurlformatstring="more.aspx?id={0}&disc={1}" />
</columns>
Figure 9 shows a grid in action configured to use the fields listed in this code. The names of the GridView column classes differ slightly from the names of corresponding classes in the DataGrid interface. Basically, the suffix "column" is replaced with the suffix "field." Aside from this name change, the behavior of matching column classes is nearly identical. Some of the new column types save you from continually using templates. For example, the CheckBoxField column renders the specified data field using a checkbox, while the enhanced HyperLinkField column provides a long-awaited feature—support for multiple URL parameters. As in the code snippet just shown, the DataNavigateUrlFields property accepts a comma-separated list of field names and merges them with the text of the DataNavigateUrlFormatString property.

Figure 9 GridView with Fields in Action
Figure 9 GridView with Fields in Action

Note the difference between ButtonField and CommandField. Both columns add a button to the user interface of the grid, but a CommandField displays command buttons to perform select, edit, insert, or delete operations. A ButtonField simply represents a field that is displayed as a button. Finally, the GridView can embed images through the ImageField column type.

<asp:imagefield datafield="photo" headertext="Picture" />
Figure 10 shows the ImageField column in action on the photo field of the Northwind Employees table. It is interesting to note that ImageField uses the ASP.NET 2.0 DynamicImage control to display images from both databases and URLs. Furthermore, when in edit mode the ImageField column pops up a Browse button so you can locate a new file on the local machine to upload.

Figure 10 Image Field Column
Figure 10 Image Field Column

Template columns are supported as well and require syntax similar to that of a DataGrid in ASP.NET 1.x:

<asp:templatefield headertext="Product">
   <itemtemplate>
     <b><%# Eval("productname")%></b> <br />
     available in <%# Eval("quantityperunit")%>
  </itemtemplate>
</asp:templatefield>
It is interesting to note the more compact syntax for data-bound expressions that is permitted in ASP.NET 2.0. To build a templated content in ASP.NET 1.x, you had to use the following expression:
DataBinder.Eval(Container.DataItem, "fieldname")
Thanks to a smarter data-binding mechanism, you can now avoid using the static Eval method on the DataBinder class. Instead, you now call a new protected method named Eval that's defined on the Page class. You pass Eval the name of the field to evaluate and the method, in turn, determines the current data item and prepares a regular call through DataBinder.Eval.

Eval is declared as a protected method on the TemplateControl class from which both Page and UserControl derive. The actual class that represents an .aspx page in action is an instance of a class that derives from Page; hence, it can call into protected methods. The same holds true for ASCX user controls.

As long as the focus is on plain data display, it's hard to imagine the need for a brand new grid control like the GridView. Sure, you can now bind data source controls to the GridView with little or no code, but does this necessitate a replacement for the DataGrid? If it doesn't, consider sorting and paging.

In the GridView control, autoreverse sorting and paging are enabled simply by turning on the AllowPaging and AllowSorting properties. If you have ever tried this in ASP.NET 1.x, you're beginning to get the picture.

Figure 11 Pageable Sortable Grid in Action
Figure 11 Pageable Sortable Grid in Action

Figure 11 shows a pageable and sortable grid. The complete code for this grid is shown in Figure 12. (It is worth noting that the C# code is needed only if you want to add the glyph to the column header to indicate the sort direction.) So, sorting and paging work quite well without writing any code of your own. You control the data retrieval mode of a SQLDataSource through the DataSourceMode property. Feasible value types are DataSet (default) and DataReader. Data source controls may optionally cache the results of the SELECT command all the time when the DataSourceMode is DataSet. This enables rich GridView scenarios in which the control can offer codeless sorting, filtering, and paging capabilities. Caching is disabled by default and therefore must be enabled on the data source control.

Figure 12 Pageable and Sortable Grid (Close Figure 12)
<%@ page language="C#" theme="SmokeAndGlass" %>

<script runat="server">
    void AddGlyph(GridView grid, GridViewRow item)
    {
        Label glyph = new Label ();
        glyph.EnableTheming = false;
        glyph.Font.Name = "webdings";
        glyph.Font.Size = FontUnit.XSmall;
        glyph.Text = (grid.SortDirection==SortDirection.Ascending ?"
                      5" :" 6");

        // Find the column you sorted by
        for(int i=0; i<grid.Columns.Count; i++) {
            string colExpr = grid.Columns[i].SortExpression;
            if (colExpr != "" && colExpr == grid.SortExpression)
            {
                item.Cells[i].Controls.Add (glyph);
            }
        }
    }
    void MyGridView_RowCreated (object sender, GridViewRowEventArgs e)
    {
        if (e.Row.RowType == DataControlRowType.Header)
            AddGlyph (MyGridView, e.Row);
    }
</script>

<html>
<head runat="server">
    <title>GridView</title>
</head>
<body>
    <form runat="server">
        <asp:sqldatasource runat="server" id="MySource"
            connectionstring="SERVER=(local);DATABASE=northwind;Integrated
                             Security=SSPI;"
            datasourcemode="DataSet"
            selectcommand="SELECT * FROM products">
        </asp:sqldatasource>

        <asp:gridview runat="server" id="MyGridView"
          datasourceid="MySource"
          allowpaging="true" autogeneratecolumns="false"
          allowsorting="true" onrowcreated="MyGridView_RowCreated" >
            <pagersettings firstpagetext="7" lastpagetext="8"
               nextpagetext="4" prevpagetext="3" mode=
               "NextPreviousFirstLast" />
            <pagerstyle font-name="webdings" />
            <columns>
                <asp:boundfield datafield="productname"
                    headertext="Product" sortexpression="productname" />
                <asp:boundfield datafield="quantityperunit"
                    headertext="Packaging" />
            </columns>
    </asp:gridview>
    </form>
</body>
</html>

Caching data in memory can dramatically increase performance, but remember that the data is then somewhat vulnerable. You'll have to decide if the trade-off is worth it because the Cache object automatically discards least-used data if the system runs low on memory. In addition, in ASP.NET 2.0 the SQLDataSource control may optionally establish an automatic dependency with the database so that any change is promptly detected. This ensures that fresh data is always displayed. For more information on the features of data source controls, see my June 2004 article that I mentioned earlier. When the SQLDataSource control retrieval mode is DataReader, data is retrieved using an IDataReader object, which is a forward-only, read-only firehose-style cursor.

Editing Data

One of the most important shortcomings of the DataGrid—and conversely one of the major strengths of the GridView control—is the ability to of handle updates to the data source. When the bound data source supports updates, the GridView can automatically perform data operations, thus providing a real solution right out of the box. The data source control reveals these abilities through a number of Boolean properties such as CanUpdate, CanDelete, CanSort, and so forth.

For a GridView control, editing data means in-place editing and record deletion. As mentioned, in-place editing refers to the grid's ability to support changes to the currently displayed records. To enable in-place editing on a GridView, you turn on the AutoGenerateEditButton Boolean property:

<asp:gridview runat="server" id="MyGridView"
   datasourceid="MySource"
   autogenerateeditbutton="true">
   •••
</asp:gridview>

When the AutoGenerateEditButton property is set to true, the GridView displays an additional column that looks like the leftmost column in Figure 13. Clicking the Edit button for a row puts that row in edit mode. When a row is in edit mode, each bound field in the row that is not read-only displays the appropriate input control, typically a TextBox. When you click to update, the GridView raises the RowUpdating event and checks the CanUpdate property on the data source. If CanUpdate returns false, an exception is thrown. Otherwise, a command object is created and configured after the UpdateCommand property of the data source object.

Figure 13 Edit Column in GridView
Figure 13 Edit Column in GridView

Your involvement with SQL is limited to defining the structure of the command—you just define statements and the control does the rest. There's no need to work with ADO.NET or worry about commands or connections. To persist changes when the user clicks Update, write code like this:

<asp:sqldatasource runat="server" id="MySource"
   connectionstring="SERVER=...;DATABASE=northwind;Integrated
   Security=SSPI;"
   updatecommand="UPDATE employees SET
           firstname=@firstname, lastname=@lastname
           WHERE employeeid=@employeeid">
</asp:sqldatasource>
<asp:gridview runat="server" id="MyGridView"
   DataSourceId="MySource"
   DataKeyNames="employeeid" AutoGenerateEditButton="true">
   •••
</asp:gridview>

The UpdateCommand attribute of the data source is set to the SQL command that the GridView will use. You can use as many parameters as needed. If you take the time to use a particular naming convention, parameter values are automatically resolved. Parameters that represent fields to update, such as firstname, must match the name of DataField property of a grid column. The parameter used in the WHERE clause to identify the working record must match the DataKeyNames property, which is the key field for the displayed records. Finally, consider that if UpdateCommand isn't defined, CanUpdate returns false and an exception is thrown if you try to submit changes. The completion of an update command is signaled by the RowUpdated event. The number of rows updated by the execution of the update command can be retrieved in the AffectedRows property of the RowUpdated event arguments.

The GridView automatically collects values from the input fields and populates a dictionary of name/value pairs that indicate the new values for each field of the row. The GridView also exposes a RowUpdating event that allows you to modify the values being passed to the data source object. In addition, the GridView automatically calls Page.IsValid before invoking the Update operation on the associated data source. If Page.IsValid returns false, the operation is canceled. This is especially useful if you're using a custom edit template that includes validators.

A similar pattern occurs for the deletion of a row. The following SQL command is valid content for the DeleteCommand property of a data source object:

DELETE employees WHERE employeeid=@employeeid
Note that the delete operation fails if the record can't be deleted due to database-specific constraints. For example, the record can't be deleted if child records refer to it through a relationship. In this case, an exception is thrown.

The GridView control doesn't have automatic support for inserting data into a data source. The missing feature is entirely due to the GridView implementation and doesn't depend on the capabilities and characteristics of the underlying data source. The data source object, in fact, provides a CanInsert property and supports an InsertCommand property. Note that a combination of GridView and DetailsView controls enables you to achieve this functionality, as you'll see in a moment.

The DetailsView Control

Many applications need to work on a single record at a time. In ASP.NET 1.x, there is no built-in support for this scenario. Creating a single record view is possible but requires some coding effort on your part. You have to fetch the record first, then bind its fields to a data-bound form, and optionally provide paging buttons to navigate between records. I devoted three installments of the Cutting Edge column to this problem—April, May, and June 2002.

The need to display the contents of a single record is fairly common when you build master/detail views. Typically, the user selects a master record from a grid and the application drills down to show all the available fields. By combining GridView and DetailsView, you build hierarchical views with very little code.

The DetailsView control can automatically bind to any data source control and take advantage of its set of data operations. The control can automatically page, update, insert, and delete data items in the underlying data source as long as the data source supports these operations. In most cases, no code is required to set up any of these operations, as shown here:

<asp:detailsview runat="server" id="det"
   datasourceid="MySource"
   autogenerateeditbutton="true"
   autogenerateinsertbutton="true"
   autogeneratedeletebutton="true"
   allowpaging="true"
   headertext="Employees">
   <pagersettings mode="NextPreviousFirstLast"
      firstpageimageurl="images/first.gif"
      lastpageimageurl="images/last.gif"
      nextpageimageurl="images/next.gif"
      previouspageimageurl="images/prev.gif" />
</asp:detailsview>
The user interface of the DetailsView control can be customized using data fields and styles in a way that is similar to the GridView. The DetailsView doesn't support custom templates as this specific capability has been entirely factored into the new FormView control. The DetailsView can have a command bar with any combination of Edit, Delete, and New buttons. When you click Edit or New, the control renders in Edit or Insert mode and the contents of fields are displayed in textboxes. The working mode can be controlled through the Mode and DefaultMode properties.

The DetailsView control lends itself very well to implementing no-code master/details scenarios. Along with Edit and Delete buttons, the GridView control supports the Select button, which is also predefined. You enable this button on a per-row basis by setting the AutoGenerateSelectButton property to true. When the users click on this button, the current row enters the selected state and its zero-based index is assigned to the SelectedIndex property of the GridView. In addition, the GridView control raises the SelectedIndexChanged event. Applications can hook up to this event and run custom code.

In ASP.NET 2.0 there's no need to handle the SelectedIndexChanged event if you want to build a master/detail view. You can drop a GridView and a DetailsView control on the page and bind each to a data source. The trick for codeless master/detail is to bind the detail control to a data source represented by the currently selected record, as shown in the following:

<asp:sqldatasource runat="server" id="MyDetailSource"
   •••
   selectcommand="SELECT * FROM customers"
   filterexpression="customerid='@customerid'">
   <filterparameters>
     <asp:ControlParameter Name="customerid"
        ControlId="masterGrid"
        PropertyName="SelectedValue" />
   </filterparameters>
</asp:sqldatasource>

The FilterExpression property of a data source object defines the WHERE clause for the base query specified by SelectCommand. Parameter values can be specified in a variety of ways, including through direct binding with a control property. The <ControlParameter> object sets the @customerid parameter to the value stored in the SelectedValue property of the master grid control. The code in Figure 14 shows the configuration of the master grid and the detail view controls. Figure 15 shows the page in action. Note that no program code was required to achieve this.

Figure 14 Configuration of the Master Grid (Close Figure 14)
<%@ page language="C#" theme="SmokeAndGlass" %>

<html>
<head runat="server">
    <title>Master-Detail View</title>
</head>
<body>
    <form runat="server">
        <asp:sqldatasource runat="server" id="MySource"
          connectionstring="SERVER=(local);DATABASE=northwind;Integrated
                           Security=SSPI;"
            selectcommand="SELECT customerid, companyname, country
                           FROM customers" />
        <asp:sqldatasource runat="server" id="MyDetailSource"
          connectionstring="SERVER=(local);DATABASE=northwind;Integrated
                           Security=SSPI;"
            selectcommand="SELECT * FROM customers"
            filterexpression="customerid='@customerid'">
            <filterparameters>
                <asp:ControlParameter Name="customerid"
                    ControlId="Master" PropertyName="SelectedValue" />
            </filterparameters>
        </asp:sqldatasource>

        <table><tr><td>
        <asp:gridview runat="server" id="Master" width="100%"
            datasourceid="MySource"
            pagesize="10"
            selectedindex="0"
            allowpaging="true"
            datakeynames="customerid"
            autogenerateselectbutton="true"
            autogeneratecolumns="false">
            <columns>
                <asp:boundfield datafield="companyname"
                    headertext="Customer" />
                <asp:boundfield datafield="country"
                    headertext="Country" />
            </columns>
        </asp:gridview>
        </td></tr>
        <tr><td>
        <asp:detailsview runat="server" id="Detail" width="100%"
            datasourceid="MyDetailSource">
        </asp:detailsview>
        </td></tr></table>

    </form>
</body>
</html>
Figure 15 Master Grid in Action
Figure 15 Master Grid in Action

The FormView Control

FormView is a new data-bound control that works like the templated version of the DetailsView. It renders one record at a time picked from the associated data source and optionally provides paging buttons to navigate between records. Unlike the DetailsView control, FormView doesn't use data control fields but allows the user to define the rendering of each item using templates. FormView supports any basic operation its data source provides.

Designed to be used mostly as an update and insert interface, the FormView control is unable to validate against a data source schema and doesn't supply advanced editing features like foreign key field dropdowns. However, by using templates you can easily provide this functionality. Two functional aspects mark the difference between FormView and DetailsView. First, the FormView control has ItemTemplate, EditItemTemplate, and InsertItemTemplate properties that the DetailsView lacks entirely. Second, the FormView lacks the command row, a toolbar on which available functions are grouped. Unlike the GridView and DetailsView controls, the FormView has no default rendering of its own. At the same time, its graphical layout is completely customizable using templates. Therefore, each template will include all command buttons needed by the particular record. The following code snippet shows the typical code you write to embed a FormView in your pages.

<asp:FormView ID="EmpDetails" runat="server"
    DataSourceId="MySource" AllowPaging="true">
    <ItemTemplate>
      •••
    </ItemTemplate>
    <EditItemTemplate>
      •••
    </EditItemTemplate>
    <InsertItemTemplate>
      •••
    </InsertItemTemplate>
</asp:FormView>

Figure 16 illustrates a page that uses a FormView control. The Edit button is added using an <asp:Button> element with the Edit command name. This will cause the FormView to switch from read-only to edit mode and display using the EditItemTemplate, if any is defined. A command name of New will force the control to change to its insert mode and render the contents defined for the InsertItemTemplate. Finally, if you add a button with the Delete command name to the item template, then the FormView invokes the Delete command on the data source when the user clicks it.

Figure 16 FormView Control
Figure 16 FormView Control

How do you retrieve values to update or insert a record? You use a new data binding keyword—Bind—specifically designed for two-way binding:

<asp:TextBox Runat="server" ID="TheNotes"
     Text='<%# Bind("notes") %>' />
The Bind keyword works like Eval to display data and can retrieve the value you've entered when updating or inserting a record. In addition, Bind is very useful in TemplateFields used in the GridView and DetailsView.

Bind stores the value of the bound control property into a collection of values that the FormView control automatically retrieves and uses to compose the parameter list of the insert or edit command. The argument passed to Bind must match the name of a field in the data container. For example, the textbox in the last code snippet provides the value for the notes field. Finally, bear in mind that the edit and insert templates must contain buttons to save changes. These are ordinary buttons named Update and Insert to save and Cancel to abort.

FormView's events work the same way that DetailsView's and GridView's do. So if you need to do more sophisticated things like pre- or post-processing data (for example, filling dropdowns) then you should write appropriate event handlers for ItemCommand, ItemInserting, ModeChanging, and the like.

Conclusion

Data-bound controls are an essential part of most, if not all, Web apps. Data-bound controls should be simple and powerful. Ideally, they should provide advanced features in a few clicks and use a limited amount of code. Although ASP.NET 2.0 is still in the works, its new generation of data-bound controls meet this requirement. The key shortcoming of ASP.NET 1.x data binding was that it required too much code for common data operations. This has been resolved with the introduction of data source objects and the GridView control. The DetailsView and the FormView are the perfect complement to the GridView and represent an important improvement to the ASP.NET 1.x data toolbox.

Dino Esposito is an instructor and consultant based in Italy. Author of Programming ASP.NET and Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ADO.NET and ASP.NET and speaking at conferences. Reach Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


Genetic Algorithms: Survival of the Fittest: Natural Selection with Windows Forms

Genetic Algorithms

Survival of the Fittest: Natural Selection with Windows Forms

Brian Connolly
This article discusses:
  • Genetic programming definition
  • Breeding new algorithm generations
  • Cross breeding
  • Mutations
  • Increasing fitness
This article uses the following technologies:
Windows, C#, and CodeDOM
Code download available at: GeneticAlgorithms.exe (190KB)
Contents
The Sample Problem
The Application UI
High-Level Class Design
Genetic Programming
Ant Class
TerminationReport Class
Function Class
ExpressionTree Class
Generator Class—Constructor
Generator Class—Replicate and Breed Children
Generator Class—Generating New Classes
Generator Class—Compiling New Code and Executing It
Building and Running the Application
Working with a Complex Problem
Further Work and Extensions

G
enetic programming (GP) is one of the most useful, general-purpose problem solving techniques available to developers. It has been used to solve a wide range of problems, such as symbolic regression, data mining, optimization, and emergent behavior in biological communities.

GP is one instance of the class of techniques called evolutionary algorithms, which are based on insights from the study of natural selection and evolution. Living things are extraordinarily complex, far more so than even the most advanced systems designed by humans. Evolutionary algorithms solve problems not by explicit design and analysis, but by a process akin to natural selection.

An evolutionary algorithm solves a problem by first generating a large number of random problem solvers (programs). Each problem solver is executed and rated according to a fitness metric defined by the developer. In the same way that evolution in nature results from natural selection, an evolutionary algorithm selects the best problem solvers in each generation and breeds them.

Genetic programming and genetic algorithms are two different evolutionary algorithms. Genetic algorithms involve encoded strings that represent particular problem solutions. These encoded strings are run through a simulator and the best strings are mixed to form a new generation. Genetic programming, the subject of this article, follows a different approach. Instead of encoding a representation of a solution, GP breeds executable computer programs.

In this article I'll take a simple GP problem from the seminal textbook on the subject: John Koza's Genetic Programming: On the Programming of Computers by Means of Natural Selection (MIT Press, 1992). The problem is to develop an artificial ant that will efficiently walk a grid that has a weaving trail of food on it. The objective is to breed the ant that will gather the maximum amount of food in the minimum number of steps.

I'll develop a UI that displays the problem grid and allows the user to control GP execution. The UI can be used to study the successive generations of ants. Any ant from any generation can be selected for study. Its generated C# code can be viewed, and the trail that it walked can be displayed on the grid.

The internal process works off of a base class that represents the problem. In this case, I developed a base ant class that implements the fundamental operations and state of an ant. The GP algorithm is used in order to breed new subclasses of this problem class. The algorithm uses the code document object model (CodeDOM) to generate a subclass and an Execute method implementation that represents a possible strategy for an ant.

This implementation is generic in the sense that it is driven entirely by a master "problem" class (the base Ant class). A developer can specify a new GP problem class that is modeled after the Ant class. For example, this might be a class representing the allowable functions for a symbolic regression problem. Since the internal GP processing doesn't have any special-purpose ant logic, the same architecture could be used to solve that problem as well.

I developed this implementation to demonstrate that the Microsoft® .NET Framework provides all of the tools that you need to do genetic programming. While this implementation meets all of the essential requirements to qualify as genetic programming, in its present state there is plenty of room for the solution to be augmented and improved. For example, complex selection, mutation, and pruning techniques could be used to explore the solution space far more thoroughly than does this simple example. But since they are essentially just mathematical refinements of the basic algorithm, this article demonstrates that .NET-based GP implementations can reach that level of sophistication.

In this article, I will describe the sample GP problem we'll be working with and will walk through the class relationships and the key implementation code. I'll also give you an overview of genetic programming and show you how ants can be evolved to walk a more complex trail, as well as how you can make changes to the problem class to make new operations available.

The Sample Problem

The abstract problem I'll deal with is to develop an intelligent strategy to be used by a simple artificial ant. The ant moves on a 32¥32 grid, and if it moves off one side it returns on the other side. The grid has a winding trail of food on it. The trail begins at the [0,0] origin, which is also the starting position of the ant. Each grid location in the trail can be empty, contain food, or contain an indicator that it is a gap between food locations. The trail must be continuous from the origin. All food must be adjacent to at least one point that has food or a marked gap.

The simple ant has two elements that fully describe its state. The first is its current location on the grid. The second is the direction it is facing, which is either left, right, up, or down. The ant has three possible things it can do. It can turn left, turn right, or move to the square that it is facing. It has only one sensor for its environment: a function called FacingFood which will tell it if there is food in the square that it is facing.

When an ant moves to a square that contains food, it removes the food from the grid and increments the total food it has gathered. When it moves or turns, it increments the number of steps it has taken to gather its food.

The problem is to create an ant that will make an intelligent series of moves and turns based on its FacingFood function. An example would be:

if (FacingFood()) {
    Move();
}
else {
    TurnRight();
    Move();
}
This would work fine for trails whose turns are always to the right. The practical problem is to use GP to create a program that will walk an arbitrary trail of food and gaps most efficiently.

The Application UI

Figure 1 shows the UI of the .NET-based implementation of the artificial ant. The grid panel is divided into a 32¥32 matrix of squares that contain the trail of food to be retrieved by the ant algorithm. A default trail is loaded at startup. You can use the "Empty Grid" button to remove the trail. The Food and Gap radio buttons allow the user to paint a new trail by clicking in the grid. The "SaveGrid" button outputs the current grid trail to an XML file, which can later be loaded back in with the "Restore Grid" button.

Figure 1 Artificial Ant Application Window (Close Figure 1)

The boxes on the left control GP execution and include the following:

Generations The algorithm will stop after this number of generations is produced. However, if a perfect ant (an ant that gathers the maximum amount of food in the minimum number of steps) is discovered, the algorithm will immediately halt execution.

Population This is the number of ants each generation will have. The population will be a mixture of the most efficient ants from the previous generation, together with their children.

Max Tree Depth The algorithm begins by generating random ants. This setting controls how deep their expression trees can be.

Food Goal and Step Goal We'd like to breed an ant that will gather this much food in these few steps. By default, these will reflect the food and gap counts in the active grid. However, the computed Step Goal doesn't reflect the turn operations that an ant will need to make when the trail changes direction.

In order to demonstrate the UI and application, let's work through a simple example. Start the application, which should look like Figure 1. There are gap squares that break the connection between the smaller food square to the upper left and the larger food square next to it.

The two squares have a total of 33 food cells and, since there are two gaps, the minimum possible number of steps required to obtain all the food is 35. Recall that the minimum step total doesn't reflect turn operations that are necessary at corners.

Hit the Generate button with the default textbox parameters to start the genetic programming process. This will generate an initial population of 100 ants, each of which will follow a randomly generated strategy. Here, Max Tree Depth is set to 5, so the initial population of ants will have at most five levels of conditional branching, moves, and steps.

The listbox beneath the Generate button shows summary data about each generation. This is the average "fitness" of the generation, and the fitness of the best and worst ants. I'll define fitness later; at this point all that's necessary is that you realize that higher fitness is better.

When the tenth generation is completed, select the generation summary from Generation 1, and all of the ants in that generation appear in the listbox beneath it. Each ant represents a randomly generated class. The name of each ant is shown with the amount of food it gathered and the number of steps it took. The name of the ant class is GenX_AntY, where X is the generation number and Y is the ant number within that generation. The best ant from a generation always appears at the top of the listbox. Select that top ant from Generation 1.

When an ant is selected, detailed information about it is displayed in the textbox beneath it. This will show the parents of the ant—that is, the name of the classes from which it was generated. Below the list of the parents, the generated code for the selected ant will be listed. I'll describe the code in more detail later.

Figure 2 Poor Performance
Figure 2 Poor Performance

When an ant is selected, two buttons are activated. Replay Step allows the user to step slowly through the trail that the ant made, one move at a time. Replay All automatically walks the entire trail. Figure 2 shows the UI display when Replay All is clicked for the best ant in Generation 1. The generated code, the lineage information about ants, and the display of the trail they took provide excellent tools for studying a genetic algorithm. The best ant in Generation 1 gathered only 18 items of food, taking the maximum allowable 400 steps. Its trail display in Figure 2 shows why it is so poor. Note that since the logic for each ant is generated with some degree of randomness, your results may vary slightly.

Figure 3 Ant Trail in 45 Steps
Figure 3 Ant Trail in 45 Steps

Next, scroll down the Generations listbox to select Generation 10. Select its best ant, the one at the top of the ant listbox. This ant evolved by selecting and combining the most efficient food gatherers from prior generations. This ant gathered 33 items of food in only 45 steps. Its trail is displayed in Figure 3. Clearly it is following a strategy that is far more efficient for this trail.

High-Level Class Design

A high-level view of the class design is shown in Figure 4. Several of the classes I use, such as the Cell, AntTrailGrid, TerminationReport, IComparable, mFoodGathered, and Ant, are problem specific. If you wanted to use the kernel of this implementation for another problem space, you would need to develop different versions of these classes. However, the Generator, Function, and ExpressionTree classes that together form the core of the GP algorithm can remain essentially unchanged.

Figure 4 Class Design
Figure 4 Class Design

The Cell class represents a grid cell. The AntTrailGrid class is used to perform graphics operations on the display grid panel. I won't describe these classes in detail as they pertain more to the user interface than to GP.

The Ant class and its associated TerminatingReport class represent the problem I'm trying to solve. An Ant is initialized by passing a grid, a food goal, and a limit on the number of steps it will take. In my implementation I limit every ant to 400 steps. The Ant has public methods to turn and move, and private member variables to record the steps taken and food gathered as it moves through its grid. The Ant also maintains a history of where it's been on the grid in the private mTrail ArrayList. It also has a single, Boolean function: FacingFood.

Whenever an ant takes a step, it checks to see if it has reached its food goal or if it has taken the maximum number of steps allowed. If one of these is true, it saves its history in the member variable mConcludingReport, a TerminationReport. The TerminationReport records the food and step counts, and also saves the trail of the ant. The Fitness method of TerminationReport implements the fitness measure of an ant by returning an integer score that accounts for the food gathered and the steps taken. Ants that gather more food in fewer steps are more fit than other Ants that perform worse. TerminationReport also implements the CompareTo method of the IComparable interface. The CompareTo method compares TerminationReports according to fitness. When a generation completes, the Generator class will have hundreds or even thousands of TerminationReport instances in an ArrayList, and the IComparable implementation allows the ArrayList to be sorted by fitness.

Generator is the kernel of the GP implementation. It performs selection, breeding, code generation, execution, and result sorting. A generator is constructed by passing a BaseClass type (the Ant class, in this case), and a maximum expression tree depth limit. It uses reflection on the BaseClass type to find all of its public methods that have no parameters with the exception of the special NotFinished method. All of the void methods will be the primitive operations available for the GP algorithm to employ. Any Boolean methods with no parameters are used to control the execution of the primitive operators. This information is stored in Function objects. When NewGeneration is called requesting an initial population of N ants, it builds N random ExpressionTree objects. Generator uses these ExpressionTrees and CodeDOM classes to construct N subclasses of Ant, each of which has a single method called Execute. The Execute method has a loop of the expression tree statements that are executed as long as the NotFinished method returns true.

Generator places all of these classes in a single namespace, compiles them, and loads this new assembly. It then creates a single instance of each generated class, invokes its Execute method, and saves the TerminationReport. In a more robust implementation, each assembly would be isolated in its own AppDomain so that it could be unloaded when no longer needed. For demo purposes, I've chosen to keep things simple and to load all assemblies into the main AppDomain.

TerminationReport has a single, static method called GoalException. Generator uses this to create a special instance of the TerminationReport. This instance will have the same fitness as a perfect ant—one that gathered all available food in the minimum number of steps. Generator will continue with successive generations, performing selection and crossover breeding until it reaches the generation limit or it evolves a perfect ant.

Generator is a general-purpose implementation of GP. I stated earlier that you could define a different problem as a different "ant." Perhaps this ant would expose a set of primitive mathematical functions such as addition, multiplication, and exponentiation. Its grid would be a set of (x,y) input and output values from a function for which you would need to develop a mathematical expression. You could then essentially use Generator without change to solve this problem as well.

To be truly generic, the design needs to be changed, but only slightly. Generator uses TerminationReport directly to construct the goal exception of the perfect ant, and it also has some knowledge of the Init method that is used to pass the Grid to the ants that it instantiates. In the detailed discussion of Generator that follows, I'll describe what is needed to make it truly generic instead.

Genetic Programming

In his textbook, John Koza presented a large number of sample problems that can be solved with genetic programming. He developed a systematic way of relating particular problems to a canonical set of design decisions that a developer uses to model a problem.

One step is the selection of what he calls terminals and functions. The ant will be controlled by behavior expressions, which are tree-like structures of terminals and functions. Terminals are primitive operations or literals that cannot be decomposed further. Functions are higher-level operations that use other functions or terminals. In our case, the artificial ant has terminals Left, Right, and Move. One member of the function set is the FacingFood Boolean, which takes two terminal or function arguments. The statement "if FacingFood, then Left else Right" is an example of an expression that has the function FacingFood and two children: the terminals Left and Right. I'll also generate two other functions. The first, which I'll call P2, has two clauses which are themselves either terminals or functions. An example of this is the expression "Right;Move". A second function that I'll add is P3. This is just like P2, except that it has three clauses. An example of P3 is the expression "Move; Right; if FacingFood then Right else Move".

The GP algorithm begins by generating a population of random expressions. One parameter that controls the algorithm is population size; this is one of the parameters that the user can set in the UI. These expressions are then executed and evaluated, using a fitness measure. Designing this fitness measure is another step in the canonical GP design fitness criteria. I want to select efficient ants, those that gather a lot of food with few steps, so the fitness measure weighs both food gathered and the number of steps used. In my implementation, I subtract the steps from the food gathered and add 1000, so that fitness is always positive. Hence an ant with a higher fitness measure is going to be more successful than an ant with a lower measure.

When the initial, random first generation is completed, the real work of GP begins. The first step is selection, which is choosing the best ants and copying them to the next generation. Most GP implementations use one of a number of sophisticated selection techniques, but for the sake of simplicity this implementation copies the best 10 percent, as measured by our fitness criteria. These selected ants will be the "parents" of all the remaining ants in the subsequent generation.

Subsequent generations begin with the pool of parents that were replicated from the prior generation. Since this is only 10 percent of the available population, I need to fill the remaining slots by selecting parents and breeding them to create the next generation. Pairs of parents are selected, and each pair of parents yields a pair of children, each of which combines elements of both parents. Parents are selected according to fitness. In general, fitter parents are more likely to be selected to breed.

The crossover algorithm (see Figure 5) is used to breed a pair of children from a pair of parents. A random crossover point is chosen in each parent, and the subtree beneath these crossover points is swapped between the parents, creating the two offspring.

Figure 5 Crossover (Breeding) Operation
Figure 5 Crossover (Breeding) Operation

Most GP algorithms also implement mutation. This involves making a random change in a small number of offspring in order to maintain diversity in the population. Mutation is essential because otherwise the population becomes completely dominated by a few good bloodlines, which makes it difficult to continue progress beyond a certain point. This implementation performs a simple mutation step during the crossover operation to introduce randomness. When a terminal node is copied, a random terminal is substituted two percent of the time.

Ant Class

An Ant is initialized by passing it a reference to a grid, integers that specify the food goal, and the maximum number of steps:

public Void Init(Cell[,] grid, int StepLimit, int FoodGoal)
The ant places itself at the [0,0] origin and faces right. When an ant walks the grid, it removes any food it gathers. For this reason it makes its own copy of the grid.

The turn and move methods are simple. For example, the Move method moves to the facing grid location, increments the number of steps, and checks to see if it should terminate, as you can see if you take a look at Figure 6.

Figure 6 Move Method (Close Figure 6)
public void Move()
{
    if(NotFinished())
    {
        GetFacingCellCoord(outxLocation, outyLocation);
        mTrail.Add(newTrailSpace(xLocation, yLocation));
        mTotalSteps++;
        if(mGrid[xLocation,yLocation].HasFood())
        {
            //gather the food and remove it from the grid
            mTotalFood++;
            mGrid[xLocation,yLocation].SetEmpty();
        }
}

The NotFinished method checks for completion and saves a concluding TerminationReport:

public bool NotFinished()
{
    if (!mDone)
    {
        if (mTotalSteps==mStepLimit | mTotalFood==mFoodGoal)
        {
            mConcludingReport= new TerminationReport(
                mTotalSteps,mTotalFood,mTrail);
            mDone=true;
        }
    }
    return !mDone;

TerminationReport Class

TerminationReport is used to save the history of ant executions and to compare the relative fitness of ants. It exposes a Fitness method that returns a positive measure of an ant's fitness:

public static int Fitness(TerminationReport RunResult)
{
    // more food and/or fewer steps is better.
    // we add 1000 so that it's always positive
    return RunResult.mFoodGathered -
        RunResult.mNumberOfSteps +
        1000;
}
The IComparable interface CompareTo method uses this fitness value to determine the relative order of ants. This allows Collections of TerminationReports to be sorted by fitness:
public int CompareTo(object obj)
{
    if(obj is TerminationReport)
    {
        TerminationReport temp = (TerminationReport)obj;
        return Fitness(this).CompareTo(Fitness(temp));
    }

    throw new ArgumentException(
        "object is not a TerminationReport");
}

Function Class

The Function class is a private class that a Generator uses to represent the available functions from the base problem class. There are two types of functions:

  • If statements represent the Boolean-valued methods from the base class. In the ant application, this represents the FacingFood method.
  • Child count functions hold slots for a defined number of child functions or terminals. Examples of these would be the P2 and P3 internal tree nodes in Figure 5.

Function has two constructors corresponding to these two types:

public Function(MethodInfo methodInfo)
{
    mFunctionType = FunctionType.IfStatement;
    mMethodInfo = methodInfo;
    mChildCount = 2;
}
public Function(int childCount)
{
    mFunctionType = FunctionType.ChildCount;
    mChildCount = childCount;
}
MethodInfo is the reflection information on a method that the Generator gets from the base problem class. The code assumes that any Boolean-valued function can be used as the condition in an If statement (except for the special NotFinished function).

The set of Function objects created by the Generator remain unchanged during GP processing. These objects control the types of manipulation it can perform on ExpressionTrees which are the dynamic objects that represent potential solutions to the problem.

ExpressionTree Class

The program will generate ants that repeatedly perform a set of moves and turns, subject to checks of the FacingFood condition. These operations are represented internally by an ExpressionTree for each ant. ExpressionTrees are used to generate C# code and are the subject of selection and reproduction.

ExpressionTree has a single constructor, shown in Figure 7. This builds a random tree from the set of Functions and an ArrayList of terminals. MaxDepth controls how deep the tree can grow—when it reaches that point, only terminals can be selected for the next level. ExpressionTree also exposes methods to create a new copy of a tree node and also to clone the entire tree. These are used to implement the crossover operation.

Figure 7 ExpressionTree Constructor (Close Figure 7)
// generate a random ExpressionTree from the set of
// terminals and nodes we found in mBaseType.  The
// tree will be at most MaxDepth levels deep
public ExpressionTree (int MaxDepth,
    Generator Generator, Object Node, Object Parent)
{
    mNode = Node;
    mParent = Parent;
    if(mNode.GetType() == typeof(Function))
    {
        Function mFunction = (Function)mNode;
        mChildren = new
            ExpressionTree[mFunction.ChildCount];
        for( int i = 0; i< mFunction.ChildCount; i++)
        {
            if (MaxDepth > 0)
            {
                mChildren[i] = new ExpressionTree (MaxDepth-1,Generator,
                    Generator.
                        GetRandomTerminalOrFunction(),
                        this);
            }
            // if we've reached MaxDepth, choose only
            // terminals
            else
            {
                mChildren[i] = new ExpressionTree(
                    MaxDepth-1,Generator,
                    Generator.GetRandomTerminal(),
                    this);
            }
        }
    }
}

Generator Class—Constructor

The Generator class is the kernel of the genetic programming application. It discovers available base class terminals and functions. It generates, compiles, and executes C# code to search for a good solution to the problem it is given. The constructor is passed a System.Type which is the root class for .NET reflection operations. In this case, the Type is Ant. The Generator constructor fills the list of terminals and functions, treating void methods as terminals and Boolean methods as Functions (see Figure 8). After it does this, the final for loop adds some additional functions. This application passes a MaxSteps value of 3, which makes the generator add two additional functions. The first will have two children; the second will have three children.

Figure 8 Generator Constructor (Close Figure 8)
public Generator(Type BaseClass, int MaxSteps)
{
    mBaseType = BaseClass;
    mTerminals = new ArrayList();
    mFunctions = new ArrayList();
    // first create the terminals
    BindingFlags flags = BindingFlags.Public |
        BindingFlags.Instance;
    MethodInfo[] mBaseClassMethods =
        BaseClass.GetMethods(flags);
    foreach(MethodInfo mi in mBaseClassMethods)
    {
        // Terminals are parameterless void methods, except
        // for "Execute"
        if(mi.GetParameters().Length == 0 &&
            mi.ReturnType == typeof(void) &&
            mi.Name != "Execute")
        {
            mTerminals.Add(mi);
        }
        else if(mi.GetParameters().Length == 0 &&
            mi.ReturnType == typeof(bool) &&
            mi.Name != "NotFinished")
        {
            mFunctions.Add(new Function(mi));
        }
    }
    for(int i = 2; i<= MaxSteps; i++)
    {
        mFunctions.Add(new Function(i));
    }
}

Generator Class—Replicate and Breed Children

When the user clicks on the Generate button, the form object calls a Generator method called NewGeneration, once for each generation. NewGeneration controls the evolution towards the problem solution. NewGeneration returns a result object that contains a summary of the generation, as well as all of the TerminationReport objects exposed by its ants. When the form object calls NewGeneration again, it passes back this result object.

NewGeneration begins by calling a private method called ReplicateAndBreedChildren. This method handles two cases. The first case is when it is working on the first generation. It creates new, random expression trees, assigns class names to the ants that will be generated from them, and creates new result objects that will hold their execution results when they've been executed:

// first generation - just generate random trees
for(int i = 0; i<= NumberOfPrograms-1;i++)
{
    ExpressionTree newTree = new ExpressionTree(
        MaxTreeDepth,this, GetRandomFunction(), null);
    string temp = NewAntName ( GenerationNumber, i);
    ThisGenerationResults.Add(
        new TypeExecutionResult(NewAntName(GenerationNumber, i),
        "","",newTree));
}

The other case is when there are results from the last generation. First, we need to select the best ants from the prior generation and copy them into this generation. GenerationResults is an ArrayList that contains a TerminationReport for each ant. The ArrayList.Sort method ultimately invokes the TerminatingReport.CompareTo method which compares ants by fitness. Since better ants have higher fitness measures, we reverse the sort to place the best ants in the lower slots (the sorting comparison operation could also be inverted to do this implicitly, which would be more efficient), and we keep 10 percent of them in the current generation:

int numToKeep = NumberOfPrograms/10;
int numKept = 0;
LastGenerationResults.Sort();
LastGenerationResults.Reverse();
IEnumerator enumLastGenerationResults =
    LastGenerationResults.GetEnumerator();
while(enumLastGenerationResults.MoveNext() && numKept++ < numToKeep)
{
    ThisGenerationResults.Add((TypeExecutionResult)
        enumLastGenerationResults.Current);
}

Next, we need to fill the remaining population slots with children. Pairs of parents are selected, and the Crossover method is used to create new children using the algorithm just described. Parents are selected based on their relative fitness. The probability of selecting a particular parent is the ratio of that parent's fitness, divided by the total fitness of all parents. Figure 9 shows the Generator.ReplicateAndBreedChildren code for this processing.

Figure 9 Generator Select-And-Breed (Close Figure 9)
// breed two children for each pair of parents
// until we've filled all the slots
for(int i = ThisGenerationResults.Count;
    i<= NumberOfPrograms-2; i=i+2)
{
    ExpressionTree child1, child2;
    ExpressionTree Parent1, Parent2;
    int Parent1Index, Parent2Index;

    SelectParentsAndCloneThem (ThisGenerationResults,numKept - 1,
        out Parent1Index, out Parent1,
        out Parent2Index, out Parent2);

    Crossover (Parent1, Parent2, out child1, out child2);

    TypeExecutionResult childResult1, childResult2;

    childResult1 = new TypeExecutionResult (
        NewAntName(GenerationNumber, i),
        ((TypeExecutionResult) ThisGenerationResults[Parent1Index])
            .mTypeClassName,
        ((TypeExecutionResult) ThisGenerationResults[Parent2Index])
            .mTypeClassName,
        child1
        );

    childResult2 = new TypeExecutionResult (
        NewAntName(GenerationNumber, i+1),
        ((TypeExecutionResult) ThisGenerationResults[Parent1Index])
        .mTypeClassName,
        ((TypeExecutionResult) ThisGenerationResults[Parent2Index])
        .mTypeClassName,
        child2
        );

    ThisGenerationResults.Add(childResult1);
    ThisGenerationResults.Add(childResult2);
}

Generator Class—Generating New Classes

Generating executable code is the essence of genetic programming because that is what distinguishes it from other evolutionary algorithms. This application uses CodeDOM to create a CodeCompileUnit containing a class for each ant in a generation, and it uses the Microsoft.CSharp.CSharpCodeProvider to create an ICodeGenerator and an ICodeCompiler.

First let's look at the type of class we want to create. An example of a particular generated ant is shown in Figure 10. A subclass of Ant is declared, and the name of the class includes the generation number and the ant number within that generation. The Ant subclass has an Execute method that overrides the abstract Execute method in the base Ant class. It contains a loop that continues as long as NotFinished returns true. The loop consists of a series of statements that invoke base class methods and functions, and it is this series of statements that will be created from the ExpressionTree corresponding to the Ant.

Figure 10 Sample Generated Ant (Close Figure 10)
public class Ant_Gen1_Ant33 : Ant {

    public override void Execute() {
        for (
        ; NotFinished();
        ) {
            if (FacingFood()) {
                Move();
            }
            else {
                TurnRight();
                if (FacingFood()) {
                    Move();
                    TurnLeft();
                }
                else {
                    Move();
                    TurnLeft();
                }
            }
        }
    }
}

The Generator has a public BuildClass method that generates this class, given an ExpressionTree. It is public because the main form uses this method to fill the contents of the code textbox when an ant is selected for examination. BuildClass creates the class declaration and adds the base type property. It calls BuildExecute to create the Execute method from the ExpressionTree:

publicCodeTypeDeclaration BuildClass(string ClassName,ExpressionTree tree)
{
    CodeTypeDeclarationmType =new CodeTypeDeclaration(className);

    //it inherits from mBaseType;
    mType.BaseTypes.Add(mBaseType.Name);
    mType.Members.Add(BuildExecute(Tree));
    return mType;
}

BuildExecute calls GenerateStatements to build the statements of the loop and passes to it the ExpressionTree. The code for BuildExecute is shown in Figure 11, and the code for GenerateStatements is shown in Figure 12.

Figure 12 GenerateStatements (Close Figure 12)
// generate the statements that express the terminals and
// functions of the Tree
private CodeStatement[] GenerateStatements
    (ExpressionTree Tree)
{
    CodeStatementCollection mCodeStatements =
        new CodeStatementCollection();
    Object mNode = Tree.Node;
    if ((System.Type)mNode.GetType() == typeof(Function))
    {
        Function mFunction = (Function)mNode;

        if (mFunction.FunctionType == FunctionType.IfStatement)
        {
            CodeSnippetExpression ifcondition =
                new CodeSnippetExpression(
                mFunction.MethodInfo.Name.ToString() + "()");
            CodeStatement[] cstrue =
                GenerateStatements(Tree.Children[0]);
            CodeStatement[] csfalse =
                GenerateStatements(Tree.Children[1]);
            CodeConditionStatement ifStatement =
                new CodeConditionStatement (ifcondition, cstrue,
                                            csfalse);
            mCodeStatements.Add(ifStatement);
        }
        else
        {
            for( int c = Tree.Children.GetLowerBound(0);
                c <= Tree.Children.GetUpperBound(0); c++)
            {
                CodeStatement[] cs =
                    GenerateStatements(Tree.Children[c]);
                for( int i = cs.GetLowerBound(0);
                    i <= cs.GetUpperBound(0); i++)
                {
                    mCodeStatements.Add(cs[i]);
                }
            }
        }
    }
    else
    {
        mCodeStatements.Add
            (new CodeSnippetExpression
            (((MethodInfo)Tree.Node).Name.ToString() + "()"));

    }
    CodeStatement[] cstatements =
        new CodeStatement[mCodeStatements.Count];
    IEnumerator enumStatements =
        mCodeStatements.GetEnumerator();
    int statement = 0;
    while(enumStatements.MoveNext())
    {
        cstatements[statement] =
            (CodeStatement)enumStatements.Current;
        statement++;
    }
    return cstatements;
}
Figure 11 BuildExecute (Close Figure 11)
private CodeMemberMethod BuildExecute (ExpressionTree Tree)
{
    // this is the fixed part of each execute method:
    CodeMemberMethod m = new CodeMemberMethod();
    m.ReturnType = new CodeTypeReference(typeof(void));
    m.Attributes = MemberAttributes.Public |
        MemberAttributes.Override;
    m.Name = "Execute";

    CodeMethodReferenceExpression loopCheckMethod = new
        CodeMethodReferenceExpression();
    loopCheckMethod.MethodName = "NotFinished";

    CodeMethodInvokeExpression loopCheck = new
        CodeMethodInvokeExpression(loopCheckMethod,
        new CodeExpression[0]);
    CodeIterationStatement whileLoop = new CodeIterationStatement
        (new CodeSnippetStatement(""),loopCheck,
        new CodeSnippetStatement(""),
        GenerateStatements(Tree));
    m.Statements.Add(whileLoop);

    return m;
}

Generator Class—Compiling New Code and Executing It

CodeDOM is used to create and populate the CodeCompileUnit for the generation namespace. The final steps needed to process a generation are code generation, compilation, and execution of all the new ant instances. These are all performed at the conclusion of the Generator.NewGeneration method. First, a C# code file is generated, as shown in the following code:

// Create the generated .cs file
CodeDomProvider provider = new CSharpCodeProvider();
// Obtain an ICodeGenerator from a CodeDomProvider class.
ICodeGenerator gen = provider.CreateGenerator();
// Create a TextWriter to a StreamWriter to
// an output file.
IndentedTextWriter tw = new
  IndentedTextWriter(new StreamWriter(
    mGeneratedNamespaceName + ".cs", false), "    ");
// Generate source code using the code
// generator.
gen.GenerateCodeFromCompileUnit(mCompileUnit, tw, new
  CodeGeneratorOptions());
// Close the output file.
tw.Close();
Then the code is compiled:
ICodeCompiler compiler = provider.CreateCompiler();
CompilerParameters cp = new CompilerParameters();
cp.GenerateInMemory = true;
cp.GenerateExecutable = false;
cp.ReferencedAssemblies.Add("CodeDom.exe");
cp.OutputAssembly = mGeneratedNamespaceName
  + ".dll";

CompilerResults cr = compiler.CompileAssemblyFromFile(
    cp, mGeneratedNamespaceName + ".cs");

I'll be generating lots of these code files, and each one will be an assembly that I'll load and execute. Finally, I'll use reflection again to find all the types in the new assembly, which is named "a". I'll invoke the Execute method on each and save its TerminationReport, as shown in Figure 13.

Figure 13 Save to TerminationReport (Close Figure 13)
Type [] mytypes = a.GetTypes();
foreach(Type t in mytypes)
{
    Ant a2 = (Ant)Activator.CreateInstance(t);
    // the ant will stop after 400 steps, or when
    // it has gathered its Food Goal
    a2.Init(ref grid, 400, FoodGoal);
    a2.Execute();

    // when the Execute() method finishes, get its terminating exception
    // to save its results
    Ant.TerminationReport doneReport = a2.mConcludingReport;
    // remaining code to save exception internally....
}

Building and Running the Application

The download consists of a single Visual Studio® .NET 2003 solution called CodeDOM. While the application runs, generated code files and DLLs accumulate in the \bin directory, so it's important to clean this directory occasionally. When the application starts up, it loads the DefaultGrid.XML file to paint an initial trail in the panel. It looks for that in the directory where the executable is running. It is currently located in the \bin\debug directory. If the application is run from another directory, the DefaultGrid.XML file should be moved there. The application will only run one experiment at a time.

Working with a Complex Problem

I'll conclude with a simple example that demonstrates how useful the application can be to perform experiments. Start the application, press the Get Grid button, and open the code file (called AlternateGrid.xml in the download). The winding trail has 76 pieces of food, with many gaps of one or two spaces. Change the number of generations to 20 and click the Generate button. When execution completes, the generation summary will show a clear increase in average fitness. Select Generation 20, and then select its best ant. Since the Generator works off of a Random instance with a fixed seed, the best ant in this generation will have gathered 65 items of food in 400 steps. The grid in Figure 14 shows the trail that appears when the Replay All button is used for this ant. It's a meandering, weaving pattern that doesn't appear to be very intelligent, but because the trail winds through the grid it manages to gather most of the available food.

Figure 14 Inefficient Ant Trail
Figure 14 Inefficient Ant Trail

Now let's add some intelligence to our base ant. Shut down the application, open the solution in Visual Studio, and find the code for the Ant class. There is a commented-out method called FacingGap. This method is just like FacingFood, but it allows the ant to see if there's a gap ahead, in addition to seeing if there is food ahead. You should uncomment the FacingGap method and rebuild the application.

Now do the same experiment that we just performed. Open AlternateGrid, select 20 Generations, and click the Generate button. Now the generator has an additional function available (FacingGap), and it will use this function as it generates the initial random ExpressionTrees. When the run is finished, find the best ant from Generation 20. Look at the code window and note its use of the FacingGap function. When the Replay All button is clicked, the trail in Figure 15 appears. The new function that was made available allowed the algorithm to generate an ant that will find all the food in close to the minimum number of steps.

Figure 15 Trail of an Efficient Ant
Figure 15 Trail of an Efficient Ant

The analogy to nature here is the availability of a new gene. FacingGap is like a gene that makes the developed organism more sensitive to something in its environment—for example, an enhanced sense of smell. It is important to note that the algorithm employed this new "gene" with no knowledge of its meaning or usefulness. Its usefulness only became apparent when the organisms that included it manifested higher fitness than others.

Figure 16 Plotting Generations
Figure 16 Plotting Generations

When the program executes, it outputs a summary of each generation and its best ant to a file, LastRun.csv. Figure 16 shows the progress that the algorithm made in solving the last problem. Note the dramatic improvement in Generation 14, when an ant was discovered that gathered 76 items of food in 126 steps. The average score increased in the remaining generations because the progeny of that ant came to dominate the population, and further improvements lowered the required steps to 106. Figure 17 maps genetic programming terms to nature.

Figure 17 Genetic Programming in Nature (Close Figure 17)

Concept | Natural Evolution | GP Framework
Genotype—the genetic blueprint for an individual | DNA | Expression tree
Phenotype—an individual with their developed characteristics | Living organism | Generated ant
Natural Selection | The process by which the fittest individuals are "chosen" to survive and reproduce | Fitness function
Reproduction | Sexual reproduction | Crossover

Further Work and Extensions

As stated before, the Generator class is not completely generic because it uses some internal knowledge about the TerminationReport class. This can be easily corrected by passing the TerminationReport as a class to the Generator constructor. It would be used in a manner similar to the way it uses the BaseClass type that is passed to it now.

This implementation of Artificial Ant is a simple one that was developed to show the potential for CodeDOM and reflection for genetic programming applications. It has a number of performance limitations that should be addressed in order to develop a useful, real-world problem solver. Fortunately, there is nothing in the architecture of .NET that would be an obstacle to high-performance genetic programming.

The first limitation is unnecessary code generation and execution. You'll probably recall that 10 percent of the ants in each generation are copied to the next generation. There is no reason to generate code and execute it for these ants—since they are unchanged, the generated code and behavior are unchanged as well.

The application generates a code file and then compiles the file. This is an anachronism from the development process because it was important to be able to see and debug code compilation during development of the code generation methods. It would be far more efficient to use the CompileAssemblyFromDOM method to build the assembly directly from the CodeCompileUnit.

Perhaps the single most important performance enhancement would be to implement one of the standard steps in GP processing. This step, called editing, takes expression trees and simplifies them. For example, functions with clauses like "Left; Right" can be removed because they just consume steps without changing the ant's state. In this sample application, the ant has only a single Boolean function, and many ants use deeply nested, completely redundant checks of this.

Editing and pruning code expressions would dramatically improve performance since the code fragments that must be manipulated would be far simpler. Editing has a practical benefit to the experimenter as well. Generated ants will be simpler, so it will be easier for an analyst to understand the problem-solving strategy they represent.

Brian Connolly is an independent consultant who specializes in transaction system design and performance analysis. He has led development teams that have produced international trading systems, data warehouses, and a major media Web site. Contact Brian at http://www.ideajungle.com.


Web Parts: Use Windows SharePoint Services as a Platform for Building Collaborative Apps, Part 2

Web Parts

Use Windows SharePoint Services as a Platform for Building Collaborative Apps, Part 2

Jason Masterman and Ted Pattison
This article discusses:
  • The WSS and SPS object models
  • Web Part Page anatomy
  • Creating and deploying Web Parts
  • Web Part security
This article uses the following technologies:
SharePoint, ASP.NET, C#
Code download available at: WebParts.exe (204KB)
Contents
The WSS and SPS Object Models
Web Part Infrastructure
Writing a Custom Web Part
Web Part Customization and Personalization
Other Web Part Development Features
Web Part Deployment and Distribution
Importing Web Parts
Conclusion

I
n the first installment of this article in the July 2004 issue, we examined the architecture of SharePoint® Products and Technologies. You saw that Windows® SharePoint Services (WSS) supplies a facility for building collaborative Web sites. Microsoft® Office SharePoint Portal Server (SPS) 2003, which is part of the Office System, complements WSS by adding features designed to assist users in navigating through the otherwise disconnected WSS-delivered SharePoint sites, as well as vast amounts of information, documents, people and application resources within an organization. SPS also supplies additional functionality to enhance portal sites using indexing, searching, audience targeting, and single sign-on (SSO).

In this second of two parts, we will start by examining the WSS and SPS object models, before moving into a detailed discussion of Web Part infrastructure where we will show how to create custom Web Parts. (For a look at the first part of this series, see SharePoint: Use Windows SharePoint Services as a Platform for Building Collaborative Applications.)

WSS provides the underlying infrastructure that lets you generate user interfaces by including Web Part Page and Web Part technology. Web Part Pages and Web Parts offer lots of power because they allow each WSS site and SPS portal site to have a browser-based user interface that is extensible and fully customizable. Web Parts also allow you to track personalization information on a user-by-user basis.

The WSS and SPS Object Models

WSS and SPS each expose their own object model. The WSS object model allows you to develop administrative applications that can create and manage virtual servers, site collections, sites, workspaces, and users. You can also use the WSS object model to design and implement user-targeted applications that manage lists and document libraries as well as the items they contain. The SPS object model allows you to programmatically manipulate areas and listings and automate other features such as the SPS services for search and SSO. Moreover, since every SPS portal site is an extension of the WSS site technology, SPS also exposes and makes use of the WSS object model.

The WSS and SPS teams have exposed their object models through class library DLLs as well as through a set of built-in Web services. Thus, when you want to program against the WSS object model using the class library you should add a reference to Microsoft.SharePoint.dll. If you want to program against the WSS object model using Web services, you should add a Web reference to one of the built-in Web services such as http://AcmeServer/_vti_bin/Lists.asmx.

There are a few things to consider when deciding if you will use the class libraries or Web services. The class libraries are easier to use and they support more features. However, a custom application that uses one of the class library DLLs can only run on a server within a WSS or SPS deployment. Programming against the object model of WSS and SPS using the Web services doesn't provide as much functionality and often requires you to program in terms of XML. If you build your application using Web services, however, it can be run on an administrator's desktop computer or even a user's laptop. The whole point of using the built-in Web services is that it allows you to write a custom application to communicate with a WSS or SPS deployment from across the network.

When you begin to program against the WSS object model, you may find that the changes in terminology between WSS and its predecessor, SharePoint Team Services (STS), cause some confusion. For example, the new WSS term "Site Collection" is the equivalent of the old STS term "Site." The new WSS term "Site" is the equivalent of the old STS term "Web" and the new WSS term "Top-level Site" is the equivalent of the old STS term "Root Web."

While the use of the new WSS terminology in the product documentation has been consistent, the names of classes in the WSS object model are based on the old STS terms. For example, you program against a site collection using an SPSite object. You program against a site using an SPWeb object. An SPSite object provides a public property named RootWeb that returns an SPWeb object representing the site collection's top-level site. Once you understand this somewhat confusing terminology, the WSS object model becomes much easier to learn.

The WSS object model exposed by the class library provides a top-level GlobalAdmin class that serves as an entry point into a WSS deployment. You can use a GlobalAdmin object to enumerate through the virtual servers, site collections, and sites within a WSS or SPS deployment, as shown in Figure 1. Within each site, you can then inspect its lists, document libraries, users, and child sites.

Figure 1 Enumerating the Lists of a Virtual Server (Close Figure 1)
// get GlobalAdmin object for current WSS deployment
SPGlobalAdmin globalAdmin = new SPGlobalAdmin();

// enumerate through IIS Virtual Servers
foreach(SPVirtualServer vServer in globalAdmin.VirtualServers)
{
  // determine which IIS Virtual Servers are extended with WSS
  if(vServer.State == SPVirtualServerState.Ready)
  {
    // enumerate through Site Collections of each Virtual Server
    foreach(SPSite siteCollection in vServer.Sites)
    {
      // enumerate through Sites of each Site Collection
      foreach(SPWeb site in siteCollection.AllWebs)
      {
        // enumerate through Lists of each Site
        foreach(SPList list in site.Lists)
        {
          Console.WriteLine(list.Title);
        }
      }
    }
  }
}

Two sample applications accompany this article. The first, SharePoint Site Browser, illustrates how you can program against the WSS object model using the class library. We wrote this Windows Forms application to demonstrate the techniques required for enumerating through virtual servers, site collections, sites, lists, document libraries, and users. A screen shot of the application is shown in Figure 2.

Figure 2 SharePoint Site Browser (Close Figure 2)

We wrote a second sample application named the Portal Area Browser that demonstrates how to program against the SPS object model. It shows how to program against the SPS class library DLLs to navigate through areas and inspect their listings. Both of these applications, complete with full source code, can be downloaded from the MSDN®Magazine Web site.

Web Part Infrastructure

As we mentioned earlier, Web Part Pages provide each site with an extensible HTML-based user interface. You can customize a Web Part Page by adding and configuring Web Parts. Users can further personalize them by modifying existing Web Parts and adding new ones. The fact that the Web Part Page and Web Part technologies created for WSS are being carried over as a major new feature in ASP.NET 2.0 is a testament to their power.

So how do Web Part Pages differ from standard ASP.NET pages? A standard ASP.NET page is stored as a text file on the file system of the Web server. The pieces that make up a Web Part Page, on the other hand, are stored in multiple tables inside the SharePoint content database. WSS constructs Web Part Page objects on the fly by retrieving data from these tables. This dimension of the Web Part architecture makes it possible to customize and personalize sites.

Let's take a moment and examine a Web Part Page from the perspective of a Web Part user. If you are a site owner or you have Web designer permissions, you can view and modify a Web Part Page in either shared view or personal view. Using the browser, you can switch back and forth using the Modify Shared Page menu, as shown in Figure 3. Once you have switched over to your personal view, the menu caption in the upper right-hand corner of the Web Part Page changes to Modify My Page.

Figure 3 Modify Shared Page (Close Figure 3)

To customize the Web Parts on a Web Part Page, you should enable the "Design this Page" option. You can also add new Web Parts to a Web Part Page using the Add Web Parts menu. If you are working in a Web Part Page in shared view, the customizations you apply to Web Parts are seen by all users. If you switch from shared view to personal view, your modifications are only seen by you. WSS is smart enough to store the shared customization data and private personalization data separately in the content database.

You must be a site owner or have Web designer permissions to modify a Web Part Page in shared view. When a Web Part Page is generated for a user who is neither the site owner nor a Web designer, that user will not be allowed to switch over to a shared view. Instead, the page only provides a "Modify My Page" menu. However, the user can still enable the "Design this Page" option and, consequently, add and customize Web Parts. Any modifications made by the user using these features will be stored as personalized data rather than shared data.

A Web Part Page is laid out in terms of Web Part Zones. You add a Web Part to a Web Part Page by placing it in a Web Part Zone. WSS allows a site owner or Web designer to create new Web Part Pages using one of the templates provided. With the browser, you can use one of these Web Part Page templates to create a new Web Part Page with a predefined layout of zones. If you use Microsoft Office FrontPage® 2003 to create and design Web Part Pages, you have even more flexibility because you can add, remove, and reposition zones on a Web Part Page using the FrontPage page designer.

WSS provides a rendering engine for Web Part Pages by leveraging and extending the underlying ASP.NET infrastructure. WSS routes each Web Part Page request to an instance of the SharePointHandler class—a custom ASP.NET handler. This handler class is defined in the Microsoft.SharePoint.ApplicationRuntime namespace and resides in the Microsoft.SharePoint.dll assembly.

For each Web Part Page request, the SharePointHandler object is responsible for retrieving all the necessary data from the content database. This includes the Web Part Page document which defines a layout of Web Part Zones. The SharePointHandler object must also retrieve data from other tables within the content database to determine how Web Parts have been customized and personalized within specific zones. WSS optimizes the retrieval of all the data it needs for a Web Part Page by obtaining it in a single round-trip to SQL Server™.

Figure 4 Assembling a Web Part Page
Figure 4 Assembling a Web Part Page

After retrieving the data associated with a Web Part Page, the SharePointHandler object is responsible for assembling the HTML page to be returned to the user as shown in Figure 4. The SharePointHandler object creates an object for each Web Part and initializes it with the appropriate customization and personalization data. Finally, the SharePointHandler object enters the rendering phase where it steps through all the Web Part objects in sequence, giving them an opportunity to contribute HTML for the output of the page.

Writing a Custom Web Part

With your newfound understanding of Web Part infrastructure, let's turn our attention to the creation of custom Web Parts using Visual Studio® .NET. To get started with Web Part development, you should download and install the Web Part Templates for Visual Studio .NET. These templates are distributed in an installer file (.msi) that can be found at SharePoint Products and Technologies. After you have installed the Web Part Templates, Visual Studio .NET will provide a new project template for creating a Web Part library DLL, along with various other templates for creating files related to Web Part development, such as a source file with the starting point for a Web Part class.

You create a Web Part by authoring a class that inherits from the WebPart class defined inside the Microsoft.SharePoint.dll assembly. The WebPart class exists within the Microsoft.SharePoint.WebPartPages namespace. You are required to override methods of the WebPart class to take advantage of many Web Part features. To create the traditional "Hello, World" Web Part class, you simply need to override a single method named RenderWebPart:

using Microsoft.SharePoint.WebPartPages;
using System.Web.UI;

namespace AcmeWebParts {
  public class TedsWebPart : WebPart {
    protected override void RenderWebPart(HtmlTextWriter output) {
      output.Write("Hello, World");
    }
  }
}

Figure 5 shows how custom Web Part classes fit into a larger inheritance hierarchy. As you can see, the WebPart class created by the Microsoft WSS team inherits from the Control class created by the ASP.NET team. Therefore, a Web Part is really a specialized type of ASP.NET control. If you have experience developing ASP.NET Web applications or server-side controls, many of the programming techniques you already use can be readily applied when developing Web Parts.

Figure 5 Web Part Class Inheritance Hierarchy
Figure 5 Web Part Class Inheritance Hierarchy

When it comes to providing the code to render HTML, however, there's an important difference between developing ASP.NET controls and developing Web Parts. When you author an ASP.NET control, you render HTML by overriding the Render method. When you author a Web Part, you override the RenderWebPart method instead. Why is this?

The Render method defined in the WebPart class contains generic code to generate the Web Part's title bar and borders. In SharePoint Products and Technologies terminology, this is known as the "chrome" around a Web Part. You should not override the Render method provided by the WebPart class because your resulting Web Part class will not render the standard Web Part chrome correctly. Instead, you should override the RenderWebPart method because this allows you to generate HTML that is displayed properly inside the chrome.

The RenderWebPart method provides the same HtmlTextWriter parameter as the Render method. This means that you can generate HTML for a Web Part in the RenderWebPart method using the same techniques you use for a custom ASP.NET control in the Render method. For example, you can generate an HTML table by calling the RenderBeginTag method and the RenderEndTag method of the HtmlTextWriter parameter, as shown in Figure 6.

Figure 6 Rendering in a Custom Web Part (Close Figure 6)
override void RenderWebPart(ByVal output As HtmlTextWriter) {

  // create HTML table with custom attributes
  output.AddAttribute(HtmlTextWriterAttribute.Cellpadding, "5");
  output.AddAttribute(HtmlTextWriterAttribute.Border, "2");
  output.RenderBeginTag(HtmlTextWriterTag.Table);

  // create new row
  output.RenderBeginTag(HtmlTextWriterTag.Tr);

  // create new cell
  output.RenderBeginTag(HtmlTextWriterTag.Td);
  output.Write("Name:");
  output.RenderEndTag();      // </td>

  // create new cell
  output.RenderBeginTag(HtmlTextWriterTag.Td);
  output.Write("Bob Smith");
  output.RenderEndTag();      // </td>

  output.RenderEndTag();      // </tr>
  output.RenderEndTag();      // </table>
}

A Web Part, like other ASP.NET controls, can be designed to be a composite control containing child controls. If you base your custom Web Parts on familiar ASP.NET server-side controls, you can create detailed views and polished data-entry screens without getting bogged down generating DHTML and JavaScript. For example, you can create sophisticated-looking Web Parts quickly using some ASP.NET controls including the DataGrid control, the Calendar control, and the ASP.NET validation controls.

You must override the CreateChildControls method to create a custom Web Part with child controls. Your implementation of CreateChildControls should create and initialize child control objects and then add them to the Web Part object's Controls collection. A simple example that creates a TextBox object and a Button object is shown in the definition for JasonsWebPart in Figure 7.

Figure 7 Custom Web Part with Child Controls (Close Figure 7)
public class JasonsWebPart : WebPart {

  // fields to hold child controls
  protected TextBox txtName;
  protected Button btnSubmit;

  // create child control
  protected override void CreateChildControls() {
    txtName = new TextBox();
    this.Controls.Add(txtName);
    btnSubmit = new Button();
    btnSubmit.Text = "Submit Name";
    this.Controls.Add(btnSubmit);
  }

  // render HTML for Web Part
  protected override void RenderWebPart(HtmlTextWriter output) {
    txtName.RenderControl(output);
    output.RenderBeginTag(HtmlTextWriterTag.Br);
    btnSummit.RenderControl(output);
  }
}

Examine the implementation of the RenderWebPart method in the JasonsWebPart class and see how it calls the RenderControl method on each child control. When you call RenderControl on a child control and pass it the HtmlTextWriter parameter, the child control responds by writing its HTML into the output stream that will be returned to the user. This example illustrates how a Web Part's implementation of the RenderWebPart method is written to combine the HTML output of all the child controls into a single HTML element.

Web Part Customization and Personalization

One of the most powerful aspects of Web Part technology is its ability to track customization and personalization data. Thankfully, it's also relatively easy to implement. You simply add properties to a Web Part class and mark these properties with special attributes that have been defined by the WSS team. This strategy represents a great example of the power of declarative programming made possible by the .NET Framework. You are not required to write any code for storing or retrieving either customization data or personalization data. All of this is handled by Windows SharePoint Services behind the scenes.

Imagine a scenario in which you are required to develop a custom Web Part that displays the user's local weather report. Since different users live in different places, your Web Part must be designed to track a personalized ZIP code for each user. You can design a Web part to track personalized ZIP codes by defining a ZipCode property with a special set of attributes. It's the presence of these attributes in your compiled code that tells WSS how you would like the property value to be customized or personalized.

Examine the WeatherReportWebPart class definition in Figure 8. This Web Part class defines a public ZipCode property together with a protected _ZipCode field to track the required personalization data. As you can see, the ZipCode property definition contains the WebPartStorage attribute which has been parameterized for personal storage. The presence of the WebPartStorage attribute tells WSS that it should automatically store and retrieve personalization data for this property.

Figure 8 Persisting Personalization Settings (Close Figure 8)
[XmlRoot(Namespace="AcmeWebParts")]
public class WeatherReportWebPart : WebPart {

  // Web Part property attributes
  [WebPartStorage(Storage.Personal),
   DefaultValue(""),
   Browsable(true),
   FriendlyName("Zip Code"),
   Category("User Info")]
  public string ZipCode {
    get { return _ZipCode; }
    set { _ZipCode = value; }
  }

  // field used as backing store for ZipCode property
  protected string _ZipCode = string.Empty;
}

When the SharePointHandler object initializes a Web Part object from the WeatherReportWebPart class, it checks to see if there is customization or personalization data for the Web Part in the content database. If WSS finds a preexisting ZipCode property value, it assigns this value to the ZipCode property while it is initializing the Web Part object. Since the ZipCode property's set block assigns the personalized value to the _ZipCode field, the user's ZIP code is available to all other methods in the WeatherReportWebPart class throughout the lifetime of the Web Part object. This example demonstrates a common technique in which a protected or private field is used to provide a backing store for the public property value.

When you define a persistent property like ZipCode in a Web Part class, WSS automatically supplies user interface elements that allow users to customize property settings. When a user runs a menu command on a Web Part Page to modify a Web Part, WSS displays a Task Pane containing Tool Parts on the right-hand side of the browser window.

A Tool Part is a WSS user interface element that allows a user to examine and modify Web Part property values in the Task Pane. When in shared view, a Tool Part allows the user to work with customized values. When in personal view, a Tool Part allows the user to work with personalized values.

WSS provides standard Tool Parts that allow users to inspect and modify Web Part properties. If you don't like the generic user interface elements created by these standard Tool Parts, WSS makes it possible for you to create your own, and you can take control of how the user interface is generated when a user wants to customize or personalize your custom Web Part properties. Authoring a Tool Part is similar to authoring a Web Part because you can build it as a composite control containing child ASP.NET controls.

Web Part technology is powerful because it provides a sophisticated facility for tracking customization and personalization data. Your sites can be customized and personalized by different users in different ways. All the while, you never have to write any code to manage user membership or to store and retrieve property values inside the content database, so you can spend all of your time writing your business logic instead.

Other Web Part Development Features

When you are writing code for a Web Part, you can program against the WSS object model to access the data for a list or a document library. You can also create a custom Web Part that allows a user to inspect or configure the various security settings for a site. However, it doesn't usually make sense to use the GlobalAdmin object as the entry point into the WSS object model as was shown earlier. Instead, you should acquire an entry point using the SPControl class.

The SPControl class provides two static methods: GetContextSite and GetContextWeb. These two methods allow you to hook into the WSS object model within the context of the current request. The GetContextSite method returns an object that represents the current site collection and the GetContextWeb method returns an object that represents the current site. Both methods require you to pass the ASP.NET HttpContext object associated with the current request. Here's an example that uses the GetContextWeb method from within a custom Web Part to enumerate through the lists of the current site:

SPWeb currentSite = SPControl.GetContextWeb(this.Context);
foreach(SPList list in currentSite.Lists) {
  this.ListBox1.Items.Add(list.Title);
}

Another important aspect of Web Parts development is communicating across the network with Web services and other back-end systems. Web Parts are often used to give SPS portal sites and standard WSS sites the ability to act as gateways into enterprise-level applications. Microsoft and other companies have already built Web Parts that act as adapters into popular line-of-business applications such as such as BizTalk®, SAP, and PeopleSoft. You can follow suit and build custom Web Parts that will allow your users to access any of your back-end systems.

A set of security credentials are typically required by Web Parts that are written to communicate with Web services and line-of-business applications. Given this common requirement, SPS was designed with a built-in SSO service for storing and retrieving security credentials. The SSO service can cache security credentials on a user-by-user basis. Alternatively, it can store a common set of security credentials used by all the members of an Active Directory® group. The SSO service does the work required to track the Windows security ID (SID) of the current user when storing or retrieving a set of security credentials from the SSO database. The SSO service also employs the necessary encryption so that security credentials are never stored in the SSO database as clear text.

The Web Part infrastructure supports a feature known as Web Part connections, which represent an extensible customization model by which two or more Web Parts can be joined in a provider/consumer relationship. A user can manually create a connection between Web Parts using either the browser or FrontPage 2003. This makes it possible to create Web Part Pages that display sophisticated views of data with features like master/detail and parent/child relationships.

The WSS infrastructure for connecting Web Parts is based on a loosely coupled model. That means two Web Parts do not require knowledge of each other to be connected on a page. The loose coupling of this model provides the flexibility to connect Web Parts created by different development teams or even by different companies. For example, you can create your own custom Web Parts and connect them to standard Web Parts that are distributed by Microsoft or a third party.

Web Part Deployment and Distribution

You can deploy a Web Part library DLL on a Web server in one of two places: inside a virtual server's \bin directory just like you would deploy assembly DLLs for a standard ASP.NET application, or in the Global Assembly Cache (GAC). A Web Part library DLL in the GAC can be loaded into any virtual server on the hosting Web server machine.

While you are developing and testing Web Parts, it's convenient to configure your Web Part library project to compile its output DLL directly into the \bin directory of the virtual server you will be using for your testing. By following this approach, you will instantly be able to test your Web Parts using the browser after you recompile the Web Part library project.

The fact that Web Part Pages are stored inside the content database poses a security risk. Imagine a scenario in which some malicious individual is able to access the content database and modify the table that holds all the Web Part Page documents. What would happen if this person attempted to mount an attack by placing in-line code within the body of the Web Part Page? You can see the potential problem here.

Fortunately, WSS can defend itself against this kind of attack by processing Web Part Pages in safe mode. When the SharePointHandler object processes the request for a Web Part Page, it will never execute any inline code it finds within the body of the page. Furthermore, the SharePointHandler object will only load Web Parts and ASP.NET controls that have been explicitly configured as "safe controls." You can configure Web Parts and ASP.NET controls using the <SafeControls> section of the hosting virtual server's web.config file.

You will experience an error whenever you run a Web Part Page that attempts to load a Web Part or an ASP.NET control that has not been properly configured as a safe control. Clearly, there must be a set of standard Web Part and ASP.NET controls that WSS marks as safe by default. Whenever you extend a virtual server with WSS, the web.config file that is automatically copied to the root directory contains a <SafeControls> element that includes a set of standard Microsoft Web Parts and ASP.NET server-side controls.

When you deploy a custom Web Part library DLL onto a front-end Web server running WSS, you must modify the web.config file for each virtual server where you plan to run these Web Parts. The <SafeControls> entry in the web.config file should be modified to look like the following code:

<!–- in web.config of hosting virtual server -->
<configuration>
  <SharePoint>
    <SafeControls>
      <SafeControl Assembly="AcmeWebParts" Namespace="AcmeWebParts"
                   TypeName="*" Safe="True" />
   </SafeControls>
  </SharePoint>
</configuration>

Note that the preceding example of a <SafeControl> element identifies an assembly using its friendly name (AcmeWebParts). When you deploy a strongly named assembly DLL, you should not use the simple friendly name. Instead, you should use the fully qualified, four-part assembly name which, by the way, you will always be required to use when you deploy a Web Part library DLL in the GAC (the GAC only contains strongly named assemblies).

The security layer provided by the Web Part architecture goes beyond requiring that Web Part and ASP.NET controls be configured as safe controls. A Web Part library DLL running inside the \bin directory is further restricted in what actions it can perform by the <trust> element defined in the web.config file, as shown here:

<configuration>
  </system.web>
    <!-- set level to (1) WSS_Minimal (2) WSS_Medium or (3) Full -->
    <trust level="WSS_Minimal" originUrl="" />
  </system.web>
</configuration>

The trust element contains an attribute named level. The value of this attribute configures Code Access Security (CAS) permissions that restrict Web Part library DLLs from performing potentially dangerous actions such as connecting to a SQL Server database or accessing the local file system. The trust element's level attribute is set to WSS_Minimal by default, which severely restricts what a Web Part library can do. This default setting often makes it difficult or impossible to test and debug custom Web Parts during the development cycle.

You can adjust the trust level to WSS_Medium if you want to relax the CAS restrictions. A setting of WSS_Medium usually allows you to test and debug a Web Part without security-related problems. Occasionally, you might have to adjust the trust level setting to Full in order to grant your part full trust. Note that you should run the IISRESET command after adjusting the trust level inside a web.config file.

Remember that the trust level defined in a web.config file only affects Web Part library DLLs running within the \bin directory. The trust-level setting does not affect Web Part library DLLs that have been installed in the GAC. Web Part library DLLs in the GAC are considered to be fully trusted and always run without any CAS restrictions. This factor can influence your decision to deploy a Web Part library DLL in the \bin directory or in the GAC.

Importing Web Parts

The final aspect of Web Part deployment centers on importing description information about the Web Part into a Web Part gallery. WSS supports Web Part galleries at the site-collection level as well as at the virtual-server level. However, before you can import a Web Part's description information into a Web Part gallery, you must first create a Web Part description file.

A Web Part description file is an XML file with a .dwp extension (see Figure 9). The Title and Description elements provide friendly text that is displayed to Web Part consumers as they are browsing through the list of Web Parts that can be added to a Web Part Page. The Assembly and TypeName elements provide WSS with the information it needs to load the associated Web Part class at run time. It's important to remember to use the fully qualified, four-part assembly name when creating a .dwp file for a Web Part library DLL with a strong name.

Figure 9 Web Part Description File (Close Figure 9)
<?xml version="1.0" encoding="utf-8"?>
<WebPart xmlns="http://schemas.microsoft.com/WebPart/v2" >

    <!-- display information for Web Part consumers -->
    <Title>Weather Report</Title>
    <Description>Acme Weather Report Web Part</Description>

    <!-- type information for loading Web Part at run time -->
    <Assembly>AcmeWebParts</Assembly>
    <TypeName>AcmeWebParts.WeatherReportWebPart</TypeName>

    <!-- property initialization values -->
    <ZipCode xmlns="AcmeWebParts">98052</ZipCode>

</WebPart>

A .dwp file can optionally contain initialization values for one or more Web Part properties. The .dwp file in Figure 9 shows how to use a custom <ZipCode> element to provide an initial default value for the Web Part's ZipCode property. The default ZIP code that's being used in this example is 98052, for Redmond, WA. This means that after being placed on a Web Part Page, this Web Part will initially show the weather report for the Microsoft campus. However, this initial ZipCode property value will be overwritten once it has been customized or personalized by a user.

Once you have created the Web Part description file, you can import the associated Web Part into WSS and use it on a Web Part Page. The simplest way to import it is to choose the Import command from the "Add Web Part" menu while customizing or personalizing a Web Part Page in the browser. The Add Web Parts menu is available as a sub menu of the "Modify Shared Page" menu and the "Modify My Page" menu in the upper right-hand corner of the browser window.

You can also import a Web Part description file into the Web Part gallery of a site collection to make it available to all sites in that site collection. To do this, navigate to the top-level site administration page and scroll down to the Site Collection Galleries section. Next, click the link with the caption "Manage Web Part Gallery." That will bring you to the site collection's Web Part Gallery page. This page provides the option of importing .dwp files.

In addition to importing a Web Part at the site-collection level, you can also import a Web Part into the Web Part Gallery for a virtual server. The process for doing this, however, is more complex because you must first build a Web Part Package. Once you have built the Web Part Package, you can install it using the STSADM.EXE administration tool.

A Web Part Package is a CAB file that contains a Web Part library DLL and all its associated .dwp files. You can easily build a Web Part Package with Visual Studio .NET by creating a new CAB project. When you create a Web Part Package, you must add a special header file named manifest.xml because it contains information that STSADM.EXE needs to properly install and configure a Web Part library DLL.

An example of manifest.xml is shown in Figure 10. As you can see, it contains the file name of the Web Part library DLL and a list of .dwp files. It isn't necessary to add the fully qualified, four-part assembly name of a Web Part library DLL to a manifest.xml file because the STSADM.EXE administration utility is able to discover the assembly name at run time when it loads the Web Part library DLL.

Figure 10 manifest.xml (Close Figure 10)
<?xml version="1.0"?>
<WebPartManifest xmlns="http://schemas.microsoft.com/WebPart/v2/
   Manifest">
  <Assemblies>
    <Assembly FileName="AcmeWebParts.dll">
      <SafeControls>
        <SafeControl Namespace="AcmeWebParts" TypeName="*" />
      </SafeControls>
    </Assembly>
  </Assemblies>
  <DwpFiles>
    <DwpFile FileName="TedsWebPart.dwp"/>
    <DwpFile FileName="JasonsWebPart.dwp"/>
    <DwpFile FileName="WeatherReportWebPart.dwp"/>
  </DwpFiles>
</WebPartManifest>

Once you have built a Web Part Package that contains a manifest.xml file, the Web Part library DLL and all the associated .dwp files, you can then install it using the STSADM.EXE administration tool. For example, if you have built a Web Part Package named AcmeWebPartsPack.cab, you can install it using the following command-line instruction:

STSADM.EXE -o AddWPPack -filename AcmeWebPartsPack.cab
When you run this command, the STSADM.EXE utility copies the Web Part library DLL into the \bin directory of each virtual server within the current WSS deployment. You can add the -url parameter to the preceding command-line instruction if you would like to install the Web Part library into only one specific virtual server within the current WSS deployment.

Installing a Web Part Package with STSADM.EXE provides the convenience of adding the required <SafeControl> elements to web.config files to configure each Web Part that is being installed as a safe control. This certainly eases large-scale deployment when you are rolling a new Web Part library DLL out into a Web farm environment.

When installing a Web Part Package with the STSADM.EXE utility, you have the option of installing Web Part library DLLs in the GAC. This can be accomplished by using the -globalInstall parameter, as shown here:

STSADM.EXE -o AddWPPack -filename AcmeWebPartsPack.cab -globalInstall
Using this technique makes it possible to install a Web Part library DLL only once per machine, no matter how many virtual servers use it. As mentioned earlier, Web Part library DLLs installed in the GAC are not restricted by the trust level defined inside the web.config file. Therefore, you should not install Web Part library DLLs into the GAC casually. You should only do that when you know that the DLL has come from a trusted source and that it poses no security risk whatsoever.

If you need to provide CAS settings as part of your Web Part installation, a CAB file installed with STSADM.EXE will not be sufficient. There is, however, a Web Part Packager tool and accompanying white paper that can be downloaded from SharePoint Products and Technologies. This tool creates .msi installation files that can contain CAS instructions in addition to all of the facilities common to CAB file deployments.

Conclusion

WSS and SPS are two different offerings that have been designed to work well together. WSS provides the foundation for creating collaborative Web sites that support customization and personalization and SPS complements WSS by playing the role of a content aggregator. Because SPS is fully dependent on the underlying infrastructure of WSS, the more you know about WSS, the easier it will be to master SPS.

On one hand, SharePoint Products and Technologies decrease the need for custom software development because WSS and SPS provide so much functionality right out of the box. Yet they also provide valuable opportunities to write custom applications and Web Parts. Now that you have a roadmap, it's time to get started on your SharePoint development.

Jason Masterman and Ted Pattison have both been working in the software industry for over a decade writing software, consulting, and educating developers. In 2003, Jason and Ted founded Barracuda .NET, a leading-edge consulting and training company that focuses on Microsoft collaboration technologies. Go to http://Barracuda.net to find out more about their training classes and online events.


Virtual Server 2005: Program Customized Testing Environments Without Trashing Your Machine

Virtual Server 2005

Program Customized Testing Environments Without Trashing Your Machine

Ben Waldron
This article discusses:
  • Virtual Server 2005
  • Consuming the Virtual Server 2005 COM API
  • .NET/COM interop
  • Test automation
This article uses the following technologies:
C#, Virtual Server, COM
Code download available at: VirtualServer2005.exe (677KB)
Contents
Introducing Virtual Server 2005
Using the COM API
Application Testing Scenario
Design of the Rapid Test Application
Virtual Server .NET Wrapper
Rapid Test Windows Form Application
VMRC ActiveX Control
Conclusion

O
ften when I walk by the desk of a software developer or tester I notice the arsenal of hardware used to test applications on different operating system versions or configurations. Testing in different configurations is an integral part of the development process, but the time and energy consumed configuring and managing these environments can be taxing. This ultimately comes at the expense of application quality because less time is spent testing while more time is spent building or reconfiguring machines. Microsoft® Virtual Server 2005 improves developer productivity by simplifying the provisioning and management of testing environments. This article demonstrates how the extensive Virtual Server COM API can be used to create an automated application testing environment.

In the context of this article, a virtual machine can be best described as an isolated computer implemented entirely in software, running inside a host hardware-based computer. This makes it possible to have multiple operating systems running simultaneously on a single machine. Windows® 95 and above and even the next version of Windows, code-named "Longhorn," can run concurrently on a single piece of hardware using Virtual Server. Non-Microsoft operating systems can also run inside a virtual machine. Applications running in a virtual machine behave as if they were running on their own physical system. The virtual machine concept has been around for more than 30 years, but until the advent of products like Virtual Server 2005 it has not been an affordable or manageable solution for application developers and testers.

Introducing Virtual Server 2005

Microsoft offers two products that employ virtual machine technology: Virtual PC 2004 and Virtual Server 2005. Virtual Server builds upon Virtual PC, adding improved resource utilization and an extensive API to automate deployment and management. The API is most significant for developers and is largely the subject of this article, but before looking at the API I'll take a moment to describe the architecture of Virtual Server 2005.

Figure 1 shows a high-level architecture of Virtual Server. Virtual machines are isolated from the host operating system by running atop a virtualization layer that emulates industry-standard hardware. The host, running Windows Server™ 2003, creates the "Virtual Service" Windows service responsible for isolating and managing virtual machines as part of the Virtual Server installation. It also houses the COM interface, event logging, resource management, and performance counters. IIS plays an important role because it acts as the host for the Virtual Server Web application—the primary interface for managing Virtual Server and its virtual machines.

Figure 1 Architecture
Figure 1 Architecture

In addition, the Virtual Service runs the Virtual Machine Remote Control (VMRC) server, which clients access in order to control virtual machines remotely. VMRC is also the name of the protocol used by the server to talk to client applications. The VMRC protocol is an extended form of the standard Virtual Network Computing (VNC) protocol. It uses an enhanced secure version of the Remote Frame Buffer (RFB) interface.

Users access virtual machines by using the rich VMRC client or Web browser via the VMRC ActiveX® control. If a virtual machine's configuration is such that it can be accessed directly from the network, then it is possible to use Remote Desktop Protocol (RDP) to connect to it and to control it remotely.

Virtual machines themselves can be thought of as traditional workstations or servers. They have the common peripherals that you would expect in a modern hardware configuration with the exception of USB and FireWire devices as well as PCI cards. Virtual machines can be configured to run multiple hard drives (IDE or SCSI), DVD-ROMs, network adapters, and other devices. Although the virtual machines see them as normal devices, hard disks are actually files on the host system and other devices are created virtually in the software by the Virtual Service.

Although I'll discuss the basic tools you'll need in order to develop for Virtual Server, you can find more information on the Virtual Server Architecture at Microsoft Virtual Server 2005.

Using the COM API

As you may have noticed, there has been a move towards products that are designed with customization in mind and away from shipping applications with limited programmatic access. Microsoft products like Office have full APIs that allow developers to do anything that can be done in the core product and provide many additional options. This shift is great news because it means less time searching for workarounds and more time developing solutions.

Virtual Server uses a COM API that can be accessed from the Windows scripting environment and from applications written in managed .NET code, Visual Basic® 6.0, or unmanaged C++. I'll use C# to build a Windows Forms app that shows the core functions of the API. The COM interface is contained within the Virtual Service (vssvrc.exe), and it supports DCOM to enable remote programmatic access. The best reference for the API is in the documentation delivered as part of the installation.

The most important interfaces and classes in the COM API are IVMVirtualServer and IVMVirtualMachine and their respective coclasses VMVirtualServer and VMVirtualMachine. All other classes, events, and enumerations support these two interfaces. The VMVirtualServer is the core of the API and its functionality can be summed up by the functions listed in Figure 2. Once virtual machines are provisioned (using a VMVirtualServer), they can be managed using instances of the VMVirtualMachine class, also described in Figure 2.

Figure 2 VMVirtualServer and VMVirtualMachine (Close Figure 2)

VMVirtualServer Functions | Description
Retrieve host information | Returns information about the host machine running Virtual Server. Host information includes operating system version, processor speed, and available memory to assist in sizing and scaling.
Provision virtual machines | Able to create, delete, find, or register virtual machines.
Manage peripherals | Create, configure, and remove peripherals such as hard disks, network adapters, or DVD-ROM drives.
Security | Configure access rights through the COM API to virtual machines.
VMRC | Manages VMRC service. Administrators can enable or disable VMRC and access properties such as encryption type, VMRC port, and screen resolution.
VMVirtualMachine Functions | Description
Peripheral management | Add or remove hard disks, network adapters, floppy drives, and other peripherals.
Virtual machine actions | Able to start, shut down, save, reset, pause, and resume virtual machines.
Virtual machine information | Manage machine properties such as BIOS, base board, and chassis.

Once virtual machines are provisioned using the VMVirtualServer class, the VMVirtualMachine class is used to manage virtual machines. The VMVirtualMachine class performs the functions that are described at the bottom of Figure 2.

Application Testing Scenario

As mentioned earlier, one of the most important capabilities of Virtual Server is that it allows a developer or tester to deploy and test applications quickly in a variety of operating systems and configurations. After developing Windows Forms applications and performing unit testing, I find myself creating installation packages that will need to be installed on many different versions of Windows and on a variety of service pack levels. This is the part of the development cycle that I like least because it is less creative and sometimes tedious, though still important. I invariably end up having to image machines over and over until I am comfortable with the installation package and application.

Fortunately, there is one feature of Virtual Server that saves a lot of time in this scenario—undoable disks. This feature allows any changes made to the virtual machine to remain uncommitted so that when the virtual machine is stopped, it will not persist any transactions to the disk. This means that if you encounter an error, you can simply roll back the disk without having to completely re-image the virtual machine.

I've created an application called Rapid Test to automate the management of virtual machines and to quickly and easily deploy new applications for testing. The goal of the Rapid Test application is to take a newly developed piece of software and run it quickly in many different environments. Imagine taking a setup program that you just created and installing it on four or five operating systems in less than five minutes. This is the main task that I built Rapid Test to perform.

Rapid Test enables you to create new virtual machines from an existing virtual hard disk or from a new build image that has been encapsulated in a virtual disk. An existing virtual hard disk is a file on the host system that has been preconfigured within Virtual Server or Virtual PC and which can also be attached to and used in a new system. The build image approach takes a fresh copy of a configuration that has been constructed using deployment automation techniques like SysPrep or Windows Automated Deployment Services. With either approach, it is possible to create a new virtual machine in just a matter of minutes.

The other important ability of Rapid Test is that it lets you deploy applications to virtual machines quickly. If virtual machines have network adapters configured for client access, application deployment can be performed over the network. I chose not to assume that the network was configured, so I took a different approach to deploying applications. Instead of deploying over the network, the user selects a directory to deploy to the virtual machines and an ISO DVD-ROM image is created and copied to the host server. Virtual machine DVD-ROM devices can be bound to either a physical DVD-ROM drive on the host or an ISO file image that mimics DVD-ROM media. This way of deploying the app is analogous to a user installing an application from a CD or DVD.

When the virtual machine is started, it recognizes that it needs to bind one of its DVD-ROM drives to the ISO image. When the machine boots or comes out of a saved state, the DVD-ROM is recognized and the application is available to install and test. Using an autorun.inf file within the ISO image can eliminate the need to run the app manually; the installation will start automatically.

Design of the Rapid Test Application

There are two main components that comprise the Rapid Test application: the Windows Forms application and my Virtual Server .NET wrapper classes. The high-level design of the application is shown in Figure 3.

Figure 3 Rapid Test Architecture
Figure 3 Rapid Test Architecture

The Virtual Server wrapper exposes the VMVirtualServer and VMVirtualMachine COM classes in .NET managed code. It has two purposes. First, it allows easier consumption from managed applications by abstracting the COM interface. Second, it allows other applications to take advantage of the library or extend it. For example, a great enhancement to the solution would be to develop a Web service layer that consumes the wrapper. The wrapper does not expose all the functionality contained within the COM API, but it does expose the core properties, methods, and events needed for the Rapid Test solution.

Virtual Server .NET Wrapper

All classes within the Virtual Server .NET wrapper use a naming convention that starts with the letters VSW. The wrapper class for the VMVirtualServer class is VSWVirtualServer. The code in Figure 4 shows the constructors for the VMVirtualServer class. You can use the default constructor if you are connecting to a local virtual server while you can provide the name or address of a remote server to the other constructor. If you specify a remote server, DCOM is used to connect to the Virtual Server remotely.

Figure 4 VSWVirtual Server Constructors (Close Figure 4)
/// <summary>
/// Create a Virtual Server object connected to the local server
/// </summary>
public VSWVirtualServer()
{
  virtualServerCOM = new VMVirtualServerClass();
}

/// <summary>
/// Create a Virtual Server object connected to a remote server
/// </summary>
/// <param name="server">Name or address of remote server</param>
public VSWVirtualServer(string server)
{
  Type VMVirtualServerClassType = typeof(VMVirtualServerClass);
  // create remote type from class identifier
  Type DCOMType = Type.GetTypeFromCLSID(VMVirtualServerClassType.GUID,
                                        server, true);
  object DCOMObject = Activator.CreateInstance(DCOMType);
  // create local object from remote object
  virtualServerCOM = (VMVirtualServerClass)
    Marshal.CreateWrapperOfType(DCOMObject,
      VMVirtualServerClassType);
}

The VSWVirtualServer class exposes methods for creating and removing virtual machines. The creation of a new virtual machine requires you to use the COM API method CreateVirtualMachine (see Figure 5) and pass it the name of the new virtual machine and the location to create configuration files. Once it is created, the wrapper adds a hard disk, DVD-ROM, and then configures the amount of memory.

Figure 5 Creating a New Virtual Machine (Close Figure 5)
/// <summary>
/// Create a new virtual machine
/// </summary>
/// <param name="virtualMachineName">Name of Virtual Machine to
/// create</param>
/// <param name="hardDrivePath">Path to hard disk</param>
/// <param name="amountMemory">Amount of memory</param>
/// <returns>.NET wrapped Virtual Machine</returns>
public VSWVirtualMachine CreateVirtualMachine(string virtualMachineName,
    string hardDrivePath, int amountMemory)
{
  VMVirtualMachine virtualMachineCOM =
    virtualServerCOM.CreateVirtualMachine(virtualMachineName,
    virtualServerCOM.DefaultVMConfigurationPath);
  // add the initial hard drive - assumes IDE (bus 1, device 0)
  virtualMachineCOM.AddHardDiskConnection(hardDrivePath,
    VMDriveBusType.vmDriveBusType_IDE, 1, 0);
  // add a DVD drive - DVD always IDE (bus 1, device 1)
  virtualMachineCOM.AddDVDROMDrive(VMDriveBusType.vmDriveBusType_IDE,
                                   1, 1);
  virtualMachineCOM.Memory = amountMemory;
  VSWVirtualMachine vswVirtualMachine =
    new VSWVirtualMachine(virtualMachineCOM);
  vmCollection.Add(vswVirtualMachine.Name, vswVirtualMachine);
  return vswVirtualMachine;
}

Deleting a virtual machine is easier than creating one. The COM API DeleteVirtualMachine method simply takes a VMVirtualMachine object and deletes it. For some added value, the wrapper ensures that the virtual machine is in the proper state to be removed. If it is in a state in which it cannot be removed, for instance, if it is paused or running, the method will throw an exception detailing why it could not be removed at that particular time, as shown in Figure 6.

Figure 6 Removing a Virtual Machine (Close Figure 6)
/// <summary>
/// Delete Virtual Machine
/// </summary>
/// <param name="virtualMachine">Virtual Machine to remove</param>
/// <exception cref="VSWVirtualMachineException">Thrown if Virtual Machine
/// cannot be removed in present state</exception>
public void RemoveVirtualMachine(VSWVirtualMachine vswVirtualMachine)
{
   if (vswVirtualMachine.COMVirtualMachine.State ==
           VMVMState.vmVMState_Paused ||
       vswVirtualMachine.COMVirtualMachine.State ==
           VMVMState.vmVMState_Running)
   {
      throw new VSWVirtualMachineException(
          "Cannot Remove Virtual Machine while in a " +
           vswVirtualMachine.State + " state.");
   }
   COMVirtualServer.DeleteVirtualMachine(
      vswVirtualMachine.COMVirtualMachine);
   vmCollection.Remove(vswVirtualMachine.Name);
}

The VSWVirtualServer class contains a Hashtable of .NET-wrapped virtual machine objects. This property allows clients to quickly gain access to the virtual machines registered to the Virtual Server (see Figure 7). Each time the property is accessed, the COM API VirtualMachines property is used to return a collection of the virtual machines. If the virtual machine does not exist in the Hashtable, then a .NET virtual machine object is created and added to the Hashtable. Likewise, if the .NET virtual machine object has been deleted, it will be removed from the list.

Figure 7 Virtual Machine Collection (Close Figure 7)
/// <summary>
/// Hashtable of Virtual Machine where key is virtual machine name
/// </summary>
public Hashtable VMCollection
{
  get {
    StringCollection virtualServerVMs = new StringCollection();
    // add dictionaries that are not already in hashtable
    foreach ( VMVirtualMachine virtualMachineCOM in
              virtualServerCOM.VirtualMachines )
    {
       virtualServerVMs.Add(virtualMachineCOM.Name);
       if (!vmCollection.ContainsKey(virtualMachineCOM.Name))
       {
         vmCollection.Add(virtualMachineCOM.Name,
            new VSWVirtualMachine(virtualMachineCOM));
       }
     }
     // remove virtual machines that no longer exist from hashtable
     foreach (string vmName in vmCollection.Keys)
     {
       if (!virtualServerVMs.Contains(vmName))
       {
              vmCollection.Remove(vmName);
       }
     }
     return vmCollection;
   }
}

The VSWVirtualMachine class provides the .NET wrapper for the COM VMVirtualMachine class. The class is responsible for attaching virtual devices to the virtual machine and controlling state and actions. It also exposes events so that a consuming client can react when virtual machines change state such as changing from "running" to "saved." The most interesting method of this class that I use for the Rapid Test application is the AttachISO method because it is responsible for binding the application to test to the DVD-ROM. The AttachISO method is shown in Figure 8.

Figure 8 Attach an ISO Image to a Virtual Machine (Close Figure 8)
/// <summary>
/// Attach an ISO Image of DVDROM to Virtual Machine
/// </summary>
/// <param name="ISOImagePath">Path to ISO Image to attach</param>
/// <exception cref="VSWVirtualMachineException">Thrown if Virtual
/// Machine
/// ISO could not be attached</exception>
public void AttachISO( string ISOImagePath )
{
   try {
     // attach to first DVDROM
     if (virtualMachineCOM.DVDROMDrives.Count == 0)
       throw new VSWVirtualMachineException(
           "Virtual Server: Cannot attach ISO because there are no
            DVD Drives" );
     VMDVDDrive dvdDrive = null;
     foreach (VMDVDDrive drive in virtualMachineCOM.DVDROMDrives)
     {
        dvdDrive = drive;
        break;
     }
     if ( dvdDrive.Attachment ==
          VMDVDDriveAttachmentType.vmDVDDrive_HostDrive )
     {
        dvdDrive.ReleaseHostDrive();
     }
     else if ( dvdDrive.Attachment ==
               VMDVDDriveAttachmentType.vmDVDDrive_Image )
     {
        dvdDrive.ReleaseImage();
     }
     dvdDrive.AttachImage( ISOImagePath );
   }
   catch(Exception Ex)
   {
      throw new VSWVirtualMachineException(
         "Unable to attach ISO Image", Ex);
   }
}

Rapid Test Windows Form Application

Rapid Test is a Windows Forms application that uses three forms: the main status form, the form to create new virtual machines, and the form that allows remote control of the virtual machines.

Before getting too far into the application, I should talk a little bit about COM security. Ultimately, the Windows Forms application accesses the Virtual Server API, which is a standard COM object. Before any Virtual Server interface is accessed, the COM interface layer is initialized. By default, COM uses the Identification security level, which is lower than the Impersonation security level required as a minimum by Virtual Server. This means that you have to specify the security level explicitly by calling the Win32® API methods CoInitializeEx and CoInitializeSecurity through P/Invoke before doing anything else in the application. Likewise, at application shutdown, you must also call the CoUninitialize method. All of these methods are contained in Ole32.dll. The code in Figure 9 shows how to set the proper security level to access the Virtual Server API. The main form in the Rapid Test application initializes and uninitializes COM security.

Figure 9 COM Security Initialization (Close Figure 9)
/// <summary>
/// Initialize COM Security settings with threading option
/// </summary>
/// <param name="threadType">
/// Desired threading model
/// </param>
public static void Initialize(APARTMENTTYPE threadType)
{
  uint coinit = COINIT_APARTMENTTHREADED;
  // if Multi Threaded Apartment (MTA)
  if (threadType == APARTMENTTYPE.Multi)
     coinit = COINIT_MULTITHREADED;
  // Initializes the COM library for use by the calling thread
  CoInitializeEx(IntPtr.Zero, coinit);
  // use SecurityImpersonation
  CoInitializeSecurity(IntPtr.Zero, 0, IntPtr.Zero, IntPtr.Zero,
    RPC_C_AUTHN_LEVEL_NONE, RPC_C_IMP_LEVEL_IMPERSONATE,
    IntPtr.Zero, 0, IntPtr.Zero );
}

Users are able to create new virtual machines quickly and easily by selecting the "Create New VM" option in the file menu. The form has a simple UI for selecting the type of virtual machine to create and the amount of memory to use (see Figure 10). Virtual Server recommends the amount of memory to use for virtual machines through the SuggestedMaximumMemoryPerVM property, but the user can choose to ignore the recommendation. After the form makes the proper validation checks, a single call to the CreateVirtualMachine method in the VSWVirtualServer class is made to create the virtual machine.

Figure 10 Create New Virtual Machine
Figure 10 Create New Virtual Machine

The Rapid Test main form calls the VSWVirtualServer class to retrieve all of the virtual machines and lists them in the form. Users can also see how long the virtual machine has been running and the average CPU usage. These properties are available for the VMVirtualMachine class in the API. I assumed that users will want to see this type of information on a regular basis, so I implemented a simple timer that is user-configurable and that will regularly refresh the list of virtual machines.

With the list of virtual machines, users are able to perform actions such as start, remove, and save using a context menu. Below the list of virtual machines, a user can select the directory from which an ISO image should be created. If the user chooses to attach an ISO image, the directory is made into the ISO image using a command-line utility called mkisofs.exe that is included in the code download at the link at the top of this article. The CreateISOFile method, shown in Figure 11, in the Utility class is used to automate the creation of the ISO images.

Figure 11 Create ISO Image (Close Figure 11)
/// <summary>
/// Creates an ISO Image for a given directory
/// </summary>
/// <param name="ISOFileName"></param>
/// <param name="directoryToCreate"></param>
/// <returns>name of resulting ISO file</returns>
public static string CreateISOFile(string ISOFileName, string
                                   directoryToCreate)
{
   //create a unique name for the ISO file
   string ISOName = ISOFileName + ".iso";
   // create the ISO image from the selected directory
   Process process = new Process();
   process.StartInfo.FileName = "mkisofs.exe";
   // make joliet file system and call the image "RapidTest"
   process.StartInfo.Arguments = String.Format("-joliet -volid {0}
      -o \"{1}\" \"{2}\"", "RapidTest", ISOName, directoryToCreate);
   process.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
   process.Start();
   process.WaitForExit();
   return ISOName;
}

The process to connect to the virtual machine is fairly simple, but there is a lot going on under the covers. For example, let's say that I have a virtual machine that is currently off. When I go to the virtual machines list and right-click on the Options context menu, I am presented with various choices. If I select the option to turn on the virtual machine, the code will first check to see if I chose to attach an ISO image to the virtual machine. Since I am testing an application, I have chosen to deploy the c:\MyApplications directory. The ISO image is created from the chosen directory and then copied to a location to which the Virtual Server has access. The location is configurable in the application's configuration file.

Once the file is successfully copied, a new form is created that is responsible for controlling the virtual machine remotely. The constructor for the class that creates this form takes a VSWVirtualMachine class as a parameter. This lets the form register for events and track the status of the virtual machine. Keep in mind that the virtual machine has not started yet. The StartVirtualMachine method of the Virtual Machine Display form is overloaded to accept either one or zero parameters. The overload that takes a string parameter then represents the path to an ISO image. If this path is truly a path to an ISO image, the image is attached right before starting the virtual machine. Once all of this has completed, the virtual machine starts and is ready to be controlled remotely.

VMRC ActiveX Control

The Virtual Machine Display form uses the VMRC ActiveX control to enable the user to control the application remotely. The VMRC ActiveX control has many properties and methods that make it ripe for automation. I will touch on some of the key properties, methods, and events that are used to connect to and display a virtual machine remotely.

There are three key pieces of information that are required for the VMRC ActiveX control to connect to a virtual machine. The name or IP address of the Virtual Server is set with the ServerAddress property and the port number of the VRMC is set with the ServerPort property. This is all the information needed to connect to the admin display. The admin display lists all of the virtual machines running on the Virtual Server and allows the user to select which one he wants to connect to. You may turn off the admin display capability in the ActiveX control.

The default port running VMRC is 5900, but it can be changed using the Virtual Server administrative options. You need the name of the virtual machine to connect directly to it, and this is set using the ServerDisplayName property. Once this information is set, calling the Connect method will establish the connection. I chose to subscribe to the events that the control exposes to let me know when the state of the connection changes.

You will notice in the code in Figure 12 that I use a class called VMTask when starting the virtual machine. This class enables you to track the status of long-running tasks using a status bar or any other user notification mechanism. This class comes in handy when virtual machines are coming out of a saved state. Depending on how much memory is allocated to a virtual machine, it could take a while for it to come out of hibernation. The VMTask is used to display how much longer it will take until the virtual machine is ready. Once the task is completed, the connection to the virtual machine is established.

Figure 12 Connect to a Virtual Machine from the VMRC (Close Figure 12)
public void StartVirtualMachine()
{
  vm.vmStateChange +=new VSWVirtualMachineEventHandler(vm_vmStateChange);
  if (vm.State != "Running")
  {
    VMTask task = vm.COMVirtualMachine.Startup();
    UpdateStatusBar(task);
  }

  axVMRCClientControl1.ServerAddress = virtualServerAddress
  axVMRCClientControl1.ServerPort = virtualServerPort
  axVMRCClientControl1.ServerDisplayName = vm.Name;
  axVMRCClientControl1.OnStateChanged += new
     OnStateChangedEventHandler(axVMRCClientControl1_OnConnectingStatus);
  axVMRCClientControl1.Connect();
  axVMRCClientControl1.Visible = true;
}

Conclusion

With relative ease, I have created a Windows Forms application that takes advantage of the extensive Virtual Server COM API to create a virtual machine test environment. I can create virtual machines from scratch and use them for testing in a matter of minutes. I can even deploy applications to those virtual machines by automatically creating and attaching an ISO image. I can do this without ever configuring the network on the virtual machine. This application saves development and testing time by quickly provisioning new machines to test.

Virtual Server 2005 is a great product for app developers and testers. It converts a single server into a virtual test lab to test apps on a variety of operating systems and configurations. The rich COM API and VMRC controls allow developers to automate deployments quickly and to test scenarios in a multitude of ways.

Ben Waldron is a Senior Consultant in Microsoft Consulting Services providing architecture guidance and developing solutions for the Federal government. He is based in Washington D.C. and can be contacted at bwaldron@microsoft.com.


Editor's Note: New Technologies and a New Magazine

Editor's Note
New Technologies and a New Magazine

Hey, remember the Web? Back around 1996 it was big news. People suddenly realized that with just a telephone line and Internet Explorer, they could go shopping without ever leaving the comfort of their home! Over the Web you could buy furniture, art, books, and appliances, all of which would make your home comfortable enough to stay there and do some more online shopping, accelerating the vicious cycle of consumerism. Cool!

Over the years, however, the Internet has become an integrated part of the technology picture. Standalone client programs without Internet connectivity are a rarity today. More and more, it seems like innovation is happening at the fringes rather than at the core. Ideas like "breadcrumb" navigation and those annoying Flash ads that obscure the screen you're trying to read until the "low mortgage prices" cube has finished spinning (before you can find the faint "Close X" button) are just a few examples. But don't be fooled. There's still a lot of new stuff to learn about Web programming.

This month, we are highlighting a couple of these Web and Internet technologies. We're talking about the new GridView component in ASP.NET 2.0. It's a lot like the DataGrid, only better. Carl Franklin discusses peer-to-peer chats and how to design them using the .NET Framework. And Dino Esposito uses his Cutting Edge column to show you how to do partial page refreshes with either ASP.NET 2.0 or 1.x.

We've received a flurry of letters from readers of late asking whether we can provide source code in more languages. As part of our { End Bracket } column, we've asked contributing editor John Robbins to take a look at this request and his solution was to whip up a tool that helps translate C# projects into Visual Basic .NET. To see it, flip to the back page right now!

Back yet? Great. If you're like many of our readers, your job takes you beyond mere development into the world of IT. Perhaps you're a Windows power user, able to punch your way through arcane Control Panel settings and weird registry entries. Perhaps you know how to make a wireless network function properly without getting your building chalked for being an open access point. Who knows, maybe you even have a second career in the offing—driving in the South American racing circuit. (Probably not, but we need to set up the photo that's on this page.)

It's become an MSDN Magazine tradition to produce an annual issue that discusses security. While we were planning this year's version (due out in November), we realized something that in retrospect seems obvious: the people who are interested in security as developers are also looking for solutions for administration. From this simple idea comes a new project. This autumn, we will be publishing a special sister publication, designed for the IT professional—TechNet Magazine.

The inaugural issue will provide gobs of technical information about security in the IT world, as well as other real, hands-on information that you (or your admin friends) can use right away. Just surf on over to http://www.technetmagsubs.com and sign up today. We'll hold the rest of the Editor's Note until you get back.

While you were off signing up for TechNet Magazine, we took the liberty of wheeling our new MSDN Magazine race car into the Editor's Note. We have been introducing MSDN Magazine to several countries worldwide, including a Portuguese edition in Brazil. So how did the Brazilian publishers choose to advertise the magazine? A t-shirt? A squishy ball? Okay, let's stop the "list of novelty items" charade—we all know the answer is the semi-official MSDN Magazine race car! At press time, we have no news of how the car actually finished, so we won't make any painful quips about how we crossed the finish line first with great content or how we displayed the yellow caution flag when covering beta software. All in all, that's probably for the better.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Uwa Agbonile, Mike Ammerlaan, Keith Ballinger, Jay Bazuzi, Rebecca Dias, Mike Fitzmaurice, Kirill Gavrylyuk, Matthew Gibbs, Chris Hays, Polita Huff, Bryan Keller, John Koropchak, Ronald Laeremans, Rangan Majumder, Constantin Mihai, Erik Olson, Matt Powell, Kurt Schmucker, Maura Van Der Linden, and Reid Wilkes.

Active Directory, ActiveX, BizTalk, DirectX, FrontPage, JScript, Microsoft, Microsoft Press, MSDN, SharePoint, Visual Basic, Visual C++, Visual SourceSafe, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Server is a trademark of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Geography for Developers
Components by Subscription
Inspect Web Sites
UI Components
Enhance Windows Server
Themed Components for Windows Forms Apps
TCP/IP Networking Libraries
Editing SQL Queries
Is Your Web Site Down?
The Bookshelf

Geography for Developers

Blue Marble Geographics has released its developer toolkit, GeoTransform 5.0. GeoTransform is a DLL for programmers using Visual Basic®, C++, Delphi, PowerBuilder, C++ Builder, and other Windows®-based development tools. The toolkit allows you to embed image reprojection and tiling in applications with just a few function calls.

GeoTransform creates an image map from a source image, geo-referencing information, and destination image parameters, and creates ESRI world and MapInfo TAB referencing files.

Blue Marble includes a coordinate conversion parameter database that contains the most common coordinate systems used throughout the world. You can even define your own custom units, ellipsoids, datums, and coordinate systems. The toolkit will automatically cut projected images into tiles.

http://www.bluemarblegeo.com/products/geotransform.php

Components by Subscription

ComponentOne has released ComponentOne Studio Enterprise 2004 2.0. Studio Enterprise is a component subscription targeting all runtime environments including the Microsoft® .NET Framework, ASP.NET, Mobile, and ActiveX®, and all application layers including data as well as presentation, reporting, and transformation. This integrated framework delivers over 100 grid, reporting, charting, data, UI, and e-commerce components. Some of the new tools in this release include the following:

XLS for .NET allows users to read and write Microsoft Excel files from their applications. Users can load a workbook, manipulate sheets as if they were grid controls (composed of rows, columns, and cells), and then save the workbook to an XLS file.

WebGrid for ASP.NET lets users create reusable layouts for their WebGrid, and save or load them at design time or run time. Users can also scroll the grid both horizontally and vertically, and lock any row or column.

Menus and Toolbars for .NET is a new Docking Tab control that was added to deliver tabs in the style of Visual Studio® .NET and docking or floating windows.

PDF for .NET supports TIFF images, plus improved image clipping and image rendering.

Preview for .NET encompasses print document features that include new export formats consisting of RTF and Excel.

Finally, Spell for .NET provides improved speed when working with custom dictionaries.

http://www.componentone.com

Inspect Web Sites

Paessler Site Inspector, released by Paessler GmbH, allows you to discover a variety of data about a Web site, which can help Webmasters and Web developers in debugging and analyzing Web sites or Web pages. Site Inspector adds a toolbar and several new menu items to the context menu of Internet Explorer. This provides a number of analysis functions that give you a look behind the technology and structure of a Web page, which allows you to uncover background information about the Web site being viewed.

Site Inspector contains functions to confirm links and HTML validity of a page as well as performance checks. The owner of the domain or the Web site's current global traffic ranking can be accessed from inside Internet Explorer. Site Inspector features include lists of all forms and images available in the viewed Web page, including usually hidden information such as the size of the image files, URLs, and dimensions.

You can also generate lists of all scripts, hyperlink URLs, and stylesheets of the viewed Web page. Site Inspector displays http-header information, including status code, server information, content type, and shows structured source based on the Document Object Model. HTML elements are highlighted by adding colored borders around each object.

http://www.paessler.com

UI Components

Dacris Software has released NetXP 3.0, an overhaul of its NetXP component suite, which developers can use to enhance the way their applications interact with users. NetXP provides a library of user interface controls and components. This update consists of fundamental changes to the command-bar controls and improved renderer support, all designed to coordinate visually with Microsoft Office 2003.

The changes in NetXP 3.0 include a command-bar dock control with MDI support, and a command bar manager component that can save and load layouts using XML, binary format, or the system registry. There's also a status bar control and progress bar panel, and a list bar control (called an outlook bar) with Office 2003, Office XP, and Windows renderers.

In addition, you'll find a desktop alert form, a dropdown treeview control, and a reusable, inheritable pop-up form, providing the functionality of a pop-up window. The command-bar layout now supports wrapping, and the renderer framework has been redesigned.

NetXP added tabbed groups support for docking (for tabbed MDI), plus the ability to change colors by setting a color table for renderers. The command bar supports embedded controls, including a special flat combobox. Custom color tables are now supported by the Task Pane.

http://www.dacris.com/netxp

Enhance Windows Server

Jetro Platforms Ltd has launched Jetro CockpIT 3.5, an update of its enterprise management software for Windows Server™ 2003. Version 3.5 features the Universal Client for Windows CE, allowing enterprises that use Windows CE to implement server-based computing without the complications of client-side installations.

Other new features in Jetro CockpIT version 3.5, such as the Advanced Secure Connector and Seamless File Transfer, make server-based computing more secure. These combine with existing Jetro CockpIT features such as seamless delivery of server-based applications using RDP, performance-based load balancing, universal print driver, and print job compression to make Windows Server 2003 a robust enterprise IT solution.

http://www.jp-inc.com

Themed Components for Windows Forms Apps

DevComponents LLC is providing DotNetBar 3.7, a framework for Visual Studio .NET that provides components with the user experience of Office 2003, Office XP, Office 2000, and Windows XP. The DotNetBar Suite includes controls such as fully customizable menus and toolbars, dockable auto-hide windows, a navigation bar control, a side bar control, a tab strip control, a balloon and balloon-tip control, and an alert control.

This release includes a Navigation Panel control which allows content presentation based on grouping similar tasks, and provides access to most common task commands and views. Navigation Panel blends into the user environment by automatically adjusting its display based on the color scheme selected on the user's system.

The Panel control also has customization options available, so developers can specify gradient background colors and alpha-blended background images with full control over text display.

http://www.devcomponents.com

TCP/IP Networking Libraries

Catalyst Development Corporation has published SocketTools Library Editions 4.0, a collection of TCP/IP networking libraries for Windows. New features include lower resource utilization and no external dependencies, an expanded API with 18 libraries and over 800 functions, a new library which provides an interface for the IMAP4 protocol and full compatibility with Visual Studio .NET, including Visual Basic .NET and C#.

The update includes support for both synchronous and asynchronous network connections and improved support for multithreaded applications. Additional features include support for Secure Sockets Layer (SSL) and TLS security protocols with 128-bit encryption, support for both implicit and explicit SSL and TLS connections, support for client certificates when required by the remote host, support for using certificate files and the Windows certificate store, and creation of both secure client and server applications.

http://www.catalyst.com

Editing SQL Queries

msSQLed has published a query editor called msSQLed for Microsoft SQL Server™ 2000 and SQL Server 2000 Desktop Engine. The editor provides coding help with pick lists for tables, views, procedures, functions, columns, databases, variables/parameters, as well as dbcc statements.

msSQLed also provides filtering and grouping on the client side of resultsets, the ability to get context-sensitive information about database objects, and object notations that can be viewed by multiple users.

The query manager stores queries with optional parameters and connections so the queries can be opened against the database they were created on. The program also includes integration with Visual SourceSafe®, SQL Server Reporting Services integration, and other features that let a user drill down through retrieved results to get additional information.

msSQLed will show image columns as images in resultsets and can also convert and print results in a variety of different formats. It will execute SQL statements against multiple connections simultaneously, providing the ability to execute a DML statement or perform data retrieval across multiple connections.

http://www.mssqled.com

Is Your Web Site Down?

ExclamationSoft Corporation has released WebWatchBot 2.0, its Web site availability and performance-monitoring software. WebWatchBot 2.0 alerts designated personnel of errors, poor response time, and other customizable conditions using e-mail, on screen display, cell phone, or pager.

New features in version 2.0 include the ability to run as a Windows Service for 24x7 monitoring without user intervention, an enhanced user interface, real-time charting, improved scheduling and reporting functionality, and integration with Microsoft Access for data manipulation and management.

WebWatchBot 2.0 can be installed on any Windows-based PC and configured to monitor multiple URLs for a variety of conditions.

http://www.exclamationsoft.com

The Bookshelf

Charles River Media has published Introduction to Computer Graphics, Second Edition by H. Bungartz, M. Griebel, and C. Zenger. The authors, all professors of com-puter science in Germany, begin the guide with the basics of gen-erating images from scratch on a computer screen, discussing coordinate systems and transformations, rudimentary shapes, and the representation of grays and colors. The book moves into ways to model and represent a three-dimensional figure, covering mathematical models, the vef graph, Euler operators, Bézier curves, as well as three-dimensional clipping, lighting effects and shading, visualization, and ray tracing.

The final chapter addresses specific applications of techniques such as mapping, stereography, and image processing, explaining their use in representing real objects as well as in virtual reality programming.

http://www.charlesriver.com

Mike Murach & Associates has released Murach's C#, which shows readers how to use C# and Visual Studio to design, code, and test Windows-based applications. The book teaches essential C# programming skills, such as event wiring, using loops and if structures, handling dates and strings, and using arrays and collections. It also teaches how to use structured exception handling and how to validate user input to prevent common exceptions.

This book treats object-oriented programming in great depth, starting with advice on how to write simple business classes and how to enhance those classes with profes-sional features such as class events, property validation, and overloaded operators.

The book explains inheritance, polymorphism, and interfaces. It contains a five-chapter section on database processing using ADO.NET, which starts by showing how to create two-tiered applications that handle data using typed DataSets with bound or unbound controls.

A dozen model applications, including one for an order entry system, help developers see how all the code works together. Exercises at the end of each chapter provide practice for building new skills.

http://www.murach.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Bulletin Board: News from Online Communities

Bulletin Board
News from Online Communities

This is the first installment of a new page in MSDN Magazine, which we're calling Bulletin Board. Like a physical bulletin board, this is a place where members of the development community will be able to find news and information about what's going on in the "real world."

This month's column introduces you to Code Project, a vast site of resources for Web development. Code Project is a member of the Microsoft Code Wise Community, a source for tips, code samples, news, and advice from experts outside of Microsoft.

CodeProject.com has one of the largest collections of Web development articles and tutorials available online. It contains more than 800 Web development articles, about half of which cover ASP.NET. With more than one million members, Code Project has become a valuable resource for Web developers (see http://www.codeproject.com/aspnet). Aside from the storehouse of articles (with hundreds of new articles posted each month), Code Project has tutorials for beginners, discussion boards on various topics, a newsletter, industry news, and an Internet lounge. Some parts of the site are available only to members.

Code Project is a great place to seek out Web resources. Whether you're just getting started with ASP or you're moving from ASP to ASP.NET, you'll find plenty of articles.For example:

  • "Beginner's Introduction to ASP.NET" by Sudhir Mangla
  • "Overview of Caching in ASP.NET" by Smitha Vijayan

If you're looking to further your creativity and roll your own controls, explore the articles on custom server controls:

  • "Eventing Within Composite Web Custom Controls" by Nick Pirocanac
  • "Heathmans XmlTransformer Control" by Paul Watson

If you want a little more speed, check out the ATL Server articles. If you're moving from ISAPI to .NET, you'll want to learn more about HttpHandlers and HttpModules:

  • "ATL Server—Web Application/Web Service" by Leon Finker
  • "Using HTTP Modules To Combat Leeching" by KingLeon

Data access and data binding are critical requirements in ASP.NET development. The following articles provide an assortment of tips and techniques to help tie your data to your design:

  • "ASP.NET Query Control" by Mike Ellison
  • "Paging of Large Resultsets in ASP.NET" by Jasmin Muharemovic
  • "Designing and Implementing a Data Access Tier" by Paul Abraham

Code Project also offers two Web development Message Boards, one on ASP.NET and one on general Web development, including HTML, JavaScript, classic ASP, and other Web-building tools. They also award prizes to the top articles submitted in each of several categories each month, so if you have some code, tips, or words of wisdom you can share them with the rest of the community. Details can be found at Monthly competition - Rules and Prize list. If you prefer to consume your information via RSS, subscribe to the feed at http://www.codeproject.com/webservices/articlerss.aspx?cat=4.

Send comments for Bulletin Board to bboard@microsoft.com.


Web Q&amp;A: Page Ready State, DataGrid Row Deletion, and More

Web Q&A
Page Ready State, DataGrid Row Deletion, and More
Edited by Nancy Michell

Q JScript® seems to automatically add serializable to classes. When I compile the class shown in Figure 1, it becomes serializable although it is not declared to be so. Why is this?

Figure 1 Serializable (Close Figure 1)
//@cc_on
//@set @debug(off)

package tempuriorg.Test
{
  public class JSTest
  {
    private var aField : int;

    public final function get a() : int {
      return this.aField;
    }
    public final function set a(value : int) {
      this.aField = value;
    }
  }
}

A You're right, JScript automatically adds serializable to classes. It's by design. In JScript, unlike in C#, a class is serializable by default. If you don't want your class to be serializable, then do this:

public System.NotSerialized class JSTest {...}

Q When I use window.open to open a new HTML page and then try to access the frameset defined in the new page, the frameset may be unavailable right after window.open is called. As a result, I get a script error at that time. The problem does not occur if setTimeout is used to access the frames collection at a later time:

<html><Body>
<script language='javascript'>
function test()
{
  var a;
  a = window.open('main1.htm', '');
  a.frames[0].location = 'frame3.htm';
}
</script>
<a href="javascript:void(0);" onclick="test();">
TEST</a>
</Body></html>

A This code is vulnerable to a timing problem because the script code attempts to access the object model of the page in the new window immediately after the window.open call. That forms collection doesn't exist until main1.htm has been parsed, so a script error could occur. The problem can be reproduced by replacing main1.htm with an ASPX page that has a built-in delay.

It would be better to check readyState or wait for an event so you know the page is ready to script against. Figure 2 shows one way.

Figure 2 Is the Page Ready? (Close Figure 2)
<script language='javascript'>
var a;

function test() {
  a = window.open('main1.htm', '');
  a.document.attachEvent("onreadystatechange", changeFrame0 );
  changeFrame0();
}

function changeFrame0() {
  var state = a.document.readyState;
  if (state == "complete") {
    a.frames[0].location = 'frame3.htm';
    // debugOut.insertAdjacentText("BeforeEnd", "Complete\n");
    a.document.detachEvent(changeFrame0);
  } else {
    // debugOut.insertAdjacentText("BeforeEnd", "ReadyState "+state+"\n");
  }
}
</script>

Q I have a DataGrid that looks like this:

<asp:datagrid id="dgHome" runat="server" Width="248px"
  CssClass="taskItem" AutoGenerateColumns="False">
<Columns>
  <asp:BoundColumn DataField="task_Text" HeaderText="Task">
    <HeaderStyle CssClass="dgHead" BackColor="#FFFFCC"></HeaderStyle>
  </asp:BoundColumn>
  <asp:TemplateColumn HeaderText="Done">
    <HeaderStyle CssClass="dgHead" BackColor="#FFFFCC"></HeaderStyle>
    <ItemStyle Width="15%"></ItemStyle>
    <ItemTemplate>
         <asp:CheckBox id="Checkbox1" runat="server" />
•••
</asp:datagrid>

When a user selects the checkbox in the row and clicks a button, I want to delete the node in the source XML file to which the grid is bound. What is the best way to accomplish this in C#?

A On the postback, you can easily delete the row by importing the XML file to a DataSet and then deleting the row from the table in the DataSet. You can then rebind the grid to the updated data source and serialize the remaining rows to the source file.

Q Are there any new data-bound controls in Visual Studio® 2005? In particular I am interested in a full-featured grid control. I have heard about the GridView but need more info. Will there be an XSL designer for the XML control or will you need to link to existing XSL? What about a data-bound version of the XML control?

A There are tons of new controls including Menu, TreeView, Wizard, DynamicImage, FileUpload, Login, LoginView, LoginStatus, PasswordRecovery, ChangePassword, CreateUserWizard, SiteMap, WebParts, just to mention a few. In this issue of MSDN®Magazine, Dino Esposito devotes a whole article to the GridView.

A new version of the XML Designer is scheduled to be shipped with Visual Studio 2005 (not specific to ASP.NET) which is supposed to support XSLT debugging and many new features. If you're looking for a full-featured grid control, GridView is what you want. Other controls include: GridView (the successor to DataGrid), DataList, Repeater, DetailsView (which gets one record at a time and is displayed in a standard layout), and FormView (which gets one record at a time and uses templates).

For data binding, you can use data source controls (see Figure 3). They perform the actual data read/write and expose an interface that data-bound controls can bind to.

Figure 3 Data Source Controls (Close Figure 3)

Control | Description
SqlDataSource | Performs SQL queries
AccessDataSource | Access-specific
ObjectDataSource | Middle-tier object; invokes methods
XmlDataSource | Reads XML; exposes as a hierarchy
DataSetDataSource | Reads XML; exposes as a DataSet
SiteMapDataSource | Used with navigation controls

To implement a data-bound version of the XML control you don't need a special control. You can use the DataSetDataSource control, which reads an XML file and exposes it as a DataSet; you can then bind any data control to it.

Q How can I log HTTP header sizes for all incoming Web requests?

A It depends on what you're trying to log. Do you want one number that represents the total size of all headers? Do you want multiple numbers per request with one for each header? If you want the former, then in an ISAPI filter just call GetHeader in PREPROC_HEADERS for ALL_RAW. Call it with a zero size buffer and it will fail with ERROR_INSUFFICIENT_BUFFER and return the size to you (the size will represent the names and values for all headers, as well as the delimiters between them). If you want multiple numbers, then do the same thing, but pass a buffer sufficient for the call to succeed and then parse it for each header.

Q I have a hidden field containing some XML data. When the user clicks a button, I want to display this data in a new window with the default XML formatting provided by Microsoft® Internet Explorer. With the following code, I just get the text, but no formatting. I see the XML in the source of the document, so how can I get it to display properly in the new window?

function DisplayXML()
{
newWin = window.open("","");
if( newWin != null )
{
    newWin.document.write(
        document.all["subsXMLDiv"].innerHTML);
}
}

A Try something like the code in Figure 4, which explicitly uses the XSL that Internet Explorer uses to display XML.

Figure 4 Using the Internet Explorer XML Stylesheet (Close Figure 4)
function DisplayXML()
{
    newWin = window.open("","");
    if( newWin != null )
    {
        var oStyle = new ActiveXObject( "Msxml2.DOMDocument.3.0" );
        var oXmlDoc = new ActiveXObject( "Msxml2.DOMDocument.3.0" );
        oStyle.async = true;
        oXmlDoc.loadXML( subsXMLDiv.innerText );
        //load the stylesheet that Internet Explorer uses...
        oStyle.load( "res://msxml3.dll/defaultss.xsl" );
        newWin.document.write( oXmlDoc.transformNode( oStyle ) );
    }
}

Q I have hundreds of classic ASP applications joined together in a portal fashion that use an older custom ISAPI filter for authentication. It's a familiar scenario—hundreds of virtual apps running under a single root and sharing a common authentication filter.

I need to move the applications over to use ASP.NET with forms authentication. I would like to be able to use the ASP.NET session created on login on the old ASP site to keep users from having to log in twice based on the app used in the portal.

So my question is, can I use the ASP.NET session (and session cookie) or do I need to create a new custom authentication provider that's shared across ASP.NET and ASP apps?

A You could use a custom ISAPI filter that would check for the custom authentication cookie. If the cookie is there, it lets the request go through; if not, it sends the user to a logon page. That will keep you from having to modify any of the other apps. Create an HttpModule in .NET (see How To Create an ASP.NET HTTP Module Using Visual C# .NET), set ASP.NET authentication to "None", and do your own cookie encryption and session timeouts in the module. See "Implement Forms-Based Authentication in Your ASP.NET Application by Using C# .NET".

Q I'm having trouble with Response.Redirect. Here's my code:

Try
    ...
    Response.Redirect("somepage.aspx")
Catch ex As Exception
    Response.Redirect("error.aspx?ErrMsg=" & ex.Message)
EndTry
The code goes through the first Response.Redirect line, then hesitates for a second or two, then goes to the Catch block and generates an error that reads "Thread was aborted". As soon as I remove the second Response.Redirect in the Catch area, the error goes away.

A By design, Response.Redirect generates a ThreadAbortException. If your redirect is not at the end of an execution flow (more code could be executed after), you should return right away.

So the correct pattern should be:

Try
    targetURL = "somepage.aspx"
    •••
Catch ex As Exception
   Session("errormesage") = ex.message
   targetURL = "errorpage.aspx"
Finally
   ' clean up
End Try
Response.Redirect(targetURL)
That would be the best solution, if there is such a thing in this situation. If you have any code after the Response.Redirect, though, you'll still have to put in a return statement. You could also specify a second Boolean parameter to Response.Redirect:
Response.Redirect(targetURL, False)
This prevents Redirect from calling Response.End and throwing an exception.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Bret Bentzinger, Leon Braginski, Jonah Burke, Brian Combs, Silvano Coriani, Steve Dunker, Ravi Gopinath, Geoff Gray, Anand Hegde, Michael Hestness (Volt), Wade Hilmo, Zhisheng Huang, Polita Huff, Duy Le, Drew Leaumont, Igor Leybovich, Eric Lippert, Carlos Aguilar Mares, Richie Meyer, Matt Neerincx, Mike Pope, Angel Saenz-Badillos, Derek R. Smith, Charles Torre, Mark Townsend, Stefan Waldenburger, Michael Whalen, Simon Wu, and Jinsong Yu.


Data Points: Migrating from ADO to ADO.NET, Part 2

Data Points
Migrating from ADO to ADO.NET, Part 2
John Papa

Code download available at: DataPoints0408.exe (111KB)
Contents
Recordset Divergence
Firehose Cursors
Data-Sensitive Cursors
Server-Side Cursors and Concurrency
Batch Updates
Disconnected Rowsets

As you move forward with your use of ADO.NET, you'll need to know how to approach situations that you previously learned to handle in ADO and now have to tackle with ADO.NET. Just as n-tiered solutions developed using Visual Basic®, C++, and ASP often rely on ADO for their data access needs, Windows® Forms, Web Forms, and Web services rely on ADO.NET. Last month's Data Points column discussed how to tackle several data access scenarios using ADO.NET from the perspective of developing with traditional ADO (see Data Points: Migrating from ADO to ADO.NET). Some of these topics include persisting rowsets to XML, dealing with forward-only firehose cursors, and the multiple ways that Command objects can be executed.

In this month's column I will continue to discuss development situations using ADO.NET and how they are handled with traditional ADO techniques. I will start by discussing situations in which forward-only, static, keyset, and dynamic cursors of classic ADO were commonly used. I will also discuss how concurrency issues are handled and how disconnected rowsets have evolved from ADO to ADO.NET. Then, I will demonstrate how to take code that handles batch updates in traditional ADO and translate that to work with ADO.NET using the DataAdapter and its four command objects.

Recordset Divergence

Most of the ADO Recordset's features are split into three key objects in ADO.NET: the DataReader, the DataSet, and the DataAdapter (see Figure 1).

Figure 1 The ADO Recordset
Figure 1 The ADO Recordset

The ADO.NET DataReader object is designed to be a server-side, forward-only, read-only cursor. The ADO.NET DataSet object is a disconnected storage tool for rowsets. It stores records without holding a connection to a data source and, in fact, it does not care from what data source its rowsets are derived. The DataSet is a binary object when stored in memory, but it can also be serialized easily to and from XML. It is similar to the ADO Recordset object with its CursorType set to adOpenStatic and CursorLocation set to adUseClient, when it has been disconnected from its associated Connection object (via the Recordset's ActiveConnection property). The ADO.NET DataAdapter object is a bridge between the connection and DataSet objects. It can load a DataSet from a data source using a connection and it can update a data source with the changes stored in a DataSet. The behavior of the ADO Recordset depends on its property settings (including the CursorType and CursorLocation properties). In ADO.NET there are separate objects built to handle these specific situations instead of requiring one object to fit all scenarios.

Firehose Cursors

Traditional ADO exposes four different types of cursors that change how the ADO Recordset object functions. An ADO Recordset object behaves very differently depending on how its CursorType property is set. For example, by setting the CursorType to adOpenForwardOnly, the Recordset remains connected to its data source and must be traversed in a forward-only direction. However, when you set the CursorType property to adOpenDynamic, the Recordset can be traversed forward or backward, or you can even jump the cursor to a particular row. Through its CursorType and CursorLocation properties, the ADO Recordset object takes the approach of wrapping many solutions into a single object. ADO.NET takes a different approach where it has distinct objects and methods that are designed to handle specific situations.

In classic ADO forward-only, read-only cursors are implemented by setting the CursorType to adOpenForwardOnly and the CursorLocation to adUseServer (which are also the default settings). This makes the Recordset object take the form of a forward-only, server-side cursor. The MoveNext method repositions the Recordset to the next row and the MovePrevious method is not allowed at all. You can, however, invoke the MoveFirst method on a Recordset. It is a bit misleading, however, as it does not reposition itself to the beginning of the current rowset. Rather, it invokes the original SQL statement and refills the Recordset from scratch, thus moving to the first record again.

This can easily be seen by opening the SQL Profiler tool and watching the SQL execute every time the MoveFirst method is executed on a traditional ADO Recordset with a CursorType of adOpenForwardOnly. This type of cursor is commonly used in applications where thousands (or more) rows need to be traversed one at a time or when a smaller rowset is required but only needs to be traversed once, perhaps to be loaded into a pick list:

'-- Forward-only Firehose Cursor in ASP and ADO
Set oRs.ActiveConnection = oCn
oRs.LockType = adLockReadOnly
oRs.CursorType = adOpenForwardOnly
oRs.CursorLocation = adUseServer
oRs.Open sSQL

The closest equivalent to this type of traditional ADO firehose cursor is the ADO.NET DataReader object. Just like the traditional forward-only ADO Recordset, the DataReader remains connected to its data source while it is open and can be traversed only in a forward direction. However, there are differences. One is that individual DataReader types are written specifically for a data provider such as SQL Server™ (the SqlDataReader class) or an ODBC data source (the OdbcDataReader class). The ADO.NET DataReader object is very efficient as it is built specifically for the purpose of implementing forward-only, read-only cursors. The classic ADO forward-only cursor is implemented through the same Recordset object as a disconnected Recordset or even as a data source-sensitive Recordset. The DataReader is designed solely to be a lean firehose cursor.

' ASP.NET and ADO.NET in Visual Basic .NET
Dim oDr As SqlDataReader = oCmd.ExecuteReader()

// ASP.NET and ADO.NET in C#
SqlDataReader oDr = oCmd.ExecuteReader();

For more information on the ADO.NET DataReader, see the June 2004 Data Points column.

Data-Sensitive Cursors

Forward-only cursors in traditional ADO are now handled by the DataReader object. But what happened to the keyset and dynamic server-side cursors (CursorType = adOpenKeySet and CursorType = adOpenDynamic) that are sensitive to changes in the underlying database? The current versions of ADO.NET do not expose a multidirectional, scrollable, updateable, server-side cursor. However, ADO.NET does provide ways around these types of server-side cursors since there are other recommended techniques that you should consider.

Outside of using the DataSet combined with the DataAdapter to retrieve and update data, there are some good alternatives to using scrollable server-side cursors on the client. For example, you could use stored procedures, which are very efficient at running server-side SQL processing. You could also retrieve the data through a server-side forward-only cursor via a DataReader and then make separate updates via command objects. However, often there are more efficient ways to make data modifications than using updateable server-side cursors.

One of the issues with server-side scrollable cursors in an n-tiered environment is that they require state to be held on the server. Therefore, if an application used a server-side scrollable cursor in the middle tier, the client tier would need to maintain a connection to the business tier where the scrollable cursor resides. Thus, the business objects might linger for the life of the client application's screen that uses the scrollable cursor as well. These scalability issues are well documented, but, of course, there are situations in which server-side cursors are valuable, such as when the need for a forward-only firehose cursor exists. For example, server-side cursors have been put to work when an application wants to be made aware of data concurrency issues.

Using traditional ADO, a Recordset could be opened using a dynamic cursor which would allow the Recordset to see all inserts, changes, and deletions by other users. This type of server-side cursor is sensitive to changes made by other users and can be used in applications to take a proactive approach when concurrency issues arise. For example, dynamic cursors in traditional ADO were often used when an application was about to save changes to the database but wanted to make sure it knew if anyone else made changes to the same record first. When a user changed a value of a record that was in a dynamic Recordset, the value was automatically updated in the server-side dynamic Recordset:

'-- Dynamic Cursor in ASP and ADO
Set oRs.ActiveConnection = oCn
oRs.LockType = adLockOptimistic
oRs.CursorType = adOpenDynamic
oRs.CursorLocation = adUseServer
oRs.Open sSQL

Server-Side Cursors and Concurrency

Concurrency problems are valid and common problems in many enterprise applications; however, the cost of using server-side cursors to accomplish this can be too high. So what alternatives exist in ADO.NET to deal with concurrency issues? A common technique is to allow the application to submit the changes to the database and have it throw a special type of exception when it sees a concurrency issue.

There is a special type of exception called a DBConcurrencyException which derives from the base Exception class. Because the DataSet stores the original and proposed values of each column in a row, it knows how to compare the values to the database to look for concurrency issues automatically. For example, assume that a user changes the value of the firstname field from Lloyd to Lamar and submits the changes to the database. The changes, which are stored in a DataSet, are passed on to the database via the DataAdapter's Update method. Before the data is actually saved, if the name in the database is no longer Lloyd but is now Lorenzo, a DBConcurrencyException exception will be thrown. This exception can be caught and handled appropriately in whatever way suits your application's needs best (you could, for example, enforce a "last in wins" rule, giving the user the option to cancel the changes, override, or force the update anyway, or some other technique).

In the code in Figure 2 you'll notice a sample method that accepts a DataSet and submits the changes in the DataSet to the database via a DataAdapter's Update method. (The stored procedures and the code from this figure can be downloaded from the MSDN®Magazine Web site.) If a concurrency issue is detected, the DBConcurrencyException exception will be thrown and caught by the specific catch block. At that point, the transaction could be rolled back and then the exception could be rethrown to eventually be caught by the client application so that the user could be notified and asked how they want to deal with it. Also, consider the order of the catch blocks carefully. For example, if the generic catch block appeared first, then it would have trapped the concurrency issue. A key to exception handling is to make sure that the catch blocks for the more specific exceptions appear before other catch blocks so that they are caught first.

Figure 2 Trapping Concurrency Issues in ADO.NET (Close Figure 2)
' Visual Basic .NET
Public Function SaveData(ByVal oDs As DataSet) As DataSet
    Dim sMethodName As String = "[public void SaveData(DataSet oDs)]"

    '==========================================================
    '-- Establish local variables
    '==========================================================
    Dim sProcName As String
    Dim sConnString As String = "Server=(local);Database=Northwind;" & _
        "Integrated Security=SSPI"
    Dim oDa As SqlDataAdapter = New SqlDataAdapter
    Dim oTrn As SqlTransaction = Nothing
    Dim oCn As SqlConnection = Nothing
    Dim oInsCmd As SqlCommand = Nothing
    Dim oUpdCmd As SqlCommand = Nothing
    Dim oDelCmd As SqlCommand = Nothing

    Try
        '==========================================================
        '-- Set up the Connection
        '==========================================================
        oCn = New SqlConnection(sConnString)

        '==========================================================
        '-- Open the Connection and create the Transaction
        '==========================================================
        oCn.Open()
        oTrn = oCn.BeginTransaction()

        '==========================================================
        '-- Set up the INSERT Command
        '==========================================================
        sProcName = "prInsert_Order"
        oInsCmd = New SqlCommand(sProcName, oCn, oTrn)
        oInsCmd.CommandType = CommandType.StoredProcedure
        oInsCmd.Parameters.Add(New SqlParameter("@sCustomerID", _
            SqlDbType.NChar, 5, "CustomerID"))
        oInsCmd.Parameters.Add(New SqlParameter("@dtOrderDate", _
            SqlDbType.DateTime, 8, "OrderDate"))
        oInsCmd.Parameters.Add(New SqlParameter("@sShipCity", _
            SqlDbType.NVarChar, 30, "ShipCity"))
        oInsCmd.Parameters.Add(New SqlParameter("@sShipCountry", _
            SqlDbType.NVarChar, 30, "ShipCountry"))
        oDa.InsertCommand = oInsCmd

        '==========================================================
        '-- Set up the UPDATE Command
        '==========================================================
        sProcName = "prUpdate_Order"
        oUpdCmd = New SqlCommand(sProcName, oCn, oTrn)
        oUpdCmd.CommandType = CommandType.StoredProcedure
        oUpdCmd.Parameters.Add(New SqlParameter("@nOrderID", _
            SqlDbType.Int, 4, "OrderID"))
        oUpdCmd.Parameters.Add(New SqlParameter("@dtOrderDate", _
            SqlDbType.DateTime, 8, "OrderDate"))
        oUpdCmd.Parameters.Add(New SqlParameter("@sShipCity", _
            SqlDbType.NVarChar, 30, "ShipCity"))
        oUpdCmd.Parameters.Add(New SqlParameter("@sShipCountry", _
            SqlDbType.NVarChar, 30, "ShipCountry"))
        oDa.UpdateCommand = oUpdCmd

        '==========================================================
        '-- Set up the DELETE Command
        '==========================================================
        sProcName = "prDelete_Order"
        oDelCmd = New SqlCommand(sProcName, oCn, oTrn)
        oDelCmd.CommandType = CommandType.StoredProcedure
        oDelCmd.Parameters.Add(New SqlParameter("@nOrderID", _
            SqlDbType.Int, 4, "OrderID"))
        oDa.DeleteCommand = oDelCmd

        '==========================================================
        '-- Save all changes to the database
        '==========================================================
        oDa.Update(oDs.Tables("Orders"))

        oTrn.Commit()
        oCn.Close()

    Catch exDBConcurrency As DBConcurrencyException
        '=======================================================
        '-- Roll back the transaction
        '=======================================================
        oTrn.Rollback()

        '--------------------------------------------------------
        '-- May want to rethrow the Exception at this point.
        '-- This depends on how you want to handle the concurrency
        '-- issue.
        '--------------------------------------------------------
        '-- Throw exDBConcurrency
    Catch ex As Exception

        '==========================================================
        '-- Roll back the transaction
        '==========================================================
        oTrn.Rollback()

        '--------------------------------------------------------
        '-- Rethrow the Exception
        '--------------------------------------------------------
        Throw
    Finally
        oInsCmd.Dispose()
        oUpdCmd.Dispose()
        oDelCmd.Dispose()
        oDa.Dispose()
        oTrn.Dispose()
        oCn.Dispose()
    End Try
    oCn.Close()
    Return oDs
End Function

// C#
public DataSet SaveData(DataSet oDs)
{
    string sMethodName = "[public void SaveData(DataSet oDs)]";

    //==========================================================
    //-- Establish local variables
    //==========================================================
    string sProcName;
    string sConnString = "Server=(local);Database=Northwind;Integrated" +
                          "Security=SSPI";
    SqlDataAdapter oDa = new SqlDataAdapter();
    SqlTransaction oTrn = null;
    SqlConnection oCn = null;
    SqlCommand oInsCmd = null;
    SqlCommand oUpdCmd = null;
    SqlCommand oDelCmd = null;

    try
    {
        //==========================================================
        //-- Set up the Connection
        //==========================================================
        oCn = new SqlConnection(sConnString);

        //==========================================================
        //-- Open the Connection and create the Transaction
        //==========================================================
        oCn.Open();
        oTrn = oCn.BeginTransaction();

        //==========================================================
        //-- Set up the INSERT Command
        //==========================================================
        sProcName = "prInsert_Order";
        oInsCmd = new SqlCommand(sProcName, oCn, oTrn);
        oInsCmd.CommandType = CommandType.StoredProcedure;
        oInsCmd.Parameters.Add(new SqlParameter("@sCustomerID",
                               SqlDbType.NChar, 5, "CustomerID"));
        oInsCmd.Parameters.Add(new SqlParameter("@dtOrderDate",
                               SqlDbType.DateTime, 8,"OrderDate"));
        oInsCmd.Parameters.Add(new SqlParameter("@sShipCity",
                               SqlDbType.NVarChar, 30, "ShipCity"));
        oInsCmd.Parameters.Add(new SqlParameter("@sShipCountry",
                               SqlDbType.NVarChar, 30, "ShipCountry"));
        oDa.InsertCommand = oInsCmd;

        //==========================================================
        //-- Set up the UPDATE Command
        //==========================================================
        sProcName = "prUpdate_Order";
        oUpdCmd = new SqlCommand(sProcName, oCn, oTrn);
        oUpdCmd.CommandType = CommandType.StoredProcedure;
        oUpdCmd.Parameters.Add(new SqlParameter("@nOrderID",
                               SqlDbType.Int, 4, "OrderID"));
        oUpdCmd.Parameters.Add(new SqlParameter("@dtOrderDate",
                               SqlDbType.DateTime, 8,"OrderDate"));
        oUpdCmd.Parameters.Add(new SqlParameter("@sShipCity",
                               SqlDbType.NVarChar, 30, "ShipCity"));
        oUpdCmd.Parameters.Add(new SqlParameter("@sShipCountry",
                               SqlDbType.NVarChar, 30, "ShipCountry"));
        oDa.UpdateCommand = oUpdCmd;

        //==========================================================
        //-- Set up the DELETE Command
        //==========================================================
        sProcName = "prDelete_Order";
        oDelCmd = new SqlCommand(sProcName, oCn, oTrn);
        oDelCmd.CommandType = CommandType.StoredProcedure;
        oDelCmd.Parameters.Add(new SqlParameter("@nOrderID",
                               SqlDbType.Int, 4, "OrderID"));
        oDa.DeleteCommand = oDelCmd;

        //==========================================================
        //-- Save all changes to the database
        //==========================================================
        oDa.Update(oDs.Tables["Orders"]);

        oTrn.Commit();
        oCn.Close();
    }
    catch (DBConcurrencyException exDBConcurrency)
    {
        //=======================================================
        //-- Roll back the transaction
        //=======================================================
        oTrn.Rollback();

        //--------------------------------------------------------
        //-- May want to rethrow the Exception at this point.
        //-- This depends on how you want to handle the concurrency
        //-- issue.
        //--------------------------------------------------------
        //-- throw(exDBConcurrency);
    }
    catch (Exception ex)
    {
        //==========================================================
        //-- Roll back the transaction
        //==========================================================
        oTrn.Rollback();

        //--------------------------------------------------------
        //-- Rethrow the Exception
        //--------------------------------------------------------
        throw;
    }
    finally
    {
        oInsCmd.Dispose();
        oUpdCmd.Dispose();
        oDelCmd.Dispose();
        oDa.Dispose();
        oTrn.Dispose();
        oCn.Dispose();
    }
    oCn.Close();
    return oDs;
}

Batch Updates

One situation in which the ADO Recordset object is commonly used is to make batch updates. For example, in an n-tier application, an ADO Recordset could retrieve a rowset, disconnect itself from the data source, and send the Recordset to the client tier. In the client application, changes could be made to several rows in the disconnected ADO Recordset. Then the Recordset could be sent back to the middle tier, reconnected to the database via an ADO Connection object, and its changes could be applied to the database via the UpdateBatch method. The UpdateBatch method takes all of the changed rows in the Recordset and applies the changes to the source table that is indicated in the Recordset's source query.

ADO.NET also has built-in abilities to handle batch updates. The DataSet is a self-contained, always disconnected collection of rowsets which stores both the original and current values of its rowsets rows and columns. A DataSet can be sent to a client application where changes could be made to it. The DataSet could then be sent to the middle tier where its changes could be applied to the database via a DataAdapter object.

Figure 2 shows how the ADO.NET DataAdapter can use its various Command objects to represent the SELECT, INSERT, UPDATE, and DELETE SQL commands:

' Setting the ADO.NET DataAdapter's command objects in Visual Basic .NET
oDa.InsertCommand = oInsCmd
oDa.UpdateCommand = oUpdCmd
oDa.DeleteCommand = oDelCmd
oDa.Update(oDs.Tables("Orders"))

// Setting the ADO.NET DataAdapter's command objects in C#
oDa.InsertCommand = oInsCmd;
oDa.UpdateCommand = oUpdCmd;
oDa.DeleteCommand = oDelCmd;
oDa.Update(oDs.Tables["Orders"]);

The ADO.NET technique allows greater flexibility than the traditional ADO batch update technique regarding how it can apply the updates to the database. Traditional ADO decides where to save the changes by examining the Source property of the ADO Recordset. For example, assume the source of the Recordset is:

SELECT OrderID, CustomerID, OrderDate, ShipCity, ShipCountry FROM Orders
In this case, after batch changes have been made to the Recordset and the Recordset's UpdateBatch method is invoked, the Recordset has to infer where to send the changes. It looks at the source's SQL statement and determines that the primary key (OrderID) of the Orders table is in the statement and there is only one table in use. Thus, it can use the SQL statement to create UPDATE, INSERT, and DELETE statements to hit the Orders table. In order for the Recordset to derive the action queries, it has to know the unique row identifier. Also, stored procedures cannot be used with this technique to update the database since the statements are implicitly created. The following ADO 2.x code shows how the Recordset containing Orders could be updated in a client application:
'-- Using traditional ADO,
'-- Updating two rows in the client tier
oRs.Find "CustomerID = 'VINET'"
oRs.Fields("ShipCity") = "Somewhere"
oRs.Update
oRs.Find "CustomerID = 'CHOPS'"
oRs.Fields("ShipCity") = "Elsewhere"
oRs.Update
Then, once the Recordset is passed back to the middle tier, the following code snippet could be invoked to apply the changes you've made back to the database:
'-- Using traditional ADO,
'-- Sending both updated rows to the database, in the business tier
oRs.UpdateBatch

The ADO.NET CommandBuilder works similarly in that it can automatically generate the UpdateCommand, InsertCommand, and DeleteCommand objects for the DataAdapter. The process of automatically generating the commands comes with overhead. It is more efficient to explicitly specify the INSERT, UPDATE, and DELETE statements that you want to use. The use of the ADO.NET CommandBuilder means slower performance and less control over how changes are applied to a data source compared to explicitly indicating the commands.

The exact SELECT, INSERT, UPDATE, and DELETE statements are generally known at design time and can be used to populate the ADO.NET DataAdapter's four separate Command properties. They can even be represented by the name of a stored procedure. This feature of ADO.NET was one of the key pieces missing in traditional ADO that would have made its batch update techniques much more flexible. This requires a bit more coding at design time, but it's worth the effort to get the flexibility it offers in allowing you to specify very specific SQL statements, or more often stored procedures for each of the operations.

Disconnected Rowsets

The ADO Recordset object can be disconnected from its data source by setting some of its properties appropriately. By being disconnected, it can store the entire rowset in memory and be passed around between applications that use traditional ADO. The following code sample will disconnect a Recordset:

'-- Disconnecting an ADO Recordset
oRs.CursorLocation = adUseClient
oRs.CursorType = adOpenStatic
oRs.LockType = adLockBatchOptimistic
oRs.Open
Set oRs.ActiveConnection = Nothing

The key properties involved in disconnecting a Recordset are the CursorType and CursorLocation. The CursorLocation must be set to adUseClient, which signifies that the rowset should be stored within the Recordset's memory. The CursorType should be set to adOpenStatic, which allows the rowset's cursor to move in any direction but does not allow the rowset to be automatically sensitive to changes in the underlying database. By combining these two settings, the Recordset can become disconnected. However, the final step is yet to come. Once the Recordset has been opened, for it to truly become disconnected, its ActiveConnection property should be set to the keyword Nothing, as you saw in the last code sample.

The ADO Recordset was built to work in both disconnected and connected modes. However, in ADO.NET the DataReader remains connected while the DataSet is entirely disconnected. The DataSet is the successor to the disconnected ADO Recordset as it implements a client-side cursor that can be scrolled in any direction. In addition, the ADO.NET DataSet supports many additional features. The traditional ADO Recordset object provides very limited disconnected features while the ADO.NET DataSet object is designed to be disconnected. Unlike the Recordset, the DataSet can store and represent multiple rowsets within DataTable objects and even relate them to one another using DataRelation objects. Here's a common way to create and populate a DataSet using a SqlDataAdapter's Fill method:

' Creating and filling an ADO.NET DataSet in Visual Basic .NET
Dim oDs As New DataSet("MyDataSet")
oDa.Fill(oDs)

// Creating and filling an ADO.NET DataSet in C#
DataSet oDs = new DataSet("MyDataSet");
oDa.Fill(oDs);
The DataSet can enforce relations, primary key constraints, and foreign key constraints, and it can even implement column expressions. The ADO Recordset also has the ability to return hierarchical results using its data shaping features. While these hierarchical features are available, they require the use of what I believe is an awkward data-shaping syntax. The ADO.NET DataSet is more efficient and more intuitive in this respect because it is built to work with relational data structures using its DataRelation and DataTable objects.

Over the past two columns I have discussed several techniques commonly used in traditional ADO 2.x development and how to translate them to ADO.NET. ADO.NET is the next step in the evolution of ADO 2.x and many features found in traditional ADO have been entirely revamped in ADO.NET. For example, while features were added in ADO 2.x to the Open and Save methods in order to support XML, XML support was in the blueprints for ADO.NET from the beginning. Some features of ADO have changed considerably. One such change is that the Recordset's many cursor types are split into many distinct and focused objects such as the DataReader, the DataSet, and the DataAdapter. Other features such as the connection and command classes have changed in less obvious ways. However, in the end, the migration path from ADO to ADO.NET is relatively straightforward as ADO.NET is designed to build upon your experience with traditional ADO.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Cutting Edge: Script Callbacks in ASP.NET

Cutting Edge
Script Callbacks in ASP.NET
Dino Esposito

Code download available at: CuttingEdge0408.exe (127KB)
Contents
Out-of-Band Calls
ASP.NET 2.0 Script Callbacks
Using Script Callbacks
Refresh the Data, Not the Page
Under the Hood of ASP.NET 2.0
Script Callbacks in ASP.NET 1.x
Conclusion

If you're involved in Web development you may have faced a problem that you couldn't find a good solution for—making client-to-server calls outside the current page. For example, you might want to validate the content of a textbox against data stored on the server asynchronously, without interrupting the continuity of the work or without forcing a full page refresh, which is particularly heavy for UI-rich pages. How would you code such an out-of-band call from the client to the server and back?

There are a few issues to address to solve this problem. First and foremost, you need to set up a parallel HTTP channel for sending the request and getting the response. The new request should be invisible to the user to avoid any interference with the displayed page. Finally, the response you get from this invisible request must be merged with the current page through dynamic changes to the document object model (DOM) of the page.

The great news about the scripting object model in ASP.NET 2.0 is that it allows calls to server events from client code without causing the page to post back and refresh. This is implemented through a callback mechanism.

In this column, I'll outline the problem and discuss some feasible solutions. Then, I'll focus on the details of the ASP.NET 2.0 script callback implementation and provide a tiny but effective framework that you can use in ASP.NET 1.x as well.

Out-of-Band Calls

The display of a Web page is the final step in a relatively simple process that begins with an HTTP command sent by a browser to a target URL or IP address. The browser opens a socket to the specified IP address, sends the packets, and waits for the response. When the response text arrives, it is displayed according to its content type. As a result, the previously displayed page is wiped out and fully replaced with the new one.

This pattern remains valid even when a page posts to itself—the common pattern with ASP.NET pages. In this case, the page's output is cleared and redisplayed—an effect that might not be desired by the user. If you can make assumptions about the browser's capabilities, I recommend you take another route: out-of-band calls. It's an alternative postback pattern that proves particularly effective for rich client Web applications.

In this process I have a component in the currently viewed page opening a socket to the specified remote URL and sending a request. If the request is sent asynchronously, the user can continue working with the page in a non-blocking way without experiencing delays or waiting. When the response is fully downloaded on the client, the same component detects the event, retrieves the information from the response, and decides how to update the page. While the request is ongoing, the user continues to see the same page unchanged. When the new data arrives, the component refreshes the current user interface taking full advantage of the browser's capabilities—namely, DHTML support. This generic pattern can find various implementations—the first was Remote Scripting; the latest incarnation is ASP.NET 2.0 script callbacks.

A few years ago, Microsoft released an interesting technology—Remote Scripting—that, for whatever reason, was never embraced by as broad an audience as it deserved. Remote Scripting closely follows the model I just described. It uses some client-side script to trigger a Java-language applet which, in turn, opens the socket to the destination URL. The remote URL must be a classic ASP page that includes a call to the server-side layer of Remote Scripting and exposes an object with a given name—public_description. The particular naming schema required here serves to define the contract between the caller and the server page. (In classic ASP, there's still no notion of interfaces.)

The return value of the call must be a string. The string is processed on the client and passed to a JavaScript callback function for merging the content with the rest of the page. Using Remote Scripting is effective because it works with a fair number of browsers and doesn't require a recent version of Microsoft® Internet Explorer. However, the mechanism of out-of-band calls—regardless of how you code them—is really powerful only if the browser allows for dynamic changes to the page. Otherwise, you can do little more than display a pop-up message. For more information about Remote Scripting, check out the article at Using Remote Scripting.

Another implementation of the out-of-band pattern that has been around for a while is based on Web services. The webservice.htc DHTML behavior is a simple, lightweight component that encapsulates the invocation of remote methods using the SOAP protocol. This behavior enables pages viewed with Internet Explorer 5.0 and later to communicate directly with Web services regardless of the platform. In this case, you use XmlHTTP instead of a Java-language applet, but the idea behind the mechanism is similar. The bad news is that you are limited to an uplevel family of browsers; the good news is that you can access server-side code on virtually any platform precisely because a Web service is involved. More information and the source code of the webservice.htc behavior is available for download at WebService Behavior.

All in all, out-of-band calls are an old problem that developers (including developers at Microsoft) made several attempts to solve. However, until ASP.NET 2.0 no solution was fully integrated with the hosting framework. In ASP.NET 2.0 the scripting model of the Page object is enriched with callback abilities that provide an ASP.NET-specific implementation of a kind of remote scripting. I'll take a look at that next, then I'll move on to provide an implementation that works with ASP.NET 1.x.

ASP.NET 2.0 Script Callbacks

In ASP.NET 2.0, the overall scripting model has been significantly enhanced. In addition to script callbacks, you have full access to the contents of the page's <head> tag, can programmatically assign the input focus to a particular control, read and write the title of the page, and make buttons and other controls post to any page in the application. Check out the Beta 1 documentation for quick examples and references.

To use ASP.NET 2.0 script callbacks, you define a trigger element in the page (not a Submit button) and bind it to some JavaScript code. This code will retrieve input data from the current page input fields and prepare a call to a system-provided script function named WebForm_DoCallback in Beta 1. This function is expected (in the final release) to open the HTTP connection to the specified remote ASP.NET page. The ASP.NET runtime detects a callback invocation and executes a particular method on the page. The return value of the server-side method is passed back to the client as the response to the previous request. On the client, the response gets passed to a user-defined JavaScript callback function that can then update the user interface via DHTML. The bottom line is that a round-trip still occurs, but the page is not fully refreshed. More importantly, the user can continue working with the controls in the page while the parallel request is served. Figure 1 shows the architecture of script callbacks in ASP.NET 2.0.

Figure 1 Script Callbacks
Figure 1 Script Callbacks

You can use callbacks to update individual elements of a page, such as a chart or a panel, provide different views of the same data, download additional information on demand, or auto-fill one or more fields. In particular, the ASP.NET 2.0 TreeView control uses script callbacks extensively to implement its expand/collapse features and the GridView control uses callbacks to page and sort without explicit postbacks.

Using Script Callbacks

Developing an ASP.NET 2.0 page that makes use of script callbacks is a two-step procedure. First, you write the server-side code that will be invoked from the client. In doing so, you basically define the data model for the call. You decide what information must be exchanged and in which format. The actual type being exchanged must be a string, but the contents of the string can be quite different—JavaScript, XML, a comma-separated string of values, and so on.

The second step requires you to define the client-side code to trigger the out-of-band call. The remote invocation begins when a call is made to a built-in JavaScript function named WebForm_DoCallback. You don't necessarily need to know the name and signature of this function, as you can get it from a new member of the Page class—the GetCallbackEventReference method:

string js = GetCallbackEventReference(this, arg, func, "null", "null");

A call to this JavaScript function is bound to a clickable element in the page—for example, a <button> tag. It is essential that the clickable element not be a Submit button. For this reason, you can't render this button using the <asp:button> control because such a server control outputs the submit button markup:

<input type="submit" id="button" ...>

You need to attach some client-side code (the onclick handler) to an HTML element that can fire an event if the user interacts with it. Note that if you associate a client's onclick handler with a Submit button, the client code is executed properly, but after that the page posts back and cancels all the dynamic changes brought about by the remote out-of-band call.

So far I have discussed the code needed to instruct the client page to open a channel to the server and send an HTTP request to a remote ASP.NET page. What happens next?

As expected, the ASP.NET runtime takes care of the request and begins processing it. First, the page HTTP handler (the ProcessRequest method on the Page class) determines the postback mode by looking at the request headers and body, including the __VIEWSTATE hidden field. Next, it figures out if the page is being invoked on an out-of-band call. If so, it sets the public IsCallback property of the Page to true. To determine the callback mode, the ASP.NET runtime looks for a __CALLBACKID entry in the Request collection. If such an entry is found, the runtime concludes that a callback invocation is being made. So, how is a callback invocation different from a postback invocation?

The runtime checks to see if the referenced control on a requested page implements the ICallbackEventHandler interface (which could be the page itself). A page that implements an interface will have the following directive (this holds true for ASP.NET 1.x too):

<%@ Implements Interface="System.Web.UI.ICallbackEventHandler" %>
If the control implements the required interface, the ASP.NET runtime invokes the RaiseCallbackEvent method on the interface and prepares the response from the results of the call.

The ICallbackEventHandler interface features just one method with the following signature:

string RaiseCallbackEvent(string eventArgument)
The actual parameter for the method is retrieved from the Request collection of posted values. The string representation of the input data is contained in an entry named __CALLBACKPARAM. Once again, this string can actually contain everything you want and need, including XML data or Base64 data, comma-separated values, dates, numbers, and so forth. Let's consider an example.

Refresh the Data, Not the Page

Figure 2 contains the source of an ASP.NET 2.0 page that takes advantage of script callbacks. It looks like an ordinary page except for a few additional features, the most important of which are the implementation of the ICallbackEventHandler interface and the JavaScript code to refresh the page at the end of the call. The key things to notice are the client-side mechanism to trigger the out-of-band call, the server-side code that serves the call, and the client-side JavaScript code to merge results with the current page.

Figure 2 Using Script Callbacks (Close Figure 2)
<%@ page language="C#" %>
<%@ import namespace="System.Data" %>
<%@ implements interface="System.Web.UI.ICallbackEventHandler" %>

<script language="javascript">
    function UpdateEmployeeViewHandler(result, context)
    {
        var o = result.split(",");
        e_ID.innerHTML = o[0];
        e_FName.innerHTML = o[1];
        e_LName.innerHTML = o[2];
        e_Title.innerHTML = o[3];
        e_Country.innerHTML = o[4];
        e_Notes.innerHTML = o[5];
    }
</script>

<script runat="server">
    public virtual string RaiseCallbackEvent (string eventArgument)
    {
        // Get more info about the specified employee
        int empID = Convert.ToInt32 (eventArgument);
        EmployeesManager empMan = new EmployeesManager();
        EmployeeInfo emp = empMan.GetEmployeeDetails (empID);

        string[] buf = new string[6];
        buf[0] = emp.ID.ToString ();
        buf[1] = emp.FirstName;
        buf[2] = emp.LastName;
        buf[3] = emp.Title;
        buf[4] = emp.Country;
        buf[5] = emp.Notes;
        return String.Join(",", buf);
    }

    void Page_Load (Object sender, EventArgs e)
    {
        // Populate the drop-down list
        EmployeesManager empMan = new EmployeesManager();
        DataTable dt = empMan.GetListofNames();
        cboEmployees.DataSource = dt;
        cboEmployees.DataTextField = "lastname";
        cboEmployees.DataValueField = "employeeid";
        cboEmployees.DataBind();

        // Prepare the Javascript function to call
        string callbackRef = GetCallbackEventReference(this,
            "document.all['cboEmployees'].value",
            "UpdateEmployeeViewHandler", "null", "null");

        // Bind it to a button
        buttonTrigger.Attributes["onclick"] =
            String.Format("javascript:{0}", callbackRef);
    }
</script>

<html>
<body>
    <form id="Form1" runat="server">
        <h1>Select a name and click for details</h1>
        <asp:dropdownlist id="cboEmployees" runat="server" />
        <button runat="server" id="buttonTrigger">More Info</button>
        <br />
        <table>
        <tr><td><b>ID</b></td><td><span id="e_ID" /></td></tr>
        <tr><td><b>Name</b></td><td><span id="e_FName" /></td></tr>
        <tr><td><b>Last Name</b></td><td><span id="e_LName" /></td></tr>
        <tr><td><b>Title</b></td><td><span id="e_Title" /></td></tr>
        <tr><td><b>Country</b></td><td><span id="e_Country" /></td></tr>
        <tr><td><b>Notes</b></td>
            <td><i>
            <span id="e_Notes" /></i></td></tr>
        </table>
    </form>
</body>
</html>

As I mentioned, you need an interactive HTML element that doesn't submit the page. The HTML 4.0 <button> tag is a good choice here. However, because it needs a dynamically generated onclick event handler, you must mark it with the runat attribute:

<button runat="server" id="btn">More Info</button>

The onclick handler points to a built-in piece of JavaScript code that initiates and controls the out-of-band call. The GetCallbackEventReference method on the Page class returns the script string that starts the callback:

string js = GetCallbackEventReference(this,
   "document.all['cboEmployees'].value",
   "UpdateEmployeeViewHandler", "null", "null");
btn.Attributes["onclick"] = String.Format("javascript:{0}", js);

The function receives the arguments described in Figure 3. In the previous code snippet, I'm making the call to the current page and passing the value of the item currently selected in a page's dropdown list. UpdateEmployeeViewHandler is the name of the JavaScript function that will merge callback results with the current page. Context is any DHTML reference or extra information you want to pass to the callback is useful to the merge process. Finally, the error callback is a JavaScript function that is automatically invoked if an error occurs during the callback. As you can see in Figure 4, you select an employee in the dropdown list and click the button to raise a callback event on the server. The handler for the button is shown in the following code snippet:

<button id="buttonTrigger"
   onclick="javascript:WebForm_DoCallback('__Page',
         document.all['cboEmployees'].value,
         UpdateEmployeeViewHandler, null, null)">
   More Info
</button>

Figure 3 GetCallbackEventReference Arguments (Close Figure 3)

Parameter | Description
control | The control or the page which implements RaiseCallbackEvent.
argument | Client-side script that will be executed prior to making the callback. The return value of the evaluation of this script on the client is sent to the RaiseCallbackEvent method via the eventArgument parameter.
clientCallback | Name of the client-side event handler which will receive the result of a successful server event.
context | Client-side script that will be evaluated on the client prior to making the callback. The result of this will be passed back to the client-side event handler via the context parameter.
clientErrorCallback | Name of the client-side event handler which will receive the result of the RaiseCallbackEvent method when an error occurs.

In addition, the markup of the page contains the related instructions shown in the following code:

<script src="WebResource.axd?a=s&amp;r=WebForms.js&amp;t={...}"
        type="text/javascript" />
<script type="text/javascript">
<!--
  var pageUrl='/Intro20/Samples/Ch01/Callback/Invoke.aspx';
  WebForm_InitCallback();
// -->
</script>
WebResource.axd is a new built-in HTTP handler in ASP.NET 2.0 that's used to retrieve script code into pages. This handler guarantees that all the script code necessary to perform the callback is correctly referenced within the page or control. In particular, WebResource.axd ensures that calls to WebForm_DoCallback and WebForm_InitCallback are successfully resolved. If you want to take a look at the real source code behind script callbacks in ASP.NET 2.0, open the Temporary Internet Files folder on your Web server machine and look up the file WebResource.axd. The code injected through this HTTP handler represents the callback manager in charge of sending the request and handling the results. The manager registers any JavaScript callback function and calls it back when the server event has been fully processed. The <script> blocks I just showed you are inserted in the page when you call the GetCallbackEventReference function. For this reason, you should never use WebForm_DoCallback directly; the call will fail if the right <script> blocks aren't inserted and properly initialized. Another good reason for not calling WebForm_DoCallback directly is that this name is not part of the public page API. As such, it may change without notice in future builds of ASP.NET 2.0 as well as in future minor or major upgrades of the platform.

Figure 4 Trying it Out
Figure 4 Trying it Out

The ID of the currently selected employee (shown in Figure 4) is ultimately passed to the server-side implementation of the ICallbackEventHandler interface and then it's used to retrieve the information for the client:

string RaiseCallbackEvent (string eventArgument)
{
   // Get more info about the specified employee
   int empID = Convert.ToInt32 (eventArgument);
   EmployeesManager empMan = new EmployeesManager();
   EmployeeInfo emp = empMan.GetEmployeeDetails(empID);

   // Prepare the string for the script callback.
   string buf = "";
   •••
   return buf;
}

The format of the event argument string is arbitrary and left up to you. The same is true for the string you return to the script callback—the aforementioned UpdateEmployeeViewHandler function that you see in Figure 5.

Figure 5 UpdateEmployeeViewHandler (Close Figure 5)
<script language="javascript">
    function UpdateEmployeeViewHandler(result, context)
    {
       // Comma-separated string created in RaiseCallbackEvent
       var o = result.split(",");

       // Merge data with the existing client page UI
       e_ID.innerHTML = o[0];
       e_FName.innerHTML = o[1];
       e_LName.innerHTML = o[2];
       e_Title.innerHTML = o[3];
       e_Country.innerHTML = o[4];
       e_Notes.innerHTML = o[5];
    }
</script>

To merge the server-side generated values with the existing page, you typically use the page's DHTML object model. You give each updateable HTML tag a unique ID and modify its contents scriptwise using the innerHTML property or any other property and method the DOM supplies. Figure 4 shows the state once a callback operation has completed. More importantly, there's no postback visible to the user even though a round-trip still occurs under the hood.

Under the Hood of ASP.NET 2.0

To understand how to implement script callbacks in ASP.NET 1.x, you should learn exactly what really happens in ASP.NET 2.0. In particular, you should figure out the contents of the HTTP request sent and what component is used to accomplish that. The ASP.NET 2.0 documentation states that in order to take advantage of script callbacks the user must be running Internet Explorer 5.0 or later. The prototype of WebForm_DoCallback is shown here:

function WebForm_DoCallback(
   eventTarget,
   eventArgument,
   eventCallback,
   context,
   errorCallback)
{
   •••
}
It uses a COM object to issue an HTTP POST or GET command to the specified target URL. The COM object used here is an old acquaintance of many developers:
var xmlRequest = new ActiveXObject("Microsoft.XMLHTTP");

The HTTP verb is GET or POST depending on the size of the data to send. If the size exceeds 2KB, a POST command is used. The HTTP request consists of three logical elements: __CALLBACKID, __CALLBACKPARAM, and posted data. The __CALLBACKID value contains the destination URL (the event target parameter), whereas __CALLBACKPARAM carries the input parameter for the server-side stub method. The posted data is collected by the WebForm_InitCallback method and appended to the HTTP command. The following pseudocode shows how to initiate the out-of-band request:

var xml = new ActiveXObject("Microsoft.XMLHTTP");
xml.onreadystatechange = callback;
•••
postData = __theFormPostData +
    "__CALLBACKID=" + eventTarget +
    "&__CALLBACKPARAM=" + escape(eventArgument);
xml.open("POST", pageUrl, true);
xml.setRequestHeader("Content-Type",
          "application/x-www-form-urlencoded");
xml.send(postData);

When the request terminates, the specified callback is invoked to complete the job. As an aside, be aware that the Microsoft.XmlHttp COM object ships with Internet Explorer 5.0 and later. And, of course, these implementation details are subject to change.

Script Callbacks in ASP.NET 1.x

At this point, I'd say that implementing script callbacks in ASP.NET 1.x shouldn't be that hard. You need a common piece of JavaScript code to inject into all pages that intend to support client callbacks. This code should initiate and control the remote URL invocation. The code in Figure 6 provides a possible implementation, instantiating the XmlHttp object and preparing a POST command. The function (named DoCallback in the example) extends the query string of the URL with a couple of parameters. The first parameter, Callback, contains a Boolean parameter to indicate whether or not the request is going to be a regular postback or a callback. The second parameter, Param, contains the input parameters for the server-side code. Note that the code in Figure 6 can be easily extended to work asynchronously:

xmlRequest.onreadystatechange = callbackFunction;

Figure 6 Callback in ASP.NET (Close Figure 6)
<script language="javascript">
function DoCallback(url, params)
{
   // url:    URL to invoke
   // params: string object to pass to the remote URL

   // Add some parameters to the query string
   var pageUrl = url + "?callback=true&param=" + params;

   // Initialize the XmlHttp object
   var xmlRequest = new ActiveXObject("Microsoft.XMLHTTP");

   // Prepare for a POST statement and synchronous. (All this is
   // arbitrary and can be changed in your own implementation.)
   xmlRequest.open("POST", pageUrl, false);
   xmlRequest.setRequestHeader("Content-Type",
           "application/x-www-form-urlencoded");
   xmlRequest.send(null);

   // Return the XmlHttp object
   return xmlRequest;
}
</script>

The following snippet shows a piece of client code that, attached to a button, fires the client callback and refreshes the user interface:

function MoreInfo()
{
   var selectedEmpID = document.all["EmployeeList"].value;
   var xml = DoCallback("webform1.aspx", selectedEmpID);

   // Sync updates
   Msg.innerHTML = xml.responseText;
}

How is the server page structured? Figure 7 shows the full source code of a page just like it. In the Page_Load event handler, you check to see if the request is a callback using code like this:

if (Request.QueryString["callback"] != null)
{
   string param;
   param = Request.QueryString["param"].ToString();
   Response.Write(RaiseCallbackEvent(param));
   Response.Flush();
   Response.End();
   return true;
}
If the query string contains the Callback parameter, you invoke a predefined function—here RaiseCallbackEvent—and pass the return value to Response. Next, you flush the output buffer and end the request. That's it.

Figure 7 The Server Page (Close Figure 7)
namespace MsdnMag
{
    public class CallbackPage : System.Web.UI.Page
    {
        // References to page controls
        protected DropDownList EmployeeList;
        protected HtmlButton ButtonGo;

        // Initialize the page here
        private void Page_Load(object sender, EventArgs e)
        {
            if (IsCallback())
                return;

            if (!IsPostBack)
                PopulateList();

            // Button-to-callback binding
            string callbackRef = "MoreInfo()";
            ButtonGo.Attributes["onclick"] = callbackRef;
        }

        private bool IsCallback()
        {
            if (Request.QueryString["callback"] != null)
            {
                string param = Request.QueryString["param"].ToString();
                Response.Write(RaiseCallbackEvent(param));
                Response.Flush();
                Response.End();
                return true;
            }

            return false;
        }

        // Handle the "Go Get Data" button click
        protected void OnGoGetData(object sender, EventArgs e)
        {
        }

        // Populate the drop-down list with employee names
        private void PopulateList()
        {
            SqlDataAdapter adapter = new SqlDataAdapter(
                "SELECT employeeid, lastname FROM employees",
                "SERVER=(local);DATABASE=
                    northwind;TRUSTED_CONNECTION=true;");
            DataTable table = new DataTable();
            adapter.Fill(table);

            EmployeeList.DataTextField = "lastname";
            EmployeeList.DataValueField = "employeeid";
            EmployeeList.DataSource = table;
            EmployeeList.DataBind();
        }

        // *******************************************************
        // Implement the callback interface
        string RaiseCallbackEvent(string eventArgument)
        {
            return "You clicked: " + eventArgument;
        }
        // *******************************************************
    }
}

It is worth noting that in ASP.NET 1.x you need to verify yourself if the request is a callback. This check is necessary for any implementation, but in ASP.NET 2.0 it is buried in the Page class.

Conclusion

Script callbacks allow you to execute out-of-band calls back to the server. These calls are special postbacks, so a round-trip always occurs; however, unlike classic postbacks, script callbacks don't redraw the whole page and they give users the illusion that everything is taking place on the client. ASP.NET 2.0 provides a built-in mechanism for client callbacks based on the services of a COM object that ships with Internet Explorer 5.0 and later. By using the same object, you can implement a script callback mechanism in ASP.NET 1.x as well. The companion code, available from the link at the top of this article, provides a sample implementation.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the newest Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


The XML Files: What's New in WSE 2.0

The XML Files
What's New in WSE 2.0
Aaron Skonnard

Contents
Compatibility
Side by Side Execution
Installation
Getting Started with 2.0
Messaging Enhancements
WSDL Support
Security Enhancements
WSE Security Settings Tool
Automatic Secure Conversation
Next-Hop Routing
Where are We?

Microsoft has recently released Web Services Enhancements for Microsoft® .NET (WSE) 2.0. WSE 2.0 provides extensions to the existing ASP.NET Web services framework (.asmx) as well as a standalone messaging framework that's completely transport independent. The WSE 2.0 Technology Preview release has been available on the MSDN® Web site for some time but this new release contains several interesting additions and changes worth highlighting.

Compatibility

WSE provides a shipping vehicle for introducing new Web services functionality in a timely fashion. This makes it possible for Microsoft to keep up with emerging Web services specifications that are continually changing. Since WSE serves as a platform extension, it wasn't designed under the same constraints that apply to a major platform release. This means WSE doesn't have to play by the same rules as, say, a new release of the .NET Framework where breaking changes would be disastrous.

According to the documentation, "WSE attempts to maintain a degree of compatibility" but it acknowledges that complete compatibility won't be possible given the nature of certain changes. For example, sending a SOAP message between an application built with WSE 1.0 and another application built with WSE 2.0 will always result in a SOAP fault. This is because each version implements a different set of Web services specifications. WSE 1.0 implements WS-Routing and the original WS-Security specifications, while WSE 2.0 implements WS-Addressing and the new OASIS WS-Security standard.

There may also be code incompatibilities when moving between WSE versions. The development team made several breaking changes to the WSE object model between versions 1.0 and 2.0. This means you'll have to rework some existing WSE 1.0 code to work with the new WSE 2.0 object model.

WSE 2.0 uses a different assembly and namespace name than WSE 1.0. The WSE 1.0 assembly name is Microsoft.Web.Services, while the WSE 2.0 assembly name is now Microsoft.Web.Services2. The name of the WSE 2.0 configuration element is now <microsoft.web.services2>, and the WSE 2.0 root namespace is now microsoft.web.services2 (see Figure 1).

Figure 1 Differences Between WSE 1.0 and WSE 2.0 (Close Figure 1)

Item | WSE 1.0 | WSE 2.0
Assembly name | Microsoft.Web.Services | Microsoft.Web.Services2
Namespace name | Microsoft.Web.Services | Microsoft.Web.Services2
Configuration element | <microsoft.web.services> | <microsoft.web.services2>
Version number | 1.0.0.0 | 2.0.0.0
PublicKeyToken | 31bf3856ad364e35 | 31bf3856ad364e35

These differences will affect WSE-related entries in your application configuration files or project references as well as your code wherever you're using the WSE namespace. For example, in WSE 1.0 you register the WSE SoapExtension as shown in the top of Figure 2, whereas in WSE 2.0 you register the SoapExtension as shown in the bottom of Figure 2.

Figure 2 Registering the WSE SoapExtension (Close Figure 2)

WSE 1.0
<configuration>
  <system.web>
    <webServices>
      <soapExtensionTypes>
        <add type=
         "Microsoft.Web.Services.WebServicesExtension,
          Microsoft.Web.Services, Version=1.0.0.0,
          Culture=neutral,
          PublicKeyToken=31bf3856ad364e35"
          priority="1" group="0" />
      </soapExtensionTypes>
    </webServices>
  <system.web>
<configuration>
WSE 2.0
<configuration>
  <system.web>
    <webServices>
      <soapExtensionTypes>
        <add type=
         "Microsoft.Web.Services2.WebServicesExtension,
          Microsoft.Web.Services2, Version=2.0.0.0,
          Culture=neutral,
          PublicKeyToken=31bf3856ad364e35"
          priority="1" group="0" />
      </soapExtensionTypes>
    </webServices>
  <system.web>
<configuration>

Another fundamental change in the core object model has to do with acquiring SoapContext objects in your code. WSE 1.0 introduced the SoapContext class that models WSE information passed in the SOAP message (similar to the HttpContext object for generic HTTP requests). In WSE 1.0, you access the current SoapContext object from within a WebMethod as shown in the code located at the top of Figure 3.

Figure 3 Get Current SoapContext Object (Close Figure 3)

WSE 1.0
using Microsoft.Web.Services; // WSE 1.0
public class SomeClass
{
 [WebMethod]
  public string DoSomething()
  {
    SoapContext reqCtx = HttpSoapContext.RequestContext;
    SoapContext resCtx = HttpSoapContext.ResponseContext;

    ... // use SoapContext to access WSE information
  }
  •••
}
WSE 2.0
using Microsoft.Web.Services2; // WSE 2.0
public class SomeClass
{
 [WebMethod]
  public string DoSomething()
  {
    SoapContext reqCtx = RequestSoapContext.Current;
    SoapContext resCtx = ResponseSoapContext.Current;

    ... // use SoapContext to access WSE information
  }
  •••
}

In WSE 2.0, however, the HttpSoapContext class has been marked obsolete because the object model has evolved to the point that SoapContext can be used over other protocols in addition to HTTP. Hence, in WSE 2.0 HttpSoapContext has been removed and two new classes named RequestSoapContext and ResponseSoapContext have been added. Each class has a static Current property for retrieving the current SoapContext object as shown in the bottom of Figure 3.

Several other WSE 1.0 classes have been changed or entirely removed. For example, on the security front, the IPasswordProvider and IDecryptionKeyProvider interfaces have been removed in WSE 2.0. You'll have to rework any WSE 1.0 applications that used them and port the code to the new WSE 2.0 security framework. These are the most fundamental and obvious breaking changes but there are others as well. We'll discuss these differences throughout the remainder of this column.

In order to make it easier for developers to migrate to WSE 2.0, Microsoft also made it possible to run both Web Services Enhancements versions side by side.

Side by Side Execution

If you can't afford to break existing services or deal with the compatibility issues just described, you can choose to run both WSE versions simultaneously. Thanks to the fact that WSE 2.0 uses different assembly, namespace, and configuration names, you can run WSE 1.0 and WSE 2.0 applications side by side on the same machine and even in the same process. In fact, it shouldn't cause any problems to install WSE 2.0 on a machine where WSE 1.0 services are currently deployed. The WSE 1.0 services will continue to use the WSE 1.0 bits and new services can be designed to take advantage of the newly installed WSE 2.0 bits.

However, you should be aware that a given Web service cannot use WSE 1.0 and WSE 2.0 SoapExtensions at the same time. It can only be configured to use one or the other. Client applications, on the other hand, can use a mix of WSE 1.0 and WSE 2.0 proxy classes by distinguishing between them by using fully qualified names. And you can use the WSE 2.0 messaging API together with WSE 1.0 SoapExtensions.

Installation

In order to get started with WSE 2.0, download the bits from the MSDN Web Services Developer Center and run the installation. If you have the WSE 2.0 Technology Preview installed, you must uninstall it before installing this release. However, as we discussed in the previous section, you don't have to uninstall WSE 1.0 before moving forward with WSE 2.0 since they can coexist.

When you run the installation, you'll notice several setup type options that didn't exist before, including options to install only the runtime files, runtime plus tools, documentation, Visual Studio tools, and a custom install option. These setup options help distinguish between the various WSE components needed by different usage profiles, and the Custom option lets you select the various components manually. Most developers should choose the Visual Studio® Developer option, which installs the WSE 2.0 runtime files, documentation and samples, standalone tools, and the Visual Studio .NET tools.

WSE 1.0 didn't ship with any supported tools. Microsoft provided the WSE Settings tool as a separate download from the MSDN Web site; it came with its own installation program and wasn't considered part of the supported product. Now WSE 2.0 comes with a fully integrated tool that greatly simplifies the overall development process.

Getting Started with 2.0

Assuming you choose the Visual Studio Developer option, you can begin integrating WSE 2.0 with your Visual Studio .NET solutions. Getting things configured properly is as easy as running the WSE Settings tool and checking a few boxes. You open the WSE Settings Tool by right-clicking on your project icon in Solution Explorer and selecting WSE Settings 2.0.

This will open a multi-tab dialog, which is referred to as the WSE settings tool. This tool gives you a user-friendly interface to the various WSE 2.0 configuration settings. Most of your interactions with the dialog will simply make changes to the <microsoft.web.services2> section of the application configuration file. You can always make these changes by hand, but the tool makes it easier to get it done right and to get it done quickly. Browsing through the various tabs will give you a quick overview of the new features provided in this release.

You can configure your project to use WSE 2.0 by clicking "Enable the Project for Web Services Enhancements" on the General tab. Checking this box causes the tool to add a reference to Microsoft.Web.Services2.dll and it adds the <microsoft.web.services2> section to the application configuration file. It also causes Visual Studio .NET to create WSE-enabled proxy classes when creating new Web references.

You can also check "Enable Microsoft Web Services Enhancement Soap Extensions" to add the WSE SoapExtension mapping to the project's web.config file (this only applies to ASP.NET projects). You'll want to do this if you're planning to use WSE 2.0 to extend the functionality of an ASP.NET Web Services project, such as by adding WS-Security support to a set of WebMethods. After checking both boxes and pressing OK, your project should now contain the WSE 2.0 assembly (see Figure 4) and the web.config file should look like Figure 5.

Figure 5 web.config File (Close Figure 5)
<configuration>
  <configSections>
    <!-- this registers the WSE2 configuration section -->
    <section name="microsoft.web.services2"
     type="Microsoft.Web.Services2.Configuration.WebServicesConfiguration,
     Microsoft.Web.Services2, Version=2.0.0.0, Culture=neutral,
     PublicKeyToken=31bf3856ad364e35" />
  </configSections>
  <system.web>
    <webServices>
      <soapExtensionTypes>
        <!-- this configures the WSE2 SoapExtension -->
        <add type="Microsoft.Web.Services2.WebServicesExtension,
         Microsoft.Web.Services2, Version=2.0.0.0, Culture=neutral,
         PublicKeyToken=31bf3856ad364e35" priority="1" group="0" />
      </soapExtensionTypes>
    </webServices>
  </system.web>
  <microsoft.web.services2>
    <!-- where the WSE2 configuration settings go -->
    ...
  </microsoft.web.services2>
</configuration>
Figure 4 WSE 2.0 Assembly
Figure 4 WSE 2.0 Assembly

With this in place, the project is now configured to use WSE 2.0. You can configure client applications to take advantage of WSE 2.0 the same way. When you enable WSE in a client application you ensure that all Web references will be WSE-enabled. This also takes care of setting up the <microsoft.web.services2> section in the project's app.config file.

If you're migrating an existing application from WSE 1.0 to WSE 2.0, you can follow this same procedure but then you must manually remove the WSE 1.0 assembly (Microsoft.Web.Services) from the references section as well as striking the <microsoft.web.services> section from the application configuration file.

After making these changes, try building the solution. If you're migrating a WSE 1.0 application, you'll probably get some errors that have to do with the incompatibilities described earlier. For example, you'll need to change namespace imports in order to use the WSE 2.0 namespace name and in addition you'll need to deal with object model differences.

Once you have WSE 2.0 installed and configured, you're ready to start taking advantage of its new features, most of which revolve around messaging, security, and policy. In fact, WSE 2.0 is the only supported Microsoft Web services solution currently shipping that implements the latest Web services specifications in those different areas. In addition to the baseline Web services specifications, WSE 2.0 also supports WS-Addressing, WS-Security, WS-Trust, WS-SecureConversation, WS-Policy, WS-PolicyAssertions, and WS-SecurityPolicy. The implementations of these specifications work together to provide a more robust and cohesive Web services solution. Let's take a look at some of these new features in more detail.

Messaging Enhancements

Using WSE 2.0 in conjunction with ASP.NET Web services provides a way to extend the basic functionality of a WebMethod with extra features such as security. WSE does this by taking advantage of the SoapExtension framework, which makes it possible to perform pre- and post-processing on the SOAP messages. The downside to this approach, however, is that ASP.NET Web services are tied to HTTP.

WSE also provides a lower-level messaging API that's completely transport neutral. WSE 2.0 supports sending and receiving messages over in-process, TCP, and HTTP channels. And you can even write custom channels if you need to send messages over transports that WSE doesn't currently support, such as Microsoft Message Queue Server (MSMQ).

WSE 2.0 provides two low-level classes for sending and receiving SOAP messages, called SoapSender and SoapReceiver. It also provides two higher-level classes that automate some of the most common SoapSender and SoapReceiver tasks such as object serialization, implementing request/response exchanges, and figuring out which operation to perform. These higher-level classes are called SoapClient and SoapService.

Let's take a look at an example that illustrates the differences between ASP.NET Web services and WSE 2.0. Consider the class definitions that model contact information (see Figure 6).

Figure 6 Class Definitions (Close Figure 6)
[XmlType(Namespace="http://example.org/contacts")]
public class Contact
{
   public string name;
   public Address address;
   public string phone;
   public string email;
}

[XmlType(Namespace="http://example.org/contacts")]
public class Address
{
   public string street;
   public string city;
   public string state;
   public string zip;
   public string country;
}

The sample shown in Figure 7 illustrates how to implement a simple Web service that adds and retrieves contacts from an internal store. As you can see, this sample was implemented using the traditional ASP.NET Web services framework using the [WebService] and [WebMethod] attributes.

Figure 7 ContactService Implemented in ASP.NET (Close Figure 7)
[WebService(Namespace="http://example.org/contacts")]
public class ContactManagerService : System.Web.Services.WebService
{
   static Hashtable store = new Hashtable();

   [WebMethod]
   [SoapDocumentMethod(
     ParameterStyle=SoapParameterStyle.Bare)]
   public void AddContact(Contact c)
   {
      store.Add(c.name, c);
   }

   [WebMethod]
   [SoapDocumentMethod(
     ParameterStyle=SoapParameterStyle.Bare)]
   public Contact GetContact(string name)
   {
      return (Contact)store[name];
   }
}

Figure 8 contains an equivalent implementation built using the WSE 2.0 messaging API. It contains a SoapService-derived class that can be bound to HTTP in order to provide the same functionality as the ASP.NET version. This class uses the WSE 2.0 [SoapService] and [SoapMethod] attributes. As you can see, the WSE 2.0 approach feels very similar.

Figure 8 ContactService Implemented in WSE 2.0 (Close Figure 8)
[SoapService("http://example.org/contacts")]
public class ContactManagerService2 : SoapService
{
   static Hashtable store = new Hashtable();

   [SoapMethod("http://example.org/contacts/AddContact")]
   public void AddContact(Contact c)
   {
      store.Add(c.name, c);
   }

   [SoapMethod("http://example.org/contacts/GetContact")]
   public Contact GetContact(string name)
   {
      return (Contact)store[name];
   }
}

Both ContactService implementations (see Figure 7 and Figure 8) have the same wire-level interface. In other words, they expect and produce the same XML messages on the wire. Hence, both of these implementations should provide equivalent XML Schema and WSDL definitions to consumers.

The main difference between these implementations is that the ASP.NET Web services version is tied to HTTP while the WSE 2.0 version can be used with any transport. For example, you can bind the SoapService implementation to an HTTP endpoint through an <httpHandlers> mapping in web.config, as illustrated here:

<configuration>
  <system.web>
    <httpHandlers>
      <add type="ContactService.ContactManagerService2, ContactService"
       path="contactservice.ashx" verb="*" />
    </httpHandlers>
    •••

Or, you can programmatically bind the SoapService implementation to a TCP endpoint, as illustrated in the following Windows®-based application:

private void Form1_Load(object sender, System.EventArgs e)
{
   EndpointReference er = new EndpointReference(
      new Uri("soap.tcp://localhost/contactservice"));
   SoapReceivers.Add(er, typeof(ContactManagerService));
}

WSE 2.0 delivers on the promise of SOAP as a transport-neutral messaging protocol. Now you can add additional features to your services in a way that doesn't depend on how the messages are sent. For example, the security functionality you add via WSE 2.0 will work the same over HTTP or TCP channels.

There are some differences in the messaging object model since the Technology Preview release. One difference has to do with how you reference Web services endpoints in your code. The WS-Addressing specification defines a construct named EndpointReference for referencing endpoints in XML messages. WSE 2.0 supports this concept through the EndpointReference class. In order to maintain consistency with WS-Addressing, WSE 2.0 made changes throughout the entire object model to work with EndpointReference objects instead of raw Uri objects. This provides a more consistent and flexible model for referencing endpoints.

The WSE 2.0 interfaces for transports and channels have also been reworked substantially. This will require you to rework any custom channels you may have implemented on the Technology Preview release.

Check out the September 2003 installment of The XML Files for more details on the WSE 2.0 messaging API.

WSDL Support

Web Services Enhancements 2.0 now provides full support for WSDL, both in terms of generating and consuming definitions. WSE 2.0 automatically generates WSDL definitions for SoapService endpoints when requested.

If the SoapService endpoint is bound to an HTTP endpoint, you can request the WSDL definition by appending "?wsdl" to the endpoint address. This is the same approach used in the ASP.NET Web services framework. Assuming the SoapService implementation shown in Figure 8 is bound to http://localhost/contactservice/contactservice.ashx, you can retrieve its WSDL definition using http://localhost/contactservice/contactservice.ashx?wsdl.

The "?wsdl" approach doesn't make sense for protocols such as TCP that don't support the notion of a query string. Hence, WSE 2.0 also provides a generic approach for requesting WSDL definitions that can be used with any transport. Simply send a SOAP message with an action of http://schemas.microsoft.com/wse/2003/06/RequestDescription and an empty body, as illustrated here:

<soap:Envelope xmlns:wsa="..." xmlns:soap="..." ...>
  <soap:Header>
      <wsa:Action>http://schemas.microsoft.com/wse/2003/06/
         RequestDescription</wsa:Action>
       •••
  </soap:Header>
  <soap:Body />
</soap:Envelope>
And the WSE 2.0 endpoint will return the WSDL definition in the response SOAP message.

WSE 2.0 also ships with a new command-line tool named WseWsdl2.exe that will automatically generate SoapClient proxy classes from WSDL definitions. The default installation puts this tool in C:\Program Files\Microsoft WSE\v2.0\Tools\Wsdl. WseWsdl2.exe currently supports HTTP and TCP endpoints. The following command illustrates how to generate a proxy for a TCP endpoint, which must be running for this to work:

C:> wsewsdl2.exe soap.tcp://localhost/contactservice proxy.cs
In this case, the tool will automatically send a RequestDescription message to the endpoint, retrieve the WSDL from the response, and use it to generate the proxy class.

Security Enhancements

WSE 2.0 now supports the OASIS WS-Security standard, which primarily affects the message format but not the overall functionality introduced by the prior version of WS-Security. Most of the remaining security enhancements are a result of work by Microsoft to simplify and automate the most common Web services security tasks.

Let's take authentication for example. WSE 1.0 required you to write an IPasswordProvider class to manually authenticate incoming UsernameToken elements. WSE 2.0, on the other hand, automatically authenticates incoming UsernameToken elements against Windows when the password is supplied in clear text. The IPasswordProvider approach has been completely reworked around a new interface named ISecurityTokenManager. You use ISecurityTokenManager to implement custom authentication schemes as well as binary and custom XML security tokens. WSE 2.0 ships with several samples that illustrate how you can accomplish this.

WSE 2.0 has improved overall security token support. WSE 2.0 makes it possible to sign and encrypt messages using any security token, including UsernameToken, and it provides a way to automatically generate DerivedKeyTokens that are more secure. WSE 2.0 also adds support for Kerberos security tokens. The new Kerberos support, however, requires Windows Server™ 2003 or Windows XP Service Pack 1.

WSE 2.0 tries to simplify working with X.509 certificates, one of the more commonly used binary security tokens for signing and encrypting messages. WSE 2.0 ships with a new tool named WseCertificate2.exe that makes it easy to retrieve certificate details that you need to use in application configuration settings. The tool lets you retrieve the certificate's key identifier and set the private key properties (see Figure 9). It will come in handy when you start doing a lot of work with certificates.

Figure 9 WSE Certificate Tool
Figure 9 WSE Certificate Tool

WSE 2.0 also provides a complete role-based authorization model for securing access to a service. This is provided through the IPrincipal interface on security tokens. You can use the IsInRole method for authorization-specific user roles. This can also be done declaratively through policy statements.

WSE Security Settings Tool

Another major simplification provided by WSE 2.0 is the new WSE Security Setting Tool. This wizard facilitates the creation of policy files that describe the security requirements of the application. The policy, in turn, automates the security code you had to write to lock down your service in WSE 1.0 by requiring security tokens, signatures, or encryption. The policy can also automate the process of signing and encrypting messages with security tokens before they're sent by a client.

You can run the tool by opening the WSE Settings tool, navigating to the Policy tab, and selecting Enable Policy. Then press Add, enter the endpoint you want to describe (or leave <DefaultEndpoint>), and press OK to launch the Security Settings Wizard.

When you run the wizard, it asks you if you want to secure a client or a service application so it knows which options to provide. Then it asks you to specify signatures/encryption requirements for the request and response messages. Next, it asks you to specify the type of token to use for client authentication—you can choose Username, Kerberos, or X.509 Certificate. Finally, depending on your choices, it may ask you to select a certificate from the certificate stores on the machine, and then you're finished.

When you finish the wizard, it generates a new policy file with all of the specified settings and adds it to your project. It also registers the policy file in your application configuration file, as illustrated here:

   <configuration>
     •••
  <microsoft.web.services2>
    •••
    <policy>
      <cache name="policyCache.config" />
    </policy>
  </microsoft.web.services2>
</configuration>

This represents another change from the Technology Preview release: you no longer have to register "send" and "receive" policies separately in the configuration file. Now, both send and receive requirements are described in a single policy file. As a result, the policy mapping schema has changed substantially in WSE 2.0. However, being able to create policies through the wizard shields you from the differences.

WSE 2.0 also lets you write custom policy assertion handlers if you want to extend the built-in policy functionality offered in this release. Check out WS-Policy and WSE 2.0 Assertion Handlers for more details.

Once you've finished running the WSE Security Settings Tool wizard, your application will automatically support the security features you selected. It will enforce the specified requirements (such as the restriction that the message must contain a UsernameToken and a signature) when messages are received. It will also enforce policies when messages are sent, modifying the message if necessary to conform to the requirements. You can provide security tokens for WSE to use ahead of time through PolicyEnforcementSecurityTokenCache.GlobalCache. This greatly simplifies the process of integrating new security features into your applications since you no longer have to deal with writing the code necessary to make it happen.

Automatic Secure Conversation

WSE 2.0 completely automates the use of secure conversation between clients and services. By secure conversation, I mean support for WS-Trust and WS-SecureConversation. These specifications define how to derive a session token that can be used over multiple operations, improving efficiency.

You can enable secure conversation in a service application by running the WSE Security Settings Tool and selecting "Use Secure Conversation" on the first step. This will insert additional entries into the generated policy and it will inject the <tokenIssuer> element into the application configuration file, as illustrated here:

<configuration>
  •••
  <microsoft.web.services2>
    •••
    <tokenIssuer>
      <autoIssueSecurityContextToken enabled="true" />
    </tokenIssuer>
  </microsoft.web.services2>
</configuration>

You have to do the same thing in each client application that wants to take advantage of secure conversation. Once you've finished running the wizard in both applications, they'll begin using secure conversation to communicate with each other.

Next-Hop Routing

WSE 2.0 has moved away from the WS-Routing routing model introduced by WSE 1.0 due to various security issues. WSE 2.0 now supports a next-hop routing model, where routing decisions are made on a node-to-node basis without requiring an explicit routing path in the message header. As a result, all WS-Routing functionality has been removed and the new routing model is based entirely on WS-Addressing headers.

WSE 2.0 still relies on WS-Referral for next-hop routing instructions. The built-in router class is called Microsoft.Web.Services2.Messaging.SoapHttpRouter and can be registered in web.config the same way as before.

This change in routing behavior essentially makes it possible to route messages in a more secure way, without needing to modify the message en route. Check out the MSDN online article "Moving from WS-Routing to WS-Addressing Using WSE 2.0" for more details on routing and addressing.

Where are We?

WSE 2.0 is the only existing Microsoft solution that makes it possible to take advantage of the latest emerging Web services specifications. WSE 2.0 is full of new features and tools that simplify the overall development and configuration process. WSE 2.0 automates many of the tasks that WSE 1.0 required you to implement manually, making it easier to get things up and running quickly without sacrificing correctness or flexibility. If you'd like to dig in further, check out the samples that ship with WSE 2.0, available in both C# and Visual Basic® .NET.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, an education and content creation company, where he focuses on XML and Web services technologies. He's a contributing editor to MSDN Magazine and the author of several books, including Essential XML Quick Reference (Addison-Wesley, 2001). Reach him at http://www.pluralsight.com/aaron.


Test Run: Test Automation for ASP.NET Web Apps with SSL

Test Run
Test Automation for ASP.NET Web Apps with SSL
James McCaffrey

Code download available at: TestRun0408.exe (133KB)
Contents
The Test Automation Program
Setting up a Test SSL Server
Further Work

If you're encrypting user data with Secure Sockets Layer (SSL) over HTTP and you want to test your Web applications programmatically you'll find that the techniques are not widely known. In this month's column I'll show you how to set up a test SSL server and write test automation that verifies the functionality of a simple but representative Web application.

The Microsoft® .NET environment provides powerful tools to test ASP.NET Web applications that use the SSL security mechanism. To illustrate their use, I will set up a test SSL server and demonstrate a short but powerful program that automatically tests a sample Web application that uses HTTPS. Although the individual techniques are elegant and documented, in conversations with many of my colleagues I discovered that the overall process of testing Web applications over SSL is relatively unfamiliar. The best way to show you where I'm headed is with two screen shots. Figure 1 shows the simplistic but representative ASP.NET Web application that I'll test.

Figure 1 An ASP.NET Web App
Figure 1 An ASP.NET Web App

Notice that I'm using an SSL connection because I am sending sensitive credit card information over the Internet. (Notice the "https://" protocol in the address bar and the small lock icon in the status bar.)

Now, just imagine what it would be like to test this application manually. You would have to enter hundreds or even thousands of names, quantities, and credit card numbers into the Web page, visually examine each resulting confirmation code, manually check each code against a list of expected results to determine a pass or fail result, and then record the results in some form such as an Excel spreadsheet or text document. This would be time consuming, error prone, inefficient, and just plain boring.

A much better approach is to harness the powerful capabilities of the .NET Framework to write automation that programmatically sends the test data using SSL, then examines the response stream looking for an expected confirmation code. The Console application shown in Figure 2 does just that.

Figure 2 The Testing App
Figure 2 The Testing App

As you can see, automated test case 001 corresponds to the manual test shown in Figure 1. The last name "Smith", quantity "3", and credit card number "1234 5678 9012" are encrypted and programmatically posted to the Web application using HTTP with SSL. The test program retrieves the HTTP response stream and searches it for "C3-57-ED-DA-8B". In this case, the expected confirmation code was found in the response stream so the test automation records a PASS result. In the three sections of this column that follow I will present and explain the test program that generated the output shown in Figure 2, demonstrate how to set up a test server that accepts SSL requests, and discuss how you can extend the techniques presented here to meet your own needs.

Before I move on to the test automation program, let's take a quick look at the sample Web application. As you can see in Figure 1, there are three TextBox controls; I accepted the Visual Studio® .NET default IDs of TextBox1, TextBox2, and TextBox3 to hold the user's last name, widget quantity, and credit card number, respectively. The application displays its message in the Label5 control. I'll need to know this information when I write the test automation. I'll also need to know how the order confirmation code is generated so that I can determine expected results for my test cases. Here is the heart of the code for the Web application under test:

if (TextBox3.Text.Length == 0)
  Label5.Text = "Please enter credit card number";
else
{
  byte[] input = Encoding.Unicode.GetBytes(TextBox3.Text);
  byte[] hashed;
  using(MD5 m = new MD5CryptoServiceProvider())
  {
      hashed = m.ComputeHash(input);
  }
  Label5.Text = "Thank you. Your confirmation code is " +
    BitConverter.ToString(hashed).Substring(0,14);
}

To simulate the generation of a confirmation code, I just take the credit card number entered by the user, produce an MD5 hash of it, and grab the leftmost 14 characters of the hash. In an actual production system it is likely that you would want to generate a confirmation code in a more complicated manner. In such a case it might be tricky to determine the expected result. One strategy you should definitely not use, however, is to determine the expected result by calling the application under test. This would destroy the validity of your test because you'd just be checking to see if your test automation returns the same result as your application.

The Test Automation Program

The test automation program is surprisingly short. The entire code is presented in Figure 3. Although the techniques required to programmatically post data to an ASP.NET Web application are documented in the MSDN® Library, there are several tricks you need to watch out for.

Figure 3 The Test Automation Program (Close Figure 3)
using System;
using System.Web;  // for HttpUtility class
using System.Text; // for Encoding and StringBuilder classes
using System.Net;  // for HttpWebRequest class
using System.IO;   // for StreamReader class

namespace Run
{
  class Class1
  {
    [STAThread]
    static void Main(string [] args)
    {
      Console.WriteLine("\nStarting Test Run\n");

      string url = "https://localhost/LitwareOrder/Order.aspx";
      string viewstate =
        HttpUtility.UrlEncode(
        "dDw0MDIxOTUwNDQ7Oz6E/7ailqx8X9zCUfpbWTPybfS4MA==");
      string line;
      string[] tokens;
      StringBuilder data = new StringBuilder();
      byte[] buffer;

        using(FileStream fs = new FileStream(args[0], FileMode.Open))
      {
        StreamReader tc = new StreamReader(fs);
        while ((line = tc.ReadLine()) != null)
        {
          tokens = line.Split(':');
          data.Length = 0;
          data.Append("TextBox1=" + tokens[1]);  // Last name
          data.Append("&TextBox2=" + tokens[2]); // Quantity
          data.Append("&TextBox3=" + tokens[3]); // Credit card number
          data.Append("&Button1=clicked");
          data.Append("&__VIEWSTATE=" + viewstate);

          buffer = Encoding.UTF8.GetBytes(data.ToString());

          HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
          req.Method = "POST";
          req.ContentType = "application/x-www-form-urlencoded";
          req.ContentLength = buffer.Length;
          req.CookieContainer = new CookieContainer(); // enable cookies

          using (Stream reqst = req.GetRequestStream())
          {
            reqst.Write(buffer, 0, buffer.Length);
          }

          Console.WriteLine("====================");
          Console.WriteLine("\nTest case ID = " + tokens[0] );
          Console.WriteLine("Posting " + tokens[1] + ", " + tokens[2] +
                          "," + tokens[3] );
          Console.WriteLine("To " + url);
          Console.WriteLine("Looking for " + tokens[4] );

          using (HttpWebResponse res = (HttpWebResponse)req.GetResponse())
          {
            string result;
            using(Stream resst = res.GetResponseStream())
            {
              result = new StreamReader(resst).ReadToEnd();
            }
            //Console.WriteLine(result);

            if (result.IndexOf(tokens[4]) >= 0)
              Console.WriteLine("PASS");
            else
              Console.WriteLine("FAIL");
              Console.WriteLine("");
          }
        } // while loop

        Console.WriteLine("====================");
        Console.WriteLine("Done");
      }
      Console.ReadLine();
    } // Main()
  } // class Class1
} // namespace Run

I decided to write a C# console application as my test program. It's usually a good idea to use the same language for your test automation as the system under test. However, the well-planned design and integration of the .NET environment means that you can safely use Visual Basic® .NET or any other .NET-compliant language. In general, console applications are the program type best suited for test automation. Although a slick test automation program with a nice user interface makes a great impression on your users, test automation is a tool, not a personal showcase, and console applications can be integrated into a build system more easily than a GUI application can.

The overall structure of the test automation is quite simple. I store test case data in a simple text file. Each line of data represents a single test case. Here is the test case file I used:

001:Smith:3:1234 5678 9012:C3-57-ED-DA-8B
002:Baker:2:1111 2222 3333:CE-81-8C-2F-94
003:Gates:9:9999 9999 9999:95-D6-05-31-8A

Information is separated by a colon character (:). I could have used any character as a delimiter, but it's important to avoid characters that appear in the actual test case data. The first field holds a test case ID, the second field holds the last name, the third field holds the quantity of widgets, the fourth field holds the credit card number, and the fifth field holds the expected confirmation code. If you don't want to use a text file, XML files and SQL tables make good test case storage alternatives.

The basic structure of my test automation is tied to the structure of my test case file. In pseudocode the plan looks like this:

loop
  read a test case line
  parse out test case data
  build up data to post to application
  convert post data to a byte array
  post the data
  retrieve the response stream
  if response stream contains expected confirmation code
    log "pass" result
  else
    log "fail" result
end loop

I begin by declaring the namespaces I'll use. This lets me avoid fully qualifying each .NET class and object and also documents what functionality the test automation program will employ:

using System;
using System.Web;
using System.Text;
using System.Net;
using System.IO;

The System.Web namespace contains the HttpUtility class that I will use to convert special characters into escape sequences. Because a default Console Application does not reference System.Web.dll, which houses this class, I had to manually add a project reference to the file. The System.Text namespace has an Encoding class that I'll use to perform byte array manipulation. The System.Net namespace contains the HttpWebRequest class, which is the fundamental class I'll use to post data to the ASP.NET Web application. I use the System.IO namespace because I will use data streams for the HTTP response over SSL and because I'll need it to read test case data from a text file. Note that the using directive permits the use of types in a namespace such that you do not have to qualify the use of a type in that namespace.

Next, after displaying a brief start message to the command shell, I declare the key variables that the test automation uses:

string url = "https://localhost/LitwareOrder/Order.aspx";
string viewstate = HttpUtility.UrlEncode(
    "dDw0MDIxOTUwNDQ7Oz6E/7ailqx8X9zCUfpbWTPybfS4MA==");
string line;
string[] tokens;
StringBuilder data = new StringBuilder();
byte[] buffer;
string proxy = null;

The purpose of most of these variables should be clear from their names but the viewstate variable may be new to you, so I will explain it shortly. Now, I open the test case file specified on the command line and read it line by line:

using(FileStream fs = new FileStream(args[0], FileMode.Open))
{
    StreamReader tc = new StreamReader(fs);
    while ((line = tc.ReadLine()) != null)
    {
      // parse line, post data, get response
      // determine pass or fail, log result
    }
}

There are many alternative ways to design the automation, but this simple structure has proven to be robust on several large projects. The next step is to parse each field of the test case data and build up a data string containing name-value pairs:

tokens = line.Split(':');
data.Length = 0;
data.Append("TextBox1=" + tokens[1]);   // Last name
data.Append("&TextBox2=" + tokens[2]); // Quantity
data.Append("&TextBox3=" + tokens[3]); // Credit card number
data.Append("&Button1=clicked");
data.Append("&__VIEWSTATE=" + viewstate);

I use the String.Split method to break up the test case line and store each field in the tokens array. The test case ID is stored in tokens[0], the user last name is stored in tokens[1], the widget quantity is stored in tokens[2], and the credit card number is stored in tokens[3]. For clarity, I could have created additional string variables with descriptive names like "caseID" and "lastName" and then copied values from the tokens array into them like so:

caseID = tokens[0];
lastName = tokens[1];
// etc.
But I wanted to keep the number of variables I used to a minimum. Traditional Web servers expect POST data in name-value pairs connected by an ampersand, like this:
lastName=Smith&quantity=3&creditCardNo=123456789012
However, ASP.NET extends this idea. In this example, there are five name-value pairs. The first pair, TextBox1=tokens[1], is what you might expect. It assigns the current test case last name value (stored in tokens[1]) to the control with id attribute "TextBox1". The second pair, TextBox2=tokens[2], and the third pair, TextBox3=tokens[3], should also make sense to you assigning the widget quantity and credit card number, respectively. The next pair, "Button1=clicked", is probably not what you expected if you have experience posting data to traditional ASP pages. Because Button1 is a server-side control I must post it to keep the ViewState value synchronized, as I'll explain in a moment. Assigning any value to it has no effect, so I could have written "Button1=" by itself. I like to use a value such as "clicked" anyway because it makes the code more readable. The fifth name-value pair is the __VIEWSTATE pair (note the two underscores); this is the real key to programmatically posting to ASP.NET servers.

What is this ViewState value? Even though HTTP is a stateless protocol—each Request-Response pair is an isolated transaction—ASP.NET works behind the scenes to create a stateful environment. One of the ways ASP.NET does this is through the use of an HTML hidden input named __VIEWSTATE. It is a Base64-encoded string value that represents the state of the page when last processed by the server. In this way, pages can maintain state by retaining values between successive calls. To correctly post information to an ASP.NET application, I must send the ViewState value to the server. Recall that I set the value of viewstate as:

string viewstate = HttpUtility.UrlEncode(
    "dDw0MDIxOTUwNDQ7Oz6E/7ailqx8X9zCUfpbWTPybfS4MA==");

Where did this value come from? The easiest way to get the initial ViewState value for a Web application is to just launch Microsoft Internet Explorer, get the page, and do a View | Source from the menu bar. It is important that you get the initial ViewState value because if you reload the page, the ViewState value will change and your programmatic post program will generate a server error. The raw ViewState value needs processing from the UrlEncode method. The UrlEncode method converts characters invalid in a URL into escaped sequences. For example, "=" characters are converted into %3D sequences.

After I have the ViewState value, I can construct the full data string that I will post to the Web application. Next I must convert this string into a byte array because the method that will later post the data requires the data to be stored as bytes:

buffer = Encoding.UTF8.GetBytes(data);

The GetBytes method is a member of the Encoding class in the System.Text namespace. In addition to the UTF8 property, there are ASCII, Unicode, and UTF7 properties, too. Now I instantiate an HttpWebRequest object and assign values to its properties:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = buffer.Length;
req.Proxy = new WebProxy(proxy, true);
req.CookieContainer = new CookieContainer();

Notice that, following WebRequest's factory pattern, I use an explicit Create method rather than calling a constructor with the new keyword. I use the POST method because I am sending form data. I set the ContentType property to "application/x-www-form-urlencoded". This is a MIME type that you can think of as a magic string that tells the ASP.NET server to expect form data. I set the ContentLength property to the number of bytes of post data that I stored in the byte array buffer earlier.

I could have left out the Proxy property in this example if I were not sending data through a proxy server machine. The Boolean true parameter in the WebProxy constructor means to ignore the proxy for local addresses. Assigning a value to the CookieContainer property is required in order to have cookies returned in the Cookies property of the HttpWebRequest returned by GetResponse. Notice that I assign an empty CookieContainer object. This is one of the little details that caused me a lot of trouble when I was figuring out this technique.

Before I can post our request to the ASP.NET application, I have to add the post data to the Request object like this:

using (Stream reqst = req.GetRequestStream())
{
    reqst.Write(buffer, 0, buffer.Length);
}

Next, after printing some information about what I am posting, I retrieve the resulting response stream from the server:

using(HttpWebResponse res = (HttpWebResponse)req.GetResponse())
{
  string result;
  using(Stream resst = res.GetResponseStream())
  {
      result = new StreamReader(resst).ReadToEnd();
  }
  //Console.WriteLine(result);
}

You might have expected, as I did, that I would explicitly send the request with a statement like req.Send(data). However using HttpWebRequest.GetRequestStream actually opens the connection to the server, and HttpWebRequest.GetResponse retrieves the HttpWebResponse object that represents the server's response. (If GetRequestStream is not used, GetResponse actually makes the connection to the server, as well.) I grab the entire response stream using ReadToEnd and save it into a string variable named "result". You can also read the response line-by-line using the ReadLine method. Notice I commented out a statement that would display the entire response stream to the command shell. If you are new to this kind of programming, you will find it instructive to uncomment that line so you can see the entire response stream.

Finally, I examine the response stream to see if the expected confirmation code (stored in tokens[4]) is in the stream:

if (result.IndexOf(tokens[4]) >= 0)
  Console.WriteLine("PASS");
else
  Console.WriteLine("FAIL");
If I find the expected value, I log a PASS result to the shell. Of course, you could write test case results to a text file, XML file, or SQL table, if you prefer.

Setting up a Test SSL Server

Until recently, it was quite a chore to set up a test Web server with SSL enabled. You can purchase a "real" SSL certificate from one of several providers, but that takes time and money. Another option is to generate a self-signed SSL certificate using the makecert.exe utility that is part of the .NET Framework Tools, then install it onto your Web server. But now there is a much simpler way.

The IIS 6.0 Resource Kit (available for download from Windows Deployment and Resource Kits) contains several valuable tools including one—selfssl.exe—which makes it very easy to create and install a self-signed SSL certificate for testing purposes. The screenshot in Figure 4 shows exactly how I did this.

Figure 4 Create a Self-Signed Certificate (Close Figure 4)

The key is to use the /T switch so that the local browser will trust the certificate and also to use the /N switch to specify "localhost" as the Common name. Amazingly, this is all you need to test with HTTPS directly on the Web server. If you want to test HTTP with SSL from a remote client machine, the first time you manually browse to the test server you will get a Security Alert dialog asking if you want to proceed. If you click on the View Certificate button, then click on the Install Certificate button, you will enter a wizard. If you accept all the defaults in the wizard, then after you finish installing the certificate the client will be able to access the test server without the warning dialog and your test automation will run from the client.

Although the selfssl.exe tool is part of the IIS 6.0 Resource Kit and does not explicitly support earlier versions of IIS, my colleagues and I have successfully experimented with it on IIS 5.0. I have also used the makecert.exe tool to generate a self-signed x.509 certificate that can be used for testing. The MSDN Library has instructions for the makecert.exe tool at Certificate Creation Tool, but using the selfssl.exe tool is easier.

After you are finished testing with your self-signed SSL certificate you will want to remove it to prevent possible interaction effects on your test server. The easiest way to remove the certificate is by using the Microsoft Management Console (MMC). Launch MMC and add the Certificates snap-in for a Computer Account to manage the Local Computer. Now you should expand the Certificates store and then expand the Personal folder. After selecting the Certificates folder, your self-signed certificate will be displayed and you can delete it.

Further Work

There are several ways you can extend the techniques presented in this month's column. I hardcoded some information (for example, the test URL) which you'll want to pull out, and you'll probably find it useful to parameterize to make your test system more flexible. An interesting extension would be to programmatically determine the initial ViewState value. Recall that I launched the browser manually then viewed the source to determine the initial ViewState value, then hardcoded that value into the automation. This works, but every time there is a change to the Web application code base, you'll have to get the new ViewState value. A better approach is to use the WebClient class from the System.Net namespace to programmatically request the initial Web application page, parse the response stream for the __VIEWSTATE value, and then assign that value to the viewstate variable. This adds an additional request/response round-trip to each test case call, but it greatly increases the flexibility of your test automation.

The technique presented in this column nicely complements testing that you can perform with the Microsoft Application Center Test (ACT) tool. ACT is designed to stress test Web servers and analyze the performance of Web apps so you can deal with scalability problems. However, ACT was not designed to handle functional verification like the technique shown here was designed to do.

In this column I used the HttpWebRequest class to programmatically post data to the ASP.NET Web application under test, but there are several alternatives you can use. At a lower level of abstraction you can post data using the Sockets class in the System.Net.Sockets namespace. At a slightly higher level of abstraction you can use the WebClient class in the System.Net namespace. All three techniques work fine, but most of my colleagues and I prefer the HttpWebRequest class. However, you may find that it is matter of personal preference and scenario.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Wicked Code: Foiling Session Hijacking Attempts

New information has been added to this article since publication.
Refer to the Editor's Update below.

Wicked Code
Foiling Session Hijacking Attempts
Jeff Prosise

Code download available at: WickedCode0408.exe (118KB)
Contents
Introducing SecureSessionModule
Deploying SecureSessionModule
Caveats
Conclusion

Let's face it: every minute of every day, someone, somewhere, is patrolling the Web looking for sites to hack. ASP.NET developers must constantly be on their guard to ensure attempted hacks can't be successful. That means constraining and validating user input, accessing databases securely, storing sensitive data securely, and generally writing secure code that repels rather than accommodates these malevolent hackers.

A classic form of hack attack that ASP.NET sites must defend against is session hijacking. Simply put, session hijacking entails connecting to a Web site and accessing someone else's session state. The severity of the damage incurred depends on what's stored in session state. If sessions hold shopping cart information and users are required to verify their identities before checking out, session hijacking might not be very damaging. If sessions contain credit card numbers or similarly sensitive data that can be presented back to the user, you really have to watch out.

Session hijacking attacks are typically perpetrated in one of two ways: session ID guessing and stolen session ID cookies. Session ID guessing involves gathering a sample of session IDs and "guessing" a valid session ID assigned to someone else. ASP.NET apps tend to be much less susceptible to this form of session hijacking because ASP.NET uses highly random 120-bit numbers for its session IDs. Unless you replace ASP.NET session IDs with IDs of your own (last summer I met one developer who had done just that and used sequential session IDs, which are an open door to hackers), you have nothing to fear from session ID guessing.

Stolen session cookies are another matter. Secure Sockets Layer (SSL) can be used to protect session ID cookies on the wire, but few sites restrict session ID cookies to encrypted connections. Even if you use SSL, session ID cookies can be stolen in other ways, notably through cross-site scripting attacks, man-in-the-middle attacks, and gaining physical access to the cookie stores on victims' PCs. Furthermore, executing a successful session hijacking attack with a stolen session ID cookie requires little skill on the part of the attacker. The reason is simple: ASP.NET encodes no information in a session ID cookie other than the session ID. If it receives a cookie containing a valid session ID, ASP.NET connects to the corresponding session, no questions asked.

It's virtually impossible to build a foolproof defense against attacks that rely on stolen session ID cookies, but you can take steps to make it harder. Some Web sites, for example, encode heuristic information such as IP addresses in their session IDs. That doesn't prevent session hijacking altogether—after all, some hackers possess the means to spoof IP addresses, and IP addresses are not a reliable means for identifying callers anyway—but it does raise the bar. Security is all about raising the bar. The harder you make it for a hacker to execute a successful attack, the less likely it is that successful attacks will occur.

Introducing SecureSessionModule

Rather than wishing that ASP.NET made it more difficult to use stolen session ID cookies, I decided to do something about it. I created an HTTP module, SecureSessionModule, that sits in the ASP.NET HTTP pipeline and monitors incoming requests and outgoing responses for session ID cookies (see Figure 1). [ Editor's Update - 7/20/2004: Figure 1 has been updated so as not to provide too much information to an attacker.]

Figure 1 SecureSessionModule.cs (Close Figure 1)
using System;
using System.Web;
using System.Web.Security;
using System.Configuration;
using System.Security.Cryptography;
using System.Runtime.Serialization;
using System.Globalization;
using System.Text;

public class SecureSessionModule : IHttpModule
{
    private static string _ValidationKey = null;

    public void Init (HttpApplication app)
    {
        // Initialize validation key if not already initialized
        if (_ValidationKey == null)
            _ValidationKey = GetValidationKey ();

        // Register handlers for BeginRequest and EndRequest events
        app.BeginRequest += new EventHandler (OnBeginRequest);
        app.EndRequest += new EventHandler (OnEndRequest);
    }

    public void Dispose () {}

    void OnBeginRequest (Object sender, EventArgs e)
    {
        // Look for an incoming cookie named "ASP.NET_SessionID"
        HttpRequest request = ((HttpApplication) sender).Request;
        HttpCookie cookie = GetCookie (request, "ASP.NET_SessionId");

        if (cookie != null) {
            // Throw an exception if the cookie lacks a MAC
            if (cookie.Value.Length <= 24)
                throw new InvalidSessionException
                    ("Access Denied"); // don't tell bad guys too much

            // Separate the session ID and the MAC
            string id = cookie.Value.Substring (0, 24);
            string mac1 = cookie.Value.Substring (24);

            // Generate a new MAC from the session ID and requestor info
            string mac2 = GetSessionIDMac (id, request.UserHostAddress,
                request.UserAgent, _ValidationKey);

            // Throw an exception if the MACs don't match
            if (String.CompareOrdinal (mac1, mac2) != 0)
                throw new InvalidSessionException
                    ("Access Denied"); // don't tell bad guys too much

            // Strip the MAC from the cookie before ASP.NET sees it
            cookie.Value = id;
        }
    }

    void OnEndRequest (Object sender, EventArgs e)
    {
        // Look for an outgoing cookie named "ASP.NET_SessionID"
        HttpRequest request = ((HttpApplication) sender).Request;
        HttpCookie cookie = GetCookie (
            ((HttpApplication) sender).Response, "ASP.NET_SessionId");

        if (cookie != null) {
            // Add a MAC
            cookie.Value += GetSessionIDMac (cookie.Value,
                request.UserHostAddress, request.UserAgent,
                _ValidationKey);
        }
    }

    private string GetValidationKey ()
    {
        string key = ConfigurationSettings.AppSettings
            ["SessionValidationKey"];
        if (key == null || key == String.Empty)
            throw new InvalidSessionException
                ("SessionValidationKey missing");
        return key;
    }

    private HttpCookie GetCookie (HttpRequest request, string name)
    {
        HttpCookieCollection cookies = request.Cookies;
        return FindCookie (cookies, name);
    }

    private HttpCookie GetCookie (HttpResponse response, string name)
    {
        HttpCookieCollection cookies = response.Cookies;
        return FindCookie (cookies, name);
    }

    private HttpCookie FindCookie (HttpCookieCollection cookies,
        string name)
    {
        int count = cookies.Count;

        for (int i=0; i<count; i++) {
            if (String.Compare (cookies[i].Name, name, true,
                CultureInfo.InvariantCulture) == 0)
                return cookies[i];
        }

        return null;
    }

    private string GetSessionIDMac (string id, string ip,
        string agent, string key)
    {
        StringBuilder builder = new StringBuilder (id, 512);
        builder.Append (ip.Substring (0, ip.IndexOf ('.',
            ip.IndexOf ('.') + 1)));
        builder.Append (agent);

        using (HMACSHA1 hmac = new HMACSHA1
            (Encoding.UTF8.GetBytes (key))) {
            return Convert.ToBase64String (hmac.ComputeHash
               (Encoding.UTF8.GetBytes (builder.ToString ())));
        }
    }
}

[Serializable]
public class InvalidSessionException : Exception
{
    public InvalidSessionException () :
        base ("Session cookie is invalid") {}

    public InvalidSessionException (string message) :
        base (message) {}

    public InvalidSessionException (string message,
        Exception inner) : base (message, inner) {}

    protected InvalidSessionException (SerializationInfo info,
        StreamingContext context) : base (info, context) {}
}

Figure 2 shows how SecureSessionModule works. First, it checks every outgoing response for a session ID cookie issued by ASP.NET's SessionStateModule. When it sees such a cookie, SecureSessionModule modifies it by appending a hashed message authentication code (MAC) to the session ID. The MAC is generated from the session ID, the network address portion of the requestor's IP address (for example, the 192.16 in 192.16.0.14), the User-Agent header received in the request, and a secret key stored on the server. The Framework's System.Security.Cryptography.HMACSHA1 class makes the task of generating the MAC really quite easy.Why use the network address instead of the full IP address? The node address of users that access the Internet through public proxy servers such as AOL's can change in every request, but the network address should not.

Figure 2 SecureSessionModule
Figure 2 SecureSessionModule

Second, SecureSessionModule examines every incoming request for an ASP.NET session ID cookie. Before allowing a request containing a session ID cookie to continue through the pipeline, SecureSessionModule validates the cookie by regenerating the MAC from the requestor's IP address, the User-Agent header, and the secret key. If the freshly computed MAC matches the one in the cookie, the MAC is stripped from the cookie and the request is allowed to proceed. If the MACs don't match, SecureSessionModule throws an InvalidSessionException, as shown in Figure 3.

Figure 3 Error Showing Generated Key (Close Figure 3)

The net result is that once a session ID cookie is issued, it's only considered valid if it's submitted from the same network address and with the same User-Agent header. An attacker who steals a session ID cookie can only use it if she can spoof IP addresses and HTTP headers. Both are certainly possible, but spoofing of this sort requires a higher skill level on the part of the attacker. In addition, getting a response back from a request with a spoofed IP address is much harder than simply submitting the request in the first place and can be defeated with proper egress filtering. It's impossible for the attacker to simply replace the hash in the cookie with one generated from her own IP address and User-Agent header without the secret key used to generate the MAC. To the extent that the key can be secured, casual hackers will find it difficult indeed to use stolen session IDs for nefarious purposes.

Deploying SecureSessionModule

Deploying SecureSessionModule is as simple as copying SecureSessionModule.dll into the application root's bin subdirectory and registering it in Web.config, like so:

<configuration>
  <appSettings>
    <add key="SessionValidationKey"
      value="DAD4D476F80E0148BCD134D7AA5C61D7" />
  </appSettings>
  <system.web>
    <httpModules>
      <add name="SecureSession"
        type="SecureSessionModule,
          SecureSessionModule" />
    </httpModules>
  </system.web>
</configuration>

The SessionValidationKey value in the <appSettings> section of Web.config is required. This is the "secret key" used to generate the MAC; SecureSessionModule looks for it at load time and throws an exception if it's not there. The key should be unique for every application, and it should be long and random. I used a simple tool built around the .NET Framework RNGCryptoServiceProvider class to generate the one in Figure 3. You should use a similar tool to maximize randomness. In addition, if deployed on a Web farm, SessionValidationKey should be the same on every server. Of course, storing plaintext security keys in configuration files poses risks of its own. For an added measure of security, consider encrypting the secret key. The article at Building Secure ASP.NET Applications: Authentication, Authorization, and Secure Communication describes how to use the Windows® Data Protection API (DPAPI) from ASP.NET. DPAPI is ideal for encrypting secrets in configuration files because it offloads the problem of key management—specifically, storing decryption keys—to the operating system itself.

Ideally, SecureSessionModule would use the same secret key that ASP.NET uses for hashing: the value of the validationKey attribute in Machine.config's <machineKey> element. However, there is no public API for retrieving this key, and after inspecting what the Framework does to retrieve validationKey, I elected not to duplicate the logic in the Framework to avoid potential incompatibilities with future versions of .NET. The new configuration API coming in version 2.0 of the .NET Framework will correct this omission and provide a documented means for reading the ASP.NET validation key.

Once deployed, SecureSessionModule works passively—no intervention is required. If you'd like, include an Application_Error method in Global.asax to log InvalidSessionExceptions in the Windows event log or elsewhere. That way you can check the log every morning over your first cup of coffee and find out if someone has been trying to spoof your server with stolen session IDs. Such a method might look like this:

<%@ Import Namespace="System.Diagnostics" %>

<script language="C#" runat="server">
void Application_Error(Object sender, EventArgs e)
{
    // Write an entry to the event log
    EventLog log = new EventLog ();
    log.Source = "My ASP.NET Application";
    log.WriteEntry (Server.GetLastError().ToString(),
        EventLogEntryType.Error);
}
</script>

Caveats

Before deploying SecureSessionModule on a production Web server, you should consider several potential issues.

First, SecureSessionModule is not 100 percent effective in detecting illicit session ID cookies. If the attacker has the same network address as the victim (if both, for example, use the same proxy server) or can spoof the victim's network address, then User-Agent headers are the last line of defense. And User-Agent headers are easily spoofed by someone aware that User-Agent headers are being used to validate session IDs.

Second, if for some reason a user's network address or User-Agent headers vary from request to request, that user will lose access to her session state. Third, SecureSessionModule doesn't work with cookieless session state. It assumes session IDs are passed in cookies, not URLs.

Finally, SecureSessionModule hasn't been tested in a production environment. If you use it, I'd love to hear from you—especially about any glitches that arise. Note that Microsoft has considered building similar protections into ASP.NET, but has always shied away from it for backward compatibility concerns.

Conclusion

Session hijacking remains a serious threat to security. SecureSessionModule raises the bar for hackers who hijack sessions using stolen session IDs by factoring evidence about the session owner into the session ID cookie. That evidence isn't conclusive because neither network addresses nor User-Agent headers can be used reliably to distinguish one user from another, but it nonetheless places an additional hurdle in the path of hackers who are actively seeking to compromise your Web servers by connecting to other users' sessions and grabbing their data.

Send your questions and comments to Jeff at wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect, a software consulting and education firm that specializes in Microsoft .NET. Contact Jeff at wicked@microsoft.com.


Advanced Basics: P2P Comm Using Web Services

Advanced Basics
P2P Comm Using Web Services
Carl Franklin

Code download available at: AdvancedBasics0408.exe (195KB)
Contents
Peer-to-Peer and Web Services
Building the Client
Maintaining State
Logging In Users
Sending and Receiving

Iwanted to use my first Advanced Basics column as an opportunity to strike out into new territory, to do something I haven't seen extolled much in the literature, so I've built a Windows® Forms chat program that uses Web services to communicate with other peers. It's a bit of a sneaky implementation, though, because along the way I'll call Web services asynchronously, synchronize access to the UI, and explain how powerful the Shared keyword can be.

Peer-to-Peer and Web Services

Traditionally, Web services have lived in a stateless, client/server world. With Web Services Enhancements (WSE) 2.0, with toolkits from other vendors, and certainly with "Indigo" (code name for the next-generation of .NET technologies from Microsoft for building and running connected systems), Web services are not restricted to client/server. Peer-to-peer applications exhibit a more connected nature. There are two P2P models. One is just like it sounds: my program contacts your program directly and we talk. Maybe I got your IP address and port number from a server that catalogs available services and their corresponding IP addresses (a model similar to the original Napster), or maybe I discovered you through some intelligent software like WinMX.

In the other peer-to-peer model, all peers connect to a server and talk to each other through the server, the way Internet Relay Chat (IRC) works. IRC is a chat system that defines client-to-server and server-to-server communications protocols. A client connects to the server, announces himself or logs in, and retrieves the names and IDs of the other clients. To send a message to another client, the client sends it to the server and tells the server which other client the message is for; then the server sends the recipient a new message.

That's sort of what I'm trying to accomplish here using Web services over HTTP. A client connects, sends a request, gets a response, and the connection is closed by the server. For a client to get an event notification from a server using HTTP has always required polling on the part of the client, and that is exactly what my sample app does. The Web service I'll build here defines eight methods, which are listed in Figure 1.

Figure 1 My Web Service Methods (Close Figure 1)

Method | Description
Hello | Simple PING
LogIn | Self explanatory
GetUsers | Returns a list of users currently logged in
GetNextMessage | Polls for the next message addressed to me (the client)
SendMessage | Sends a message to a single client
SendMessageToAll | Sends a message to all clients logged in
LogOut | You guessed it
ResetAll | Clears everything, thus all users must log in again

Building the Client

The WebChat client program looks deceptively simple (see Figure 2). The listbox on the left shows the names of the clients that are currently connected. The large listbox on the right shows the messages as they come through. Below that is a multiline textbox where you can type in your message. A Send All button sends the message to all clients, and the Send button sends the message to the selected client. When the client loads, a simple user name/password login dialog is displayed to collect the user's credentials and calls the LogIn WebMethod (see Figure 3).

Figure 3 LogIn Dialog Code (Close Figure 3)
Imports p2pTools
Public Class frmLogin
    Inherits System.Windows.Forms.Form

    Private server As New net.franklins.www.Service1

    Private Sub btnLogIn_Click(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles btnLogIn.Click
        Dim resp As LogInReturn
        Dim bytes() As Byte = server.LogIn(txtUserName.Text, _
            txtPassword.Text)
        resp = CType(Utils.DeSerialize(bytes), LogInReturn)
        _clients = resp.Clients
        _peerid = resp.MyPeerID
        _hash = resp.UserHash
        Me.Hide()
    End Sub

    Private Sub Login_Load(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles MyBase.Load
        server.BeginHello(AddressOf GotResponse, Nothing)
    End Sub

    Private Sub GotResponse(ByVal ar As IAsyncResult)
    End Sub

    Private _hash As String
    Public ReadOnly Property UserHash() As String
        Get
            Return _hash
        End Get
    End Property

    Private _peerid As String
    Public ReadOnly Property PeerID() As String
        Get
            Return _peerid
        End Get
    End Property

    Private _clients As ClientSideUserCollection
    Public ReadOnly Property Clients() As ClientSideUserCollection
        Get
            Return _clients
        End Get
    End Property
End Class
Figure 2 WebChat Client
Figure 2 WebChat Client

Let's jump over to the LogIn WebMethod in the p2pChat Web service application, shown in Figure 4. As you can see, the user name and password are sent in clear text over HTTP. For this reason, you might want to use a Secure Sockets Layer (SSL) connection in the real world. Using SSL is the easiest way to secure your Web service data in transit. There is a bit of overhead involved, but for a chat program you'll never notice it.

Figure 4 LogIn WebMethod (Close Figure 4)
<WebMethod()> Public Function LogIn(ByVal UserName As String, _
  ByVal Password As String) As Byte()

    '-- Do your authentication here

    '-- Does the user already exist?
    If StateManager.ExistsUserName(UserName) Then
        Throw New InvalidOperationException("User already exists")
    End If

    '-- Create a new client
    Dim PeerID As String = StateManager.AddClientAndReturnPeerID(UserName)

    '-- Return the PeerID, to be used for subsequent calls
    Dim resp As New LogInReturn
    resp.MyPeerID = PeerID
    resp.Clients = StateManager.GetClientSideUsers
    resp.UserHash = StateManager.GetHash
    Return Utils.Serialize(resp)
End Function

Maintaining State

In Figure 4 you'll notice that I make a call to the ExistsUserName method in a class named StateManager. Yes, I said a class, not an object. StateManager is a class that I've defined inside the Web service project to manage the state of each client. ExistsUserName is one of several methods that are defined with the Shared keyword. They are static methods; you do not need to instantiate an object of the class to call Shared methods. Another way to think of Shared methods is that all instances of the class share all shared methods and variables. One instance serves all requests.

This is what makes the whole thing work. Without keeping some sort of list of the clients around, you'd have no chat architecture. I could have also used the ASP.NET Application state object or the ASP.NET Cache, both accessible through the HttpContext.Current to a Web service hosted in ASP.NET.

Now let's look at the pros and cons of using Shared members and why I've chosen to use them. The first drawback that might concern you is that since this class is exposing shared methods and data, that data lives as long as the host process is alive, so you have to manage it. Another drawback is that the code that accesses the shared code and data must be thread-synchronized, but that's not very hard to do in Visual Basic® .NET. The final drawback you might worry about is that, due to synchronization, the synchronized code becomes a bottleneck because only one thread can access it at any one time. However, this chat program transmits very little data between a small number of clients, so any bottlenecks that might arise should be completely transparent.

Now let's look at the ExistsUserName method:

Public Shared Function ExistsUserName(ByVal UserName As String) As Boolean
    Dim there As Boolean = False
    SyncLock Clients
        For Each obj As P2PClient In Clients
            If String.Compare(obj.UserName, UserName) = 0 Then
                there = True
                Exit For
            End If
        Next
    End SyncLock
    Return there
End Function
This code performs a simple collection lookup on the Clients collection (P2PClientCollection), which is a custom collection of class P2PClient. Didn't get that? OK, let me explain.

I've created some classes (that are used on both the client and server) in a DLL called p2pTools.dll. I have a reference to this DLL in the Web service, and the WebChat client solution file includes p2pTools as a project. This is where the P2PClient class is defined. The P2PClientCollection class is just a custom collection strongly typed to hold P2PClient objects. Note that by creating a DLL that's required on both the client and on the server, I'm increasing the coupling between the two, which goes against some of the tenets of Web services. It's fine for our purposes, though.

The P2PClient class is used on the server side to represent a client. Its properties are PeerID (a String), which is a unique ID created by the server to identify a client; UserName (a String), which is an ID generated by the user; LastAccess (a DateTime), which is the time at which the client last connected to the server to poll for messages; and Messages, which is a custom collection of P2PMessage objects. P2PMessage defines a message: FromUserName, ToUserName, and Message (Strings), and TimeSent (DateTime). See? It's simple.

Logging In Users

Let's go back to the LogIn WebMethod. The purpose of the ExistsUserName call is to see whether or not the user is already logged in. In other words, is there already a user with this name in the shared Clients collection? If so, an exception is thrown; otherwise, we move on.

Next, I call a shared function in the StateManager class: AddClientAndReturnPeerID, shown with the PeerID property of the P2PClient in Figure 5. Anytime you access a reference-type object from more than one thread, you must synchronize access to that object, so that only one thread at a time can access it. This is not unlike how SQL Server™ locks a page or a record when reading or writing. If another request comes in during that operation, it has to wait for the current operation to remove the lock. This ensures that only one client at a time can access a record or a page in the database and keeps conflicts from happening.

Figure 5 AddClientAndReturnPeerID (Close Figure 5)
Public Shared Function AddClientAndReturnPeerID(ByVal UserName _
        As String) As String
    Dim obj As P2PClient
    SyncLock Clients
        obj = Clients.Add
        obj.UserName = UserName
    End SyncLock
    Return obj.PeerID
End Function

<Serializable()> Public Class P2PClient

    Private _peerid As String = Nothing
    Public ReadOnly Property PeerID() As String
        Get
            If _peerid = Nothing Then
                '-- create a random ID
                Dim g As Guid = Guid.NewGuid()
                _peerid = g.ToString
            End If
            Return _peerid
        End Get
    End Property

    '-- More here

End Class

The SyncLock statement does this for you in Visual Basic .NET. In C#, the keyword is lock. In this case, I'm locking the shared Clients collection so that multiple threads can access it safely.

The PeerID property of the P2PClient class will generate itself automatically if none exists. It does so by creating a new GUID using Grid.NewGrid and the retrieving string representation of it. Originally, my design called for the Web service to return an object of type LogInReturn, a class defined in the DLL, but since the LogInReturn class contains a property of a type based on CollectionBase, it wouldn't serialize out of the box, so I had to perform some manual serialization instead. I could have also used an array instead of a collection, which would serialize fine.

So, LogIn returns an array of bytes. The Serialize method of the Utils class (also defined in the p2pTools.DLL component) makes this easy. Figure 6 shows the Utils class. The BinaryFormatter object does the work, but you have to serialize to a stream, not an array of bytes. From a memory stream, you can use the ToArray method to return the bytes which, when passed from a Web service, get converted into a Base64 string. It's a roundabout way to move binary data around, but it works. It works especially well for the small amounts of data that a chat program passes around.

Figure 6 The Utils Class (Close Figure 6)
Option Strict On

Imports System.IO
Imports System.Runtime.Serialization
Imports System.Runtime.Serialization.Formatters.Binary

Public Class Utils
    Public Shared Function Serialize(ByVal Obj As Object) As Byte()
        Dim bf As New BinaryFormatter
        Dim mem As New MemoryStream
        Try
            bf.Serialize(mem, Obj)
        Catch ex As Exception
            Throw ex
        End Try
        Return mem.ToArray
    End Function

    Public Shared Function Deserialize(ByVal bytes() As Byte) As Object
        Dim bf As New BinaryFormatter
        Dim mem As New MemoryStream(bytes)
        Dim obj As Object
        Try
            obj = bf.Deserialize(mem)
        Catch ex As Exception
            Throw ex
        End Try
        Return obj
    End Function
End Class

As you saw back in Figure 3, the response from the Web server comes in as the resp byte array. That is deserialized into a LogInReturn object, which contains the PeerID, a collection of clients, and a unique code that represents the current collection of user names and Peer IDs. You already know that PeerID is a unique GUID identifying the client. The Clients property is a custom collection of ClientSideUser objects. ClientSideUser doesn't have the messages associated with it, so it's appropriate for sending to the client as a list of logged-in users.

The Hash property is a bit different. It's the result of a hash algorithm that is performed on the list of clients. So, if a client logs off, a new client logs on, or the list of clients changes in any way, the hash will also change. I'm using the SHA1 one-way hashing algorithm to compute the hash. This is worth investigating, so take a look at Figure 7 which shows the GetHash method in the StateManager class. The GetHash method returns a unique hash code representing the current state of the client list. It is used in order to determine if there are any new users.

Figure 7 GetHash Method (Close Figure 7)
Public Shared Function GetHash() As String
    '-- return a unique hash string from the usernames and peerids
    Dim ret As String
    SyncLock Clients
        Dim users As ClientSideUserCollection = _
            StateManager.GetClientSideUsers
        Dim bytes() As Byte = Utils.Serialize(users)
        Dim SHA1 As New _
            System.Security.Cryptography.SHA1CryptoServiceProvider
        Dim hash() As Byte = SHA1.ComputeHash(bytes)
        ret = Convert.ToBase64String(hash)
        SHA1.Clear()
    End SyncLock
    Return ret
End Function

I lock the Clients object so that it can't be modified by the server while I'm computing the hash. Next, I get a ClientSideCollection that contains just user names and PeerIDs, serialize it to a byte array, and compute a hash on the byte array. Returning the hash as another byte array, it is returned as a Base64 string. Upon polling for new messages, the client receives the latest hash code, which will be different if the list of clients has changed. At that time, the client calls another method in order to get the most current list and the hash code, of course.

Now let's go back to the main form's Load event handler (see Figure 8). First I show the main form. Then I create a new login form and show that modally. When the application has gathered credentials and the login dialog closes, either I wasn't validated (because server-side validation is not implemented), or else I was validated and the return object has a PeerID and a Clients list which is bound to the listbox.

Figure 8 Load Event (Close Figure 8)
Private Sub frmMain_Load(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles MyBase.Load
    Me.Show()

    Dim frm As New frmLogin
    frm.ShowDialog()

    myID = frm.PeerID
    If myID Is Nothing Then Application.Exit()

    clients = frm.Clients
    With lbUsers
        .DataSource = clients
        .DisplayMember = "UserName"
        .ValueMember = "PeerID"
    End With

    server.BeginGetNextMessage(myID, myHash, AddressOf _
        ChatCallback, Nothing)

End Sub

Finally, the polling starts. The GetNextMessage WebMethod is called asynchronously. I pass my PeerID and the address of a sub that will be called when the Web service response is received. In this case it's ChatCallBack, which is shown in Figure 9 along with the GetNextMessage WebMethod on the server.

Figure 9 Calling ChatCallBack (Close Figure 9)

Server
<WebMethod()> Public Function GetNextMessage(ByVal MyPeerID As String) _
        As Byte()
    '-- Does the sender exist?
    Dim resp As New GetNextMessageReturn

    If StateManager.ClientExists(MyPeerID) Then
        resp.Message = StateManager.GetNextMessage(MyPeerID)
        If resp.Message Is Nothing Then
            resp.Response = "No new messages"
        Else
            resp.Response = "OK"
        End If
    Else
        resp.Response = "You must log in first"
    End If
    resp.UserHash = StateManager.GetHash
    Return Utils.Serialize(resp)
End Function
Client
Private Sub ChatCallback(ByVal ar As IAsyncResult)
    Dim bytes() As Byte = server.EndGetNextMessage(ar)
    Dim resp As GetNextMessageReturn = CType(Utils.DeSerialize(bytes), _
        GetNextMessageReturn)
    If resp.Response = "OK" Then
        Dim Msg As String = "<" & resp.Message.FromUserName & "> " & _
            resp.Message.Message
        Dim args() As Object = {Msg}
        Dim dlg As New ShowMessageDelagate(AddressOf ShowMessage)
        Me.Invoke(dlg, args)
    End If
    Thread.Sleep(1000)
    server.BeginGetNextMessage(myID, AddressOf ChatCallback, Nothing)
    If resp.UserHash <> myHash Then
        '-- refresh the user list
        server.BeginGetUsers(AddressOf GetUsersCallback, Nothing)
    End If
End Sub

Sending and Receiving

On the server, I first check to see if the client exists (is logged in) and, if so, I call StateManager.GetNextMessage which does the work (see Figure 10). If a new message is received, it is displayed. Either way, the thread goes to sleep for a second and then calls the GetNextMessage WebMethod again asynchronously, ensuring that the polling will continue as long as the app is running. Now it's just a matter of getting messages into the client's Messages collection on the server, which is easy enough. It's a collection, so you just add a new message to it. The SendMessage and SendMessageToAll WebMethods do just that.

Figure 10 GetNextMessage (Close Figure 10)
Public Shared Function GetNextMessage(ByVal ToID As String) As P2PMessage
    Dim msg As P2PMessage
    Dim index As Int32

    '-- Lock the clients collection
    SyncLock Clients
        '-- Retrieve the client object from its ID
        Dim toClient As P2PClient = Clients(ToID)
        If Not toClient Is Nothing Then
            '-- Reset the timestamp for last access
            toClient.LastAccess = Now
            '-- Lock the messages collection
            SyncLock toClient.Messages
                '-- Is there more than one message waiting?
                If toClient.Messages.Count > 0 Then
                    '-- Yes! Skim the first one off
                    msg = toClient.Messages(0).Clone
                    toClient.Messages.RemoveAt(0)
                End If
            End SyncLock
        End If
    End SyncLock
    '-- Return the message (or Nothing)
    Return msg
End Function

The only other code that's interesting here is the code required to synchronize access to the Windows user interface. Because the callback happens on a different thread than the one on which the UI controls were created, you have to use the ISynchronizeInvoke.Invoke method on any control to call a delegate pointing to a local sub that actually modifies the user interface. In this case, Me—the form—works just fine. For a more detailed explanation of this requirement, check out Ian Griffiths' article on multithreading in the February 2003 issue of MSDN®Magazine.

I took a number of shortcuts in getting the chat program ready for this column. I wrote a more robust real-world version that goes way beyond what I could address here. The program addresses the design issues laid out here, and also includes two-way symmetric encryption and file transfer capabilities.

First, it's a bad idea to pass binary serialized objects in the form of a byte array with Web services. It defeats the open nature of the Web service, and requires that the user have the right version of the binary formatter installed on the client. Not that it matters much in this example, but it's still not a good idea. Second, using collections in objects through Web services is the reason that I needed to use the binary formatter in the first place. Collections don't serialize by default through Web services. A better approach is to use an array.

You can download Zeep Lite, a version of this chat program that uses arrays and classes instead of serialized classes with collections, at my Web site (http://www.franklins.net/zeep).

Now go write some code!

Send your questions and comments to basics@microsoft.com.

Carl Franklin is president of Franklins.Net where he teaches Visual Basic .NET classes and hosts .NET Rocks!, an Internet audio talk show for developers who use the .NET Framework.


.NET Matters: Debugger Visualizations, Garbage Collection

.NET Matters
Debugger Visualizations, Garbage Collection
Stephen Toub

Code download available at: NETMatters0408.exe (208KB)

Q I really enjoyed reading Morgan Skinner's article in the May 2004 issue of MSDN®Magazine covering the new debugger visualizations feature in Visual Studio® 2005 (see Debugging: DataTips, Visualizers and Viewers Make Debugging .NET Code a Breeze). His Hashtable visualizer is also really useful. I know this feature doesn't exist in Visual Studio .NET 2003, but it's just so helpful, I'd love to be able to use something like it now. What are my options?

A Debugger visualizations are an amazing feature for the debugger and will certainly save developers a ton of time. Unfortunately, as you state, they're new to Visual Studio 2005 and don't exist in the current version of the product. But that doesn't mean you're completely at a loss. With a little help from the Command/Immediate window, there's a solution that still gives you the power to visualize your data in a manner similar to the way it works in Visual Studio 2005 and with only slightly more work. As an example, I'll show you how to take Morgan's code and, with only a few very minor changes (and only slightly more work at run time), get it to work in Visual Studio .NET 2003.

The simplest way to get "visualizations" in Visual Studio .NET 2003 is to use the inline "printf style" of debugging, whereby you insert lots of calls inline in the code to help trace its execution. Let's say I wanted to visualize a Hashtable. Instead of using Console.WriteLine, I could write my own method that takes a Hashtable as input and shows a dialog with a rendered view of that Hashtable. Since I like to reuse existing code to make my life easier, I'll use Morgan's code for rendering the table. Of course, his code was written for the March 2004 Technical Preview release of Visual Studio 2005, so I had to make some minor changes to get it to compile in Visual Studio .NET 2003. I started by downgrading the implementation to compile in the current version of C#. This actually took very little effort, only requiring me to combine the pieces of the partial classes in use (since partial classes are new to C# 2.0) and removing some properties that are new to the Windows® Forms 2.0 object model.

With this new version of the Hashtable rendering code in place, which is available in the code download on the MSDN Magazine Web site, I can simply make calls to the render method inline in my program. This is a step in the right direction, but one of the great benefits of debugger visualizations in Visual Studio 2005 is that you don't need to mess with your source to use them; rather, they're built into the debugger and can be used by any solution regardless of whether it is aware of the visualization's existence. How can we accomplish this in Visual Studio .NET 2003? Through liberal use of the Command window (also known as the Immediate window), which lets us execute custom code at run time while stopped at a breakpoint. Ideally, I'd like to be able to simply execute the following code in the Immediate window and have it display my custom rendering of the Hashtable:

NetMatters.DebuggingVisualizations.DisplayHashtable(ht)
This exact usage is actually possible, but only if the assembly containing the NetMatters.DebuggingVisualizations class has already been loaded and if DisplayHashtable is a static method on the DebuggingVisualizations class.

The latter condition is easily taken care of, as shown in Figure 1. The former condition will only be satisfied if references to code in that assembly have been JIT compiled, or if the assembly has been loaded explicitly using one of the loading methods on the System.Reflection.Assembly class (Assembly.Load, Assembly.LoadFrom, and so on). If you know that the assembly containing the visualizers has not been loaded, you can easily fix this problem while debugging by loading the assembly from the Command window, as shown in the following:

System.Reflection.Assembly.LoadFrom("C:\DebugVis.dll")

Figure 1 DebuggingVisualizations (Close Figure 1)
public class DebuggingVisualizations
{
    // Display a Hashtable while debugging
    public static void DisplayHashtable(Hashtable ht)
    {
        HashtableProxy proxy = new HashtableProxy(ht);
        Hash [] hashes = proxy.WalkHashtable();
        using(HashtableVisualizerForm form =
            new HashtableVisualizerForm())
        {
            form.Hashes = hashes;
            form.ShowDialog();
        }
    }

    // Display an image while debugging
    public static void DisplayImage(Bitmap img)
    {
        string temp = Path.GetTempFileName() + ".jpg";
        img.Save(temp, ImageFormat.Jpeg);
        Process.Start(temp);
    }

    // Display an XmlDocument while debugging
    public static void DisplayXml(XmlDocument doc)
    {
        string temp = Path.GetTempFileName() + ".xml";
        using(StreamWriter writer = new StreamWriter(temp))
        {
            doc.Save(writer);
        }
        Process.Start(temp);
    }
}

I've used Assembly.LoadFrom, specifying the path to the DLL. If you put the DLL into the Global Assembly Cache, which would be useful for debugging lots of different projects, you could instead use Assembly.Load, specifying the full name of the assembly. Since the full name is a fairly long string to type into the Command window, you might also use LoadWithPartialName to make for less typing while debugging, although there are some drawbacks to using this particular method. For more information on the trade-offs, see Suzanne Cook's excellent blog on the subject at Avoid Partial Binds. In fact, LoadWithPartialName is being deprecated in the Microsoft .NET Framework 2.0. Regardless, after executing this line of code, the DebugVis assembly will be loaded, a fact you can verify by looking at the Modules window in the debugger. We can then continue by executing the previously mentioned code in the Immediate window to display the Hashtable, as shown in Figure 2. Piece of cake!

Figure 2 Visualizing a Hashtable (Close Figure 2)

There are obviously benefits to having this type of functionality built into the debugger and the IDE, but just because Visual Studio .NET 2003 lacks support for this feature doesn't mean we can't be inventive and create something very similar that can be used today. Figure 1 shows some code for visualizing Bitmaps, XmlDocuments, and Hashtables. You could obviously augment this with visualization routines for other types in the .NET Framework or even for your own custom types. The possibilities are endless. Happy debugging!

Q From within my application, I'd like to be notified when the garbage collector runs in my process. Is this possible? Is there an event raised when a collection takes place with which I could register an event handler? I looked at the System.GC class but couldn't find anything appropriate.

A As far as I know there's no perfect way to do this, though you can certainly construct a few good approximations.

One approach is to take advantage of finalization. As a very quick review (and, as with any quick review, I'm leaving out some important details), managed classes can implement a finalizer. An instance's Finalize method is called after the garbage collector (GC) determines that the instance is unreachable but before the memory for that instance is reclaimed. When an object that implements a finalizer is instantiated, a reference to that object is added to a finalization list, sometimes referred to as the RegisteredForFinalization list. When the GC runs and finds unreachable objects that are also in the finalization list, the references to these objects are moved to another list, the freachable queue, also referred to as the ReadyForFinalization queue. A dedicated and high-priority thread owned by the common language runtime (CLR) scans this queue and runs the Finalize method of all objects found. Jeffrey Richter describes the process in much greater detail in his book Applied Microsoft .NET Framework Programming (Microsoft Press®, 2003).

The net result of this is that we can take advantage of finalization in order to estimate when the GC has run. By instantiating and then losing all references to an object whose Finalize method raises an event, I can be notified when the finalization thread processes my object from the freachable queue, and this will (hopefully) be soon after the GC has collected my object.

Here is my first attempt at creating such a class:

public class GcNotify
{
    public GcNotify(){}
    public static event EventHandler Collection;
    public static void OnCollection()
    {
        EventHandler ev = Collection;
        if (ev != null) ev(null, EventArgs.Empty);
    }
    ~GcNotify() { OnCollection(); }
}
I can test this by implementing a simple command-line application that continually calls GC.Collect in order to force a collection:
static void Main()
{
    GcNotify.Collection += new EventHandler(GcNotify_Collection);
    new GcNotify();
    while(true)
    {
        GC.Collect(0);
        Thread.Sleep(1000);
    }
}

The implementation of GcNotify_Collection simply writes a message with a timestamp out to the console. Running this, I find that I'm only notified of a single garbage collection. Why? Because I only created one object and, once it was finalized and deallocated, its Finalize method was never run again. I might try to fix this by modifying the class's finalizer as follows:

~GcNotify()
{
    OnCollection();
    GC.ReRegisterForFinalize(this);
}

The GC.ReRegisterForFinalize method causes a reference to the object to be put back onto the finalization list. Thus, the next time the GC runs, it will find this instance unreachable and will move it again to the freachable queue for the finalization thread to process. However, when I run this new code I find that I'm still only being notified of a single collection. A reexamination of the test application reveals why: I'm calling GC.Collect and passing in 0 as an argument. This tells the garbage collector to collect only Generation 0, skipping the collection of Generations 1 and 2. Why is that important? Because the instance survives collections and is thus promoted to higher generations. The first time the GC runs after the object has been created (and before the object is finalized), the instance is promoted to Generation 1, and the second time the GC runs the instance is promoted to Generation 2. You can verify this in the object's finalizer by writing out the result of calling GC.GetGeneration(this), which retrieves the current generation of the specified object. Since the garbage collector should collect Generation 0 much more frequently than the other generations, I'm doing myself a huge disservice by forcing my notification object to be promoted.

Since I want an event to be raised for every collection and not just for those that include Generations 1 and 2, I can change the implementation of the finalizer to instantiate a new instance of GcNotify each time it's run. That way, I have a much better chance of having an instance of GcNotify in Generation 0 (note that I've refrained from saying that I'll always have an instance in Generation 0 for reasons that should become clear in a moment):

~GcNotify()
{
    OnCollection();
    new GcNotify();
}

Running the sample with this new finalizer gives me the desired behavior. There are still some problems with it, however. First, there are a few situations in which objects will be finalized. One of them, of course, is when the GC is run, but finalizers are also executed during AppDomain unloading and during CLR shutdown. In these situations, I don't want to cause problems by creating new objects that will need to be finalized themselves, so I test for this using Environment.HasShutdownStarted and AppDomain.CurrentDomain.IsFinalizingForUnload, only creating a new GcNotify instance if both of these return false. For another example of this, see the GCBeep example from Richter's book.

The second problem is that I haven't implemented any mechanism to prevent the user from creating multiple instances of GcNotify. If the user instantiates more than one instance of GcNotify, the Collection event will be raised erroneously multiple times per collection. To fix this, I can modify GcNotify's constructor to be private and implement a public and static initialization method that shields the user from having to explicitly create an instance of the class. The final version of my class and test harness is shown in Figure 3.

Figure 3 GcNotify and Test Harness (Close Figure 3)
class TestGcNotify
{
    static void Main()
    {
        GcNotify.Collection += new EventHandler(GcNotify_Collection);
        GcNotify.Initialize();
        while(true)
        {
            GC.Collect(0);
            Thread.Sleep(1000);
        }
    }

    private static void GcNotify_Collection(object sender, EventArgs e)
    {
        Console.WriteLine("GC collection: " + DateTime.Now);
    }
}

public class GcNotify
{
    private static bool _initialized = false;

    public static void Initialize()
    {
        if (!_initialized)
        {
            _initialized = true;
            new GcNotify();
        }
    }

    private GcNotify(){}

    public static event EventHandler Collection;

    public static void OnCollection()
    {
        EventHandler ev = Collection;
        if (ev != null) ev(null, EventArgs.Empty);
    }

    ~GcNotify()
    {
        if (!Environment.HasShutdownStarted &&
            !AppDomain.CurrentDomain.IsFinalizingForUnload())
        {
            OnCollection();
            new GcNotify();
        }
    }
}

As mentioned earlier, while this is a decent approximation, the approach isn't perfect. A major problem is that it relies on there being very few objects that require finalization and on those objects' finalizers running to completion quickly. Since there's only one finalization thread, finalizers are run in serial, and thus a poorly implemented finalizer in another class could delay or even prevent ~GcNotify from running. Consider the following class:

public class StallFinalization
{
    ~StallFinalization()
    {
        while(true) Thread.Sleep(1000);
    }
}
If an instance of this class is finalized, it will stall the finalization thread and no other objects will have their finalizers run. So much for our event notification.

An alternative to the finalization approach would be to implement a polling mechanism that checks to see if the garbage collector has recently run. If it has, an event can be raised. One way to implement this polling is to take advantage of weak references. The WeakReference class has tight integration with the garbage collector; WeakReference wraps an object which it exposes through its Target property, but as far as the garbage collector is concerned, the object is unreachable (assuming there are no other strong references to it) and it can be collected.

When the collector runs, the Target of the WeakReference will be set to null. Thus, by creating a WeakReference to an object and then polling to see when the WeakReference's Target property is null (or when its IsAlive property is false), I can determine whether the garbage collector has recently run. If it is null, the Collection event is raised and the Target is reset to be a new object for which I can continue to poll. The code for this version of GcNotify is shown in Figure 4.

Figure 4 GcNotifyWithWeakReference (Close Figure 4)
public class GcNotifyWithWeakReference
{
    private static bool _initialized = false;
    private static WeakReference _obj;
    private static Timer _timer;

    public static void Initialize(int pollTime)
    {
        if (!_initialized)
        {
            if (pollTime <= 0) throw
                new ArgumentOutOfRangeException("pollTime");
            _obj = new WeakReference(new object());
            TimerCallback callback =
                new TimerCallback(CheckWeakReference);
            _timer = new Timer(callback, null, pollTime, pollTime);
            _initialized = true;
        }
    }

    private GcNotifyWithWeakReference() {}

    public static event EventHandler Collection;

    public static void OnCollection()
    {
        EventHandler ev = Collection;
        if (ev != null) ev(null, EventArgs.Empty);
    }

    private static void CheckWeakReference(object state)
    {
        if (!_obj.IsAlive)
        {
            OnCollection();
            _obj.Target = new object();
        }
    }
}

I can also set up a polling scheme similar to that in Figure 4, but which checks the .NET CLR Memory performance counters, querying to see how many Generation 0 collections have occurred for the current process. The code for this version is shown in Figure 5. Note that it follows the same general format as the WeakReference-based version. To determine whether there has been a recent collection, it stores the number of collections and polls to see when this count changes. In order to access this information and have it be specific to the current application, I need to get the performance counter instance that represents the current process. Since the instances are created and named based on the name of the running application, I can get the appropriate text by accessing the name of the entry assembly. Additionally, with the .NET Framework 1.x, the length of the name of the instance is limited to 15 characters, so I need to ensure that I've appropriately trimmed the string before instantiating the PerformanceCounter.

Figure 5 GcNotifyWithPerfCounter (Close Figure 5)
public class GcNotifyWithPerfCounter
{
    private static bool _initialized = false;
    private static PerformanceCounter _pc;
    private static float _lastValue;
    private static Timer _timer;

    public static void Initialize(int pollTime)
    {
        if (!_initialized)
        {
            if (pollTime <= 0) throw
                new ArgumentOutOfRangeException("pollTime");
            string instanceName =
                Assembly.GetEntryAssembly().GetName().Name;
            if (instanceName.Length > 15)
                instanceName = instanceName.Substring(0, 15);

            _pc = new PerformanceCounter(
                ".NET CLR Memory", "# Gen 0 Collections",
                instanceName, true);
            TimerCallback callback = new TimerCallback(CheckForCollection);
            _lastValue = _pc.NextValue();
            _timer = new Timer(callback, null, pollTime, pollTime);
            _initialized = true;
        }
    }

    private GcNotifyWithPerfCounter(){}

    public static event EventHandler Collection;

    public static void OnCollection()
    {
        EventHandler ev = Collection;
        if (ev != null) ev(null, EventArgs.Empty);
    }

    private static void CheckForCollection(object state)
    {
        float newValue = _pc.NextValue();
        if (newValue != _lastValue)
        {
            _lastValue = newValue;
            OnCollection();
        }
    }
}

Of course, this solution still requires explicit polling, just as with the WeakReference-based solution. Using Windows Management Instrumentation (WMI) to access the performance counters, I could instead let WMI handle all of polling overhead and just register with it to receive notifications of a change (note that WMI will still be polling).

The System.Management namespace supplies the WqlEventQuery class which represents a Windows Management Instrumentation Query Language (WQL) query. I can create an instance of this class to watch for changes in a specific performance counter. To configure the query, I first need to set the type of event for which it's watching, in this case an __InstanceModificationEvent. Second, I need to specify the condition that narrows down what instances I want the query to be concerned with; this often comes in the form "TargetInstance isa 'class'", where class is the type of the WMI class that represents the performance counter class in which I'm interested. One way to find that name is to write a simple VBScript that displays all of the registered classes that derive from Win32_PerfRawData, the base class for all concrete raw performance counter classes:

Set subclasses = GetObject("winmgmts:root/cimv2")
    .SubclassesOf("Win32_PerfRawData")
For Each subclass In subclasses
    WScript.Echo subclass.Path_.Class
Next

Executing this script in a console window will cause the list of Win32_PerfRawData-derived classes to be output to the console. Scanning the list, you should notice the Win32_PerfRawData_NETFramework_NETCLRMemory class, which represents the .NET CLR Memory category. I can use this value to fill in my condition. Finally, I need to tell the query how often to poll for changes. The code to configure the query looks like this:

WqlEventQuery q = new WqlEventQuery();
q.EventClassName = "__InstanceModificationEvent";
q.Condition = "TargetInstance isa" +
    "'Win32_PerfRawData_NETFramework_NETCLRMemory'";
q.WithinInterval = new TimeSpan(0,0,0,0,250);

With the query in place, I need to configure the system to use it to inform me of events. The ManagementEventWatcher class provides this functionality, and I can create one as follows (where OnCollection is the name of the event handler that will be invoked when the GC has recently run):

ManagementEventWatcher watcher = new ManagementEventWatcher(q);
watcher.EventArrived += new EventArrivedEventHandler(OnCollection);
watcher.Start();

Inside my OnCollection event handler, I can extract from EventArrivedEventArgs information about the number of Generation 0 collections that have occurred:

private static void OnCollection(object sender, EventArrivedEventArgs e)
{
    ManagementBaseObject eventArg =
       (ManagementBaseObject)e.NewEvent["TargetInstance"];
    uint numCollections =
        (uint)eventArg.Properties["NumberGen0Collections"].Value;
    Console.WriteLine("Gen 0: " + numCollections);
}

This looks fairly clean, but unfortunately there are a few problems with it. First, the condition specified on the WqlEventQuery watches for all changes to any instance of the specified performance counter class, not just those for my process, and it watches for any change to those instances, not just a change in the property that represents the number of Generation 0 collections. These issues can be fixed by augmenting the condition in order to ensure that the TargetInstance.Name property value equals the name of our performance counter instance and that the TargetInstance.NumberGen0Collections property value does not equal the PreviousInstance.NumberGen0Collections property value.

The second and more complicated problem is that there are many environmental issues which affect whether this solution will even work correctly at all. These include access control list (ACL) issues based on which user is attempting to access the counters and as which user the process is running, as well as whether the CLR perfcounter DLL was initialized before or after the managed process was started. Most of these complications have been fixed in the .NET Framework 2.0, so in the meantime I'd suggest using one of the previously mentioned approaches.

All that said, I have to wonder why you even need this information. If it's for monitoring the health of your application, that should really be done externally to it, using performance counters or even a profiler. If you're actually using this information in your application to determine whether to perform some action, I'd highly recommend that you reevaluate. It's a poor idea to base your application's logic on something that should be fairly invisible to it, and a healthy application should really not have to worry about when the garbage collector is running. Rico Mariani, an architect on the CLR team, has tons of great advice on his blog concerning garbage collection and application performance tuning. I suggest you check it out at http://blogs.msdn.com/ricom.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine. You can reach him at stoub@microsoft.com.


C++ Q&amp;A: GetKeyState, the STL String Class, Exposing C++ Objects, and More

New information has been added to this article since publication.
Refer to the Editor's Update below.

C++ Q&A
GetKeyState, the STL String Class, Exposing C++ Objects, and More
Paul DiLascia

Code download available at: CQA0408.exe (234KB)

Q I want users to hold down the Control key while double-clicking on my program's icon to have the program start up in a particular way. Neither ::GetCommandLine nor __argc give any indication that the key was pressed; nor does there seem to be any way to do this in MFC using CCommandLineInfo. Is there a way to find this out, say by polling the key?

Dean Jones

A Yes, it's very simple. All you have to do is call GetKeyState. This function returns the status of a virtual key at the time the current message you are processing was sent. The status can be up, down, or toggled. Toggled is for keys like Caps and Shift Lock, which alternate states. For ordinary keys like the Control key (VK_CONTROL), the high-order sign bit of the state is 1 if the key is down.

Many apps use Control+F8 as the special key to launch in restore mode. For example, if your app lets users customize their workspaces, starting with Control+F8 might restore the workspace to its factory default settings. Just make sure you ask the user to confirm first. Even better would be to save the customizations in separate INI files, so users have a chance of recovering them. In any case, to check for the Control key when your app starts, you'd write something like this:

if (GetKeyState(VK_CONTROL)<0) {
  // enter special mode
}

Figure 1 shows a snippet from a sample MFC app you can download from the link at the top of this article that displays a message box and beeps if the user presses Ctl+F8 while starting the app. If you just want to check for the Control key, leave out the test for VK_F8.

Figure 1 GetKeyState (Close Figure 1)
////////////////////////////////////////////////////////////////
// MSDN Magazine — August 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
// CtlTest illustrates how to test for a special control-key combination
// when your program starts.

#include "StdAfx.h"
#include "MainFrm.h"
#include "StatLink.h"
#include "resource.h"

class CMyApp : public CWinApp {
public:
   virtual BOOL InitInstance();
   •••
} theApp;

BOOL CMyApp::InitInstance()
{
   BOOL bCtl = GetKeyState(VK_CONTROL)<0;
   BOOL bF8  = GetKeyState(VK_F8)<0;
   if (bCtl && bF8) {
      if (AfxMessageBox(_T("Enter special startup mode?"),
                        MB_YESNO)==IDYES)
         // Enter special startup mode here.
         MessageBeep(0);
   }
   •••
}

Q I use the Standard Template Library (STL) std::string class very often in my C++ programs, but I have a problem when it comes to Unicode. When using regular C-style strings I can use TCHAR and the _T macro to write code that compiles for either Unicode or ASCII, but I always find it difficult to get this ASCII/Unicode combination working with the STL string class. Do you have any suggestions?

Naren J.

A Sure. It's easy, once you know how TCHAR and _T work. The basic idea is that TCHAR is either char or wchar_t, depending on the value of _UNICODE:

// abridged from tchar.h
#ifdef  _UNICODE
typedef wchar_t TCHAR;
#define __T(x) L ## x
#else
typedef char TCHAR;
#define __T(x) x
#endif

When you choose Unicode as the character set in your project settings, the compiler compiles with _UNICODE defined. If you select MBCS (Multi-Byte Character Sets), the compiler builds without _UNICODE. Everything hinges on the value of _UNICODE. Similarly, every Windows® API function that uses char pointers has an A (ASCII) and a W (Wide/Unicode) version, with the real version defined to one of these, based on the value of _UNICODE:

#ifdef UNICODE
#define CreateFile CreateFileW
#else
#define CreateFile CreateFileA
#endif

Likewise, there's _tprintf and _tscanf for printf and scanf. All the 't' versions use TCHARs instead of chars. So how can you apply all this to std::string? Easy. STL already has a wstring class that uses wide characters (defined in the file xstring). Both string and wstring are typedef-ed as template classes using basic_string, which lets you create a string class using any character type. Here's how STL defines string and wstring:

// (from include/xstring)
typedef basic_string<char,
  char_traits<char>, allocator<char> >
  string;
typedef basic_string<wchar_t,
  char_traits<wchar_t>, allocator<wchar_t> >
  wstring;

The templates are parameterized by the underlying character type (char or wchar_t), so all you need for a TCHAR version is to mimic the definitions using TCHAR:

typedef basic_string<TCHAR,
  char_traits<TCHAR>,
  allocator<TCHAR> >
  tstring;

Now you have a tstring that's based on TCHAR—that is, either char or wchar_t, depending on the value of _UNICODE. I'm showing you this to point out how STL uses basic_string to implement strings based on any underlying character type. Defining a new typedef isn't the most efficient way to solve your problem. A better way is to simply #define tstring to either string or wstring, like so:

#ifdef _UNICODE
#define tstring wstring
#else
#define tstring string
#endif
This is better because STL already defines string and wstring, so why use templates to create another string class that's the same as one of these, just to call it tstring? You can use #define to define tstring to string or wstring, which will save you from creating another template class (though compilers are getting so smart these days it wouldn't surprise me if the duplicate class were discarded). [ Editor's Update - 7/30/2004: typedef does not create a new class, but rather introduces a scoped name for a type. A typedef never defines a new type.] In any case, once you have tstring, you can write code like this:
   tstring s = _T("Hello, world");
   _tprintf(_T("s =%s\n"), s.c_str());
The method basic_string::c_str returns a const pointer to the underlying character type; in this case, that character type is either const char* or const wchar_t*.

Figure 2 shows a simple program I wrote that illustrates tstring. It writes "Hello, world" to a file and reports how many bytes were written. I set the project up so it uses Unicode for the Debug build and MBCS for the Release build. You can compile both builds and run them to compare the results. Figure 3 shows a sample run.

Figure 2 tstring (Close Figure 2)
////////////////////////////////////////////////////////////////
// MSDN Magazine — August 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//

// TSTRING shows how to implement a tstring class that uses STL string or
// wstrings depending on the setting of _UNICODE, similar to TCHAR,
// _tprintf and all the other "t" versions of functions in the C runtime.
//
// To see the difference, compile both debug and release versions. The
// debug version uses Unicode; the release uses MBCS. Then run each
// program and compare the output files.
//
#include "stdafx.h"
#include "resource.h"

using namespace std;

// tstring is either string or wstring, depending on _UNICODE.
// This works too, but may produce an extra class:
//
//   typedef basic_string<TCHAR, char_traits<TCHAR>,
//      allocator<TCHAR> > tstring;
//
#ifdef _UNICODE
#define tstring wstring
#else
#define tstring string
#endif

static void WriteString(HANDLE f, LPCTSTR lpsz, int len);

void _tmain(int argc, TCHAR* argv[], TCHAR* envp[])
{
   // process args
   if (argc != 2) {
      _tprintf(_T("Usage: tstring [filename]\n"));
      _tprintf(_T("       writes test message to [filename]\n"));
      return;
   }

   // CreateFile will create Unicode or MBCS string
   // depending on value of _UNICODE.
   LPCTSTR filename = argv[1];
   HANDLE f = CreateFile(filename, ...);
   if (f!=INVALID_HANDLE_VALUE)
   {
      if (GetFileType(f) == FILE_TYPE_DISK)
      {
         // create STL tstring
         tstring s = _T("Hello, world");
         WriteString(f, s.c_str(), s.length());
      } else {
         tprintf(_T("ERROR: the specified file '%s' is not a disk file\n"),
                 filename);
      }
      CloseHandle(f); // close file
   } else {
      _tprintf(_T("ERROR: can't open '%s'\n"), filename);
   }
}

////////////////
// write string to file.
//
void WriteString(HANDLE f, LPCTSTR lpsz, int len)
{
   DWORD nWrite = len * sizeof(TCHAR);
   DWORD nActual;
   if (WriteFile(f, lpsz, nWrite, &nActual, NULL)) {
      // display results.
      _tprintf(_T("%d bytes written\n sizeof(TCHAR)=%d\n"), nActual,
                  sizeof(TCHAR));
   } else {
      _tprintf(_T("ERROR %d writing\n"), GetLastError());
   }
}
Figure 3 tstring in Action
Figure 3 tstring in Action

By the way, MFC's CString is now married to ATL so that both MFC and ATL use the same string implementation. The combined implementation uses a template class called CStringT that works like STL's basic_string in the sense that it lets you create a CString class based on any underlying character type. The MFC include file afxstr.h defines three string types, like so:

typedef ATL::CStringT<wchar_t,
  StrTraitMFC<wchar_t>> CStringW;
typedef ATL::CStringT<char,
  StrTraitMFC<char>> CStringA;
typedef ATL::CStringT<TCHAR,
  StrTraitMFC<TCHAR>> CString;
CStringW, CStringA, and CString are just what you would expect: wide, ASCII, and TCHAR versions of CString.

So which is better, STL or CStrings? Both classes are fine, and you should use whichever you like best. One issue to consider is which libraries you want to link with and whether you're already using ATL/MFC or not. From a coding perspective, I prefer CString for two features. First, you can initialize a CString from either wide or char strings:

CString s1 = "foo";
CString s2 = _T("bar");

Both initializations work because CString silently performs whatever conversions are necessary. With STL strings, you can't initialize a tstring without using _T() because you can't initialize a wstring from a char* or vice versa. The other feature I like about CString is its automatic conversion operator to LPCTSTR, which lets you write the following:

   CString s;
   LPCTSTR lpsz = s;
With STL, on the other hand, you have to explicitly call c_str. This is really nit-picking and some would even argue it's better to know when you're performing a conversion. For example, CStrings can get you in trouble with functions that use C-style variable arguments (varargs), such as printf:
   printf("s=%s\n", s); // Error: thinks s is char*
   printf("s=%s\n", (LPCTSTR)s); // required

Without the cast you can get garbage results because printf expects s to be char*. I'm sure many readers have made this error. Preventing this sort of mishap is no doubt one reason the designers of STL chose not to provide a conversion operator, insisting instead that you invoke c_str. In general, the STL folks tend to be a little more academic and purist types, whereas the Redmontonians are a little more practical and loosey-goosey. Hey, whatever. The practical differences between std::string and CString are slim.

Q I'm trying to expose my C++ library to C# and the .NET Framework using managed extensions and interop. One of my classes has a union in it, but .NET doesn't seem to like that:

class MyClass {
  union {
    int i;
    double d;
  };
};

The point of the union is to save space because I know the int and double are never used at the same time. Also, I have a lot of code that already references the union, and I prefer not to change it. How can I expose this class to .NET-compliant languages? Do I have to make each value in the union a separate member, or use methods instead?

John Bunt

A You could take either of those approaches, but you don't have to. In .NET interop, there's always a way to expose your C++ objects—well, almost always. The common language runtime (CLR) doesn't understand unions, but you can use an ordinary __value struct with some special voodoo to tell it where your members go. The magic attributes are StructLayout and FieldOffset. In managed C++, it looks like this:

[StructLayout(LayoutKind::Explicit)]
public __value struct MyUnion {
  [FieldOffset(0)] int i;
  [FieldOffset(0)] double d;
};
This tells the CLR that the integer i and the double d are both at offset zero (that is, the first items) in your struct, which makes them overlap and, in effect, the struct is a union. You can then use MyUnion in your __gc class, like so:
public __gc class CUnionClass {
public:
  // can access this directly because it's public
  MyUnion uval;
};

With CUnionClass defined, you can access the members i and d directly through uval from any .NET-compliant language. In C# it looks like the following code snippet:

CUnionClass obj = new CUnionClass();
obj.uval.i = 17;
obj.uval.d = 3.14159

I wrote all this up in a little program Called MCUnion that implements a managed C++ library with a CUnionClass like the one shown earlier, and a C# program utest that tests it (see Figure 4 and Figure 5). CUnionClass shows how to add properties for the union members, so you can access the values using obj.i and obj.d instead of obj.uval.i and obj.uval.d. This may or may not be what you want, depending on your design. If you like, you could make uval private/protected, so clients must use the properties. This would completely hide the union nature of uval. The test program accesses i and d both ways—through the union (uval) itself and through the properties i and d.

Figure 5 C-Style Unions (Close Figure 5)
////////////////////////////////////////////////////////////////
// MSDN Magazine — August 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//

// This module shows how to export a C-style union as a managed __value
// struct using StructLayout and FieldOffset. To compile this program and
// the C# test program that uses it, cd to this directory and type
//
// NMAKE
//
#using <mscorlib.dll>

using namespace System;
using namespace System::Runtime::InteropServices;

// Handy definitions to save typing ugly __'s.
#define DOTNET __gc
#define PROPERTY __property
#define VALUE    __value

namespace MCLib {

//////////////////
// Here's a typical C/C++ union:
//
// union {
//    int i;
//    double d;
// };
//
// And here's how to implement it for consumption by .NET:
//
[StructLayout(LayoutKind::Explicit)]
public VALUE struct MyUnion {
   [FieldOffset(0)] int i;
   [FieldOffset(0)] double d;
};

// This exported class contains the union as a member.
// The C# program test.cs shows how to use it.
//
public DOTNET class CUnionClass {
public:
   // can access this directly because it's public
   MyUnion uval;

   // additional access through properties
   PROPERTY int  get_i()      { return uval.i; }
   PROPERTY void set_i(int i) { uval.i = i; }

   PROPERTY double get_d()    { return uval.d; }
   PROPERTY void   set_d(double d) { uval.d = d; }
};

}
Figure 4 utest (Close Figure 4)
////////////////////////////////////////////////////////////////
// MSDN Magazine — August 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
// This C# program tests the union defined in MCLib.cpp.
//
using System;
using MCLib; // home-brew MC++ class lib

class MyApp {
   // main entry point
   [STAThread]
   static int Main(string[] args) {
      CUnionClass obj = new CUnionClass();

      // Note that this code accesses the embedded union both directly
      // through obj.val and indirectly through the properties i and d.
      obj.uval.i = 17;
      Console.WriteLine("Set i=17:");
      Console.WriteLine("    i={0}", obj.i);
      Console.WriteLine("    d={0}", obj.d);

      obj.uval.d = 3.14159;
      Console.WriteLine();
      Console.WriteLine("Set d=3.14159:");
      Console.WriteLine("    i={0}", obj.i);
      Console.WriteLine("    d={0}", obj.d);
      return 0;
   }
}

Q I'm writing a DirectX® screen saver and need to obtain a list of JPG, BMP, GIF, and TGA files from the user's My Pictures directory as a default before they learn where the screen saver settings are and can load them on their own. I have no problem setting the image textures into DirectX, but I'm afraid the My Pictures directory will be different for each user. On my machine it's "C:\Documents and Settings\Administrator\My Documents\My Pictures". Is there a simple way to get the location of my pictures?

Bruce MacFarlane

A Indeed, there is. The function you want is SHGetSpecialFolderPath, which gets the location of a special folder using a predefined ID like CSIDL_MYPICTURES, defined in ShlObj.h along with a bunch of other shell stuff. For example:

TCHAR buf[MAX_PATH];
SHGetSpecialFolderPath(NULL, // HWND
  buf,
  CSIDL_MYPICTURES,
  NULL); // don't create

You should always use SHGetSpecialFolderPath to get the path name of a special folder (as opposed to probing the registry directly) because it's guaranteed to work across all versions of Windows, including future ones even if the Redmondtonians ever decide to change the registry keys where the special folder path names are stored. For Windows 2000 and Windows XP, SHGetSpecialFolderPath lives in shell32.dll. Older versions of Windows such as Windows 9x and Windows NT® don't have SHGetSpecialFolderPath, but Microsoft provides it in a special DLL, SHFOLDER.DLL, which you're free to distribute with your app.

In fact, the official documentation from Redmond says, "Software vendors are encouraged to redistribute [SHFOLDER.DLL] as much as possible to enable this support on Windows operating systems prior to Windows 2000." The only catch is if you're targeting older versions of the Windows operating system, but building on Windows 2000 or Windows XP, you must explicitly link with SHFOLDER.DLL; otherwise the linker will grab SHGetSpecialFolderPath from Shell32.dll.

Since this is a C++ column, I wrote a little class called CSpecialFolder (see Figure 6) that's derived from CString and automatically calls SHGetSpecialFolderPath. To use it, all you have to write is:

CSpecialFolder mypics(CSIDL_MYPICTURES);
LPCTSTR lpszPath = mypics;

Figure 6 CSpecialFolder (Close Figure 6)
////////////////////////////////////////////////////////////////
// MSDN Magazine — August 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//

// Class to get the path name of a special folder. To use:
//
//    CSpecialFolder sf(CSIDL_MYPICTURES)
//
// Now the string holds the path name.
//
class CSpecialFolder : public CString
{
protected:
   BOOL GetSpecialFolder(int nFolder) {
      Empty();
      BOOL bRet = SHGetSpecialFolderPath(NULL, GetBuffer(MAX_PATH),
                                         nFolder, FALSE);
      ReleaseBuffer();
      return bRet;
   }

public:
   CSpecialFolder(int nFolder) {
      GetSpecialFolder(nFolder);
   }
   CSpecialFolder& operator=(int nFolder) {
      GetSpecialFolder(nFolder);
   }
};

The assignment works because CSpecialFolder is derived from CString, which has an implicit conversion operator to LPCTSTR. CSpecialFolder is available in the download, along with a test program that displays the path names of all the special folders that have CSIDL_XXX IDs defined in ShlObj.h. This includes familiar folders like Favorites, Fonts, Programs, History, and AppData—as well as some strange ones like CSIDL_BITBUCKET (Recycle Bin), CSIDL_INTERNET (path to Microsoft® Internet Explorer icon, I believe), and CSIDL_SYSTEMX86 (x86 system directory on RISC/Alpha systems for Windows 2000).

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


{ End Bracket }: C# to Visual Basic Translation Tool

{ End Bracket }
C# to Visual Basic Translation Tool
John Robbins

Code download available at: EndBracket0408.exe (206KB)

Having talked to thousands of developers who use the Microsoft .NET Framework, I've heard one consistent complaint: "I really wish all the samples were written in my programming language." Nothing is more frustrating than having braved the wilds of Internet searches for a snippet of code that does exactly what you want but is written in a language you don't use.

The good news is that you can translate code into your favorite language with a minimum of fuss. I certainly don't want to get involved in any nasty language wars, but for .NET, the two most commonly used languages are Visual Basic .NET and C#, so I'll focus on translating between those.

Since the .NET Framework forms the basis of life, most of the translation is easy (there's only one way to call a method like Debug.Assert). The rest of the translation for code surrounding the Framework method calls involves translating from one set of syntactic sugar to another. Since both Visual Basic .NET and C# show you the squiggly syntax errors, I plop smallish samples into the editor and use these errors to guide me through the translation. The more familiar you are with both languages, the easier this is to do. A good article on translating is "From VB.NET to C# and Back Again" by Darren Neimke and Scott Mitchell.

At first, I thought I could take some regular expressions I had been using and see if I could tie them together. After a few minutes of sketching, I quickly started sliding into writing a full-blown parsing engine. Since that wasn't what I wanted, I turned to every lazy developer's resource, Google, and looked to see if anyone else had done the work for me. The good news is that I found three great resources.

The first was Kamel Patel's C# to Visual Basic .NET conversion utility. Kamel has an ASP.NET app that does the conversion, but you can download the code to his parsing engine and run it locally as a Windows Forms app. While there are some bugs in the translation, Kamel's parsing engine is relatively simple to understand and should be easy to fix.

The next site was Alex Lowe's C# to VB.NET Translator. While Alex does not offer the code, his translation engine does a very nice job. C# #region statements are not translated, but comments are placed in the code indicating where the preprocessor directive was, so you can always quickly go back and poke them in. Alex's page mentions a Web service interface to his translation engine. Unfortunately, it always reports a parsing error a few lines into the code so I could not get it to work, but Alex should get the bugs wrung out by the time you read this.

The amazing Mike Krüger and crew responsible for the excellent open source #develop IDE have already built in code translation for converting between both languages. If you're looking for a great example of a large .NET-based application to see a robust architecture implementation, take a look at the #develop code. In #develop itself, to convert the current source buffer, simply pick the appropriate conversion from the Tools menu.

At the time I wrote this, #develop was in Beta 1 so the code translation is not feature complete and doesn't handle preprocessor directives and comments. The team is working feverishly on their 1.0 release and is planning to add those two missing pieces shortly.

All of these translation solutions are very nice for converting small source samples, but the real problem is that many times you need to convert a multi-file project. Having needed it myself numerous times, I wrote a tool, CSProjToVBProj, which takes a C# CSPROJ file and converts the whole project, including source, to a VBPROJ file that you can immediately open and compile.

Using CSProjToVBProj is very straightforward, but the implementation is interesting. For reasons beyond me, C# and Visual Basic .NET projects are just different enough to be annoying so there's lots of grunt work involved. For example, in C# all the using statements are in the code, so it's completely obvious what namespaces the developer is expecting. In Visual Basic .NET, most of the Import statements are in the <Imports> element inside the project file itself.

Another more logical hoop I had to jump through is ensuring that I properly update Web files with the explicit codebehind reference to look for the .vb file instead of the .cs file. For version 1.0 of CSProjToVBProj, I am using the #develop translation engine to do the source code translation so you'll need to download and build #develop first. The !READMEFIRST!.TXT file in the project explains the exact steps to build CSProjToVBProj.

If you're looking for a coding challenge, you may want to think about writing a VBProjToCSProj utility so we all can swap back and forth between language projects on a whim.