 MSDN Magazine February 2004

MSDN Magazine February 2004

Yukon Basics: XML, T-SQL, and the CLR Create a New World of Database Programming

The next version of SQL Server, code-named "Yukon," includes quite a few enhancements and expanded language support. For example, Transact-SQL now conforms more closely to the ANSI-99 SQL specification and makes querying more flexible and expressive. Yukon can execute user-defined functions, stored procedures, and triggers written in CLR-targeted languages, including Visual Basic .NET and C#. It supports a subset of the W3C standard XQuery language, and has native XML support.In this article, the author outlines the most significant language features and builds an order-entry sample app.

Eric Brown XML in Yukon: New Version Showcases Native XML Type and Advanced Data Handling

The next version of Microsoft SQL Server, code-named "Yukon," represents quite a few steps forward in the evolution of XML integration. Yukon supports native storage of XML data using the XML data type, which makes it possible to run native queries on XML data using the emerging industry standard XQuery language. Data integrity of the XML data type can be enforced through schema validation and XML-based check constraints, and special indexes can be defined that help speed up queries. In addition, Yukon has the built-in ability to expose its data through Web services. This article discusses these and other XML features of Yukon.

Bob Beauchemin T-SQL in Yukon: Powerful New T-SQL Syntax Gives SQL Server a Programmability Boost

The T-SQL language in the upcoming version of SQL Server will provide more power and flexibility than previous versions. Additions and enhancements include error handling with the TRY/CATCH construct, SNAPSHOT isolation, and WAITFOR enhancements. Also important are the BULK rowset provider, common table expressions, recursive queries, PIVOT and UNPIVOT operators, and much more. This article introduces these features so readers will be ready for the next version of SQL Server.

Itzik Ben-Gan Types in Yukon: Managed UDTs Let You Extend the SQL Server Type System

The next version of SQL Server, code-named "Yukon," will offer enhanced support for user-defined types (UDTs). Because UDTs can be managed by the CLR, you can represent a wide variety of data structures to create types not possible with previous versions of SQL Server. With UDTs you can more precisely control the kind of data your type accepts, resulting in better data management. This article explores UDTs in "Yukon" and covers their design and implementation.

Peter W. DeBetta Standard I/O: Console Appplications in .NET, or Teaching a New Dog Old Tricks

The Microsoft .NET Framework is not just about Windows Forms and Web services. This article discusses the simplest kind of Framework-based application—the console app—along with the frequently overlooked constructs of standard input/output and the pipe. When designed carefully, console applications offer a surprisingly powerful way of solving complex programming problems. One of the more exciting aspects of this approach is that while each application in the pipe is fairly simple, the result of their interaction can be a relatively complex task. Here the author explores the ins and outs of writing these console apps.

Michael Brook Timers: Comparing the Timer Classes in the .NET Framework Class Library

Timers often play an important role in both client applications and server-based components (including Windows services). Writing effective timer-driven managed code requires a clear understanding of program flow and the subtleties of the .NET threading model. The .NET Framework Class Library provides three different timer classes: System.Windows.Forms.Timer, System.Timers.Timer, and System.Threading.Timer. Each of these classes has been designed and optimized for use in different situations. This article examines the three timer classes and helps you gain an understanding of how and when each class should be used.

Alex Calvo Editor's Note: Yukon Bonanza

New Stuff: Resources for Your Developer Toolbox

Theresa W. Carey Web Q&A: ANSI Chars in XML, E-commerce Architecture, and More

Edited by Nancy Michell Cutting Edge: Owner-Drawing in .NET

Dino Esposito The XML Files: XML Report from the Microsoft PDC 2003

Aaron Skonnard The ASP Column: Web Services: ATL Server Versus ASP.NET

George Shepherd Wicked Code: Client-side Paging for DataGrids

Jeff Prosise C++ Q&A: Color Support, Console Apps, and Saving User Settings

Paul DiLascia Resource File: Mobile and Embedded Application Development


Yukon Basics: XML, T-SQL, and the CLR Create a New World of Database Programming

Yukon Basics

XML, T-SQL, and the CLR Create a New World of Database Programming

Eric Brown
This article was based on Beta 1 of Microsoft SQL Server Code Name "Yukon" and all information contained herein is subject to change
Note: This document was developed prior to the product's release to manufacturing and, as such, we cannot guarantee that any details included herein will be exactly the same as those found in the shipping product. The information represents the product at the time this document was printed and should be used for planning purposes only. Information subject to change at any time without prior notice.
SUMMARY

The next version of SQL Server, code-named "Yukon," includes quite a few enhancements and expanded language support. For example, Transact-SQL now conforms more closely to the ANSI-99 SQL specification and makes querying more flexible and expressive. Yukon can execute user-defined functions, stored procedures, and triggers written in CLR-targeted languages, including Visual Basic .NET and C#. It supports a subset of the W3C standard XQuery language, and has native XML support.

In this article, the author outlines the most significant language features and builds an order-entry sample app.

Contents
Transact-SQL Enhancements
SQL Server Yukon .NET Programming
Taking Advantage of the CLR
Choosing Between T-SQL and Managed Code
User-defined Types, Functions, and Aggregates
Managed Stored Procedures
XML and SQL Server Yukon
XML Data Type
XSD and XML Data Type
XML Query
Meet Microsoft XQuery Designer
Meet the SQL Server Service Broker
Database Integration
SQL Server Yukon Programmability Scenario
Conclusion

D
uring the planning phase of the next version of Microsoft® SQL Server™, code-named "Yukon," much consideration was given to the future of databases and the programming capabilities of SQL Server. Developers inside Microsoft realized early on that the future had to include a more symmetrical programming model, plus significantly more flexibility for different data models. The goal of programming model symmetry would mean that typical data access and manipulation tasks could be performed in a variety of ways—using XML, the Microsoft .NET Framework, or Transact-SQL (T-SQL) code.

The result of this planning is a new database programming platform that's been extended in a number of directions. First, the ability to host the .NET Framework common language runtime (CLR) extends the database into the land of procedural programming and managed code. Second, .NET Framework hosting integration provides powerful object database capabilities from within SQL Server. Deep support for XML was gained via a full-fledged XML data type which carries all the capabilities of relational data types. Additionally, server-side support for XML Query (XQuery) and XML Schema Definition language (XSD) standards was added. Finally, SQL Server Yukon includes significant enhancements to the T-SQL language.

These new programming models and enhanced languages come together to create a range of programmability that both complements and builds upon the current relational database model. The net result of this architectural work is the ability to create more scalable, reliable, robust applications, and an increase in developer productivity. Another result of these models is a new application framework called the SQL Service Broker—a distributed application framework for asynchronous message delivery. I'll get into a deeper discussion of the Yukon SQL Service Broker later in this article, but let's first look at the major changes and enhancements in the programming model, starting with T-SQL.

Transact-SQL Enhancements

Yukon provides more new language constructs and primitives for the T-SQL language than can be enumerated here. You can find them all in SQL Server Yukon Books Online. The enhancements to the T-SQL language reflect greater conformity to the ANSI-99 SQL specification, and were prompted by customer requests. Many of the improvements in T-SQL are focused on greater expressiveness in queries. There are several new query types that allow for common scenarios to be covered in T-SQL code. For example, a recursive query provides the ability to generate a bill of materials or a hierarchical resultset.

T-SQL in Yukon provides new PIVOT and UNPIVOT operators. These operators perform a manipulation on an input table-valued expression and produce an output table as a resultset. The PIVOT operator rotates rows into columns, optionally performing aggregations or other mathematical calculations along the way. It widens the input table expression based on a given pivot column and generates an output table with a column for each unique value in the pivot column. The UNPIVOT operator performs an operation opposite to the one PIVOT performs; it rotates columns into rows. The UNPIVOT operator narrows the input table expression based on a pivot column.

Yukon introduces a new and simple but very powerful exception handling mechanism in the form of a TRY/CATCH T-SQL construct. Transaction abort errors that used to cause a batch to terminate can now be caught and handled. Additionally, there are new language constructs for security, replication, Notification Services, XML, and all of the features that the .NET Framework provides. The .NET Framework implementation on the server is an important evolution in the SQL Server product cycle.

SQL Server Yukon .NET Programming

Developers can now write stored procedures, triggers, and user-defined functions (UDFs) with any Microsoft .NET-compliant language, including Visual Basic® .NET and C#. In addition, three new objects—user-defined types (UDTs), aggregates, and functions—can be created in managed code. The CLR is the heart of the .NET Framework and provides the execution environment for all .NET Framework-based code. The CLR provides various functions and services required for program execution, including just-in-time compilation, memory management and allocation, type safety enforcement, exception handling, thread management, and security. SQL Server Yukon acts as the host for this functionality in much the same way an operating system hosts an application.

As a result of CLR integration, Yukon also features automatic memory management, resource allocation, and garbage collection. SQL Server provides .NET assemblies direct access to database objects by entering the SQL Server execution space. Data access is achieved through a special form of ADO.NET. This article doesn't cover the ramifications of ADO.NET functionality within the database. However, because the access methods are similar to those developers are familiar with, using the functionality of the CLR is easy.

The way that Yukon integrates the .NET Framework with SQL Server provides several major benefits to database developers, including the following:

Enhanced programming model CLR-compliant languages are in many respects richer than T-SQL, offering constructs and capabilities previously not available to SQL developers. In addition, there is a set of class libraries (Framework APIs) which is much richer than the built-in functions SQL provides natively.

Enhanced safety and security Managed code runs in a CLR environment, hosted by the database engine. This allows .NET database objects to be safer and more secure than the extended stored procedures available in earlier versions of SQL Server.

User-defined types and aggregates Two new database objects that expand SQL Server storage and querying capabilities are enabled by hosting the CLR.

Common development environment Database development integration is a planned feature of future releases of the Visual Studio® .NET development environment. Developers use the same tools for developing and debugging database objects and scripts as they use to write middle-tier or client-tier .NET Framework components and services.

Performance and scalability In certain situations, the .NET language compilation and execution models deliver improved performance over T-SQL.

Taking Advantage of the CLR

In previous versions of SQL Server, database programmers were limited to using T-SQL when writing code on the server side. With CLR integration, database developers can now perform tasks that were impossible or difficult to achieve with T-SQL alone. Both Visual Basic .NET and C# are modern programming languages offering full support for arrays, structured exception handling, and collections. Developers can take advantage of CLR integration to write code that has more complex logic and is more suited for computational tasks than T-SQL can support.

What's more, Visual Basic .NET and C# offer object-oriented capabilities such as encapsulation, inheritance, and polymorphism. Related code can now be easily organized into classes and namespaces. When working with large amounts of code on the server, this allows you to more easily organize and maintain your code investments. This ability to logically and physically organize code into assemblies and namespaces is a huge benefit, and will allow you to better find and relate different pieces of code in a large database implementation.

When designing UDTs, you'll need to consider such things as nullability and check constraints. You can even use private functions to extend the capability of the data type; for example, you might natively add XML serialization capabilities.

Additionally, once registered, assemblies can reference other assemblies on the server. This means you can load an assembly that contains common functionality which can be used by one or more other assemblies on the server. This allows for incremental reuse of general code. The database developer will definitely appreciate this modular approach as the object model of their application gets larger and more complex.

SQL Server Yukon has implemented three levels of security that limit the capabilities of assemblies registered in the database. The security model is an integration of two security models: the SQL Server security model, which is based on user authentication and authorization, and the CLR security model, which employs code access security at the code level.

The highest security level, SAFE, allows only computation and access to data. The second level, EXTERNAL_ACCESS, also allows access to external system resources. The least secure level, UNSAFE, has no restrictions except for operations that compromise the stability of the server.

Choosing Between T-SQL and Managed Code

Managed code in the data tier allows you to perform number crunching and complicated execution logic on database objects. The .NET Framework features extensive support for string handling, regular expressions, error capture, and more. Additionally, with the functionality found in the .NET Framework base class library, database developers now have full access to thousands of pre-built classes and routines that can be easily accessed from any stored procedure, trigger, or UDF. For scenarios that require string handling, mathematical functions, date operations, system resource access, advanced encryption algorithms, file access, image processing, or XML data manipulation, managed store procedures, functions, triggers, and aggregates provide a much simpler programming model than T-SQL.

Another benefit of managed code is type safety. Before managed code is executed, the CLR performs server checks in order to verify that the code is actually safe to run. For example, the code's memory access is checked to ensure that no memory is being read if it hasn't been written to.

When writing stored procedures, triggers, and UDFs, one decision programmers will now have to make is whether to use traditional T-SQL or a CLR-compliant language such as Visual Basic .NET or C#. The answer depends on the needs of a particular project. T-SQL is best used when the code will perform mostly data access with little or no procedural logic. CLR-compliant languages are best suited for mathematically intensive functions and procedures that feature complex logic, or when you want to build a solution upon the .NET Framework base class libraries.

Code placement is another important feature of CLR integration. Both T-SQL and CLR-hosted code run inside the database engine on the server. Having .NET Framework functionality and the database close together allows you to take advantage of the processing power of a server machine. I recommend using the SQL Server Profiler to review the performance of your applications, and then make your choice based on the empirical results of your testing. The SQL Server Profiler has been enhanced to provide deep profiling of CLR performance within SQL Server, including a new graphical output format that can be used for comparison.

Now let's take a look at some of the capabilities of the .NET Framework and how they complement Yukon.

User-defined Types, Functions, and Aggregates

Yukon supports an extensible type system in the CLR. These types can be used as part of table definitions on the server side and can be manipulated as objects on the client side. UDTs allow you to extend the existing type system by creating new types whose implementation is in managed code. Some examples of UDTs are geospatial, custom financial, and specific date/time types. Think of UDTs as a contract between the server engine and the code.

In .NET terms, the UDT is a struct or a reference type, not a class or enum. This means that memory usage will be optimized. However, the UDT does not support inheritance and polymorphism. It can have both public and private functions. In fact, tasks such as constraint checking should be done as private classes. If your UDT is constructed of multiple pieces of information—for example, a geospatial type would contain longitude, latitude, and maybe altitude—you will need to provide private fields that contain those elements. Once you've created a UDT, you can use T-SQL to register the type in SQL Server. When this happens, the bits of the DLL are actually stored in the database.

UDFs come in two varieties: scalar-valued and table-valued. A scalar-valued function is one that returns a single value such as a string, integer, or bit. Table-valued functions return a resultset that can be comprised of one or more columns.

The ability to create additional aggregate functions beyond those already in the box was not available to developers using previous releases of SQL Server. With Yukon, developers can create custom aggregate functions in managed code and make these functions accessible to T-SQL or other managed code. The user-defined aggregate is also a .NET class that expects to reference an assembly already available in the database.

You can use a UDAgg to translate data stored in the database into mathematical values. Take, for example, a statistical function. You could store data containing results from a quantitative survey in a relation table. In order to figure out the weighted mean or standard deviation of these results, you could call a UDAgg that calculates and returns that information.

Managed Stored Procedures

Managed code routines that work as stored procedures are best used when you want to return a simple resultset to the caller with modifications, or if you want to simply execute a data manipulation language (DML) or data definition language (DDL) statement. Once registered on the server, taking advantage of the code is as simple as writing a stored procedure.

To wrap a managed code method using a stored procedure, use the CREATE PROCEDURE statement. CREATE PROCEDURE uses the syntax shown in this prototype:

CREATE PROCEDURE mysproc (@username NVARCHAR(4000))
AS
EXTERNAL NAME YukonCLR:[MyNamespace.CLRCode]:: myfunction

XML and SQL Server Yukon

The XML story in SQL Server Yukon really begins with SQL Server 2000. That version of SQL Server introduced the ability to return relational data as XML, bulk load and shred XML documents, and expose database objects as XML-based Web services. The Web Services Toolkit, also known as SQLXML 3.0, provides Web services capability to stored procedures, XML templates, and SQL Server UDFs. Two important pieces were missing in the XML technology: a native XML storage mechanism—the XML data type—and a sophisticated query language that supports querying semi-structured data. Yukon delivers these two elements, plus more Web services enhancements and extensions to the T-SQL FOR XML statement. Let's start with the central enhancement to the native SQL Server data types—the XML data type.

XML Data Type

XML has evolved from a representational technology to a wire format and is now seen as a storage format. Persistent storage in XML has become an interesting story, and there are many possible applications of the XML data type. First, XML is useful when you don't know the schema of the object. Second, the XML data type is useful for dynamic schemas in situations where information is reshaped on a continual basis. Third, XML persistence is central to XML document applications. There isn't a single application scenario that best exemplifies proper XML data type usage. The programming example I'll present later in this article uses XML as a storage format for a dynamic schema that represents a customer order in the form of a message to the server.

The XML data type is a full-fledged data type. It has all the powers and capabilities of the other types found in SQL Server. The importance of this can't be overstated. As a full-fledged type, the XML column can be indexed, it can have row and column constraints via XSD schema (although an XSD schema is not needed), and it can be queried using an XQuery expression embedded in T-SQL. These functions are appended to the XQuery W3C specification. In addition, using the xmldt::modify method, the user can add subtrees, delete subtrees, and update scalar values.

The XML data type is flexible. You can choose whether you want to associate an XML Schema Definition (XSD) schema with a column. When a column has an XSD schema association, it is known as typed XML. Without the XSD association, it's untyped XML. An XSD schema can be used to type an XML column after it's imported into the database. It can then be used to create indexes and other metadata for the system catalog. Typed XML columns are significantly faster and more flexible than untyped columns when you're looking to query them. The usage scenario will dictate the XML column type, though most applications will appreciate an XSD schema reference.

Note also that you can store both XML documents and XML fragments in the same column. In addition, you have to create a special "XML" index on XML columns. Doing this creates indexes for tags, values, and paths in the XML value.

XSD and XML Data Type

When creating your XML column, you should add an XSD document and associate it with the column. The XSD schema association is executed from within the SQL Server Workbench. The XSD association is also possible using DDL:

CREATE TABLE T (pk INT, xCol('mySchema'))
When a schema is imported into the database, it is parsed into various types of components, including ELEMENT, ATTRIBUTE, TYPE (for simple or complex types), ATTRIBUTEGROUP, and MODELGROUP. These components, and any associated namespaces, are then stored in various system tables. You can view these components using dedicated system views. In other words, the schema is not stored intact. This has at least two ramifications. First, the schema tag and its associated attributes are not stored. Instead, the XML tag attributes—targetNamespace, attributeFormDefault, elementFormDefault, and so on—are reassigned to the schema's components. Second, SQL Server makes no provision for recovering and viewing the original schema document once it is imported. It is recommended that you keep a copy of all schemas or store their contents in a dedicated database table. For this, an XML column will suffice. Alternatively, you can retrieve XML schemas using the built-in function XML_SCHEMA_NAMESPACE ('uri'). This returns the content of namespace 'uri'.

When the XML column and an XSD schema are combined, the query capabilities for XML data are complementary to normal relational data and queries. While I don't recommend storing all your data in XML, the ability to take advantage of XML in the database makes XML document management and other application scenarios a breeze.

Now that the data has been stored, I'll move on to XQuery, as my discussion is incomplete without it.

XML Query

The XML Query Language, commonly referred to as XQuery, is an intelligent and robust language optimized for querying all types of XML data. With XQuery, you can run queries against variables and columns of the XML data type using the type's associated methods. With the invocation of the query method, you can query across relational and XML data in the same batch.

As with many of the XML standards, the development of XQuery, which is currently a working draft, is overseen by the W3C. Yukon supports a statically typed subset of the XQuery 1.0 Working Drafts in the range of December 20, 2001 to November 15, 2002. See http://www.w3c.org for all of the drafts.

XQuery evolved from a query language called Quilt, which was based on a variety of other languages such as XPath 1.0, XQL, and SQL. It also contains a subset of XPath 2.0. You can use your existing skills with XPath and not have to learn an entirely new language. However, SQL Server Yukon contains significant enhancements that go beyond XPath, such as special functions and support for better iteration, sorting of results, and construction.

XQuery statements consist of a prologue and a body. The prologue contains one or more namespace declarations and/or schema imports that create the context for query processing. The body contains a sequence of expressions that specify the query criteria. These statements are then used within one of the aforementioned methods of the XML data type (query, value, exist, or modify).

XQuery is capable of querying typed XML (data that is associated with an XML schema) as well as an XML document.

Meet Microsoft XQuery Designer

XQuery Designer is a new tool, integrated into the new SQL Server Workbench, which makes working with XML data easy. XQuery Designer helps you write queries that manipulate and retrieve data from both XML columns and XML documents. The central development theme for the XQuery Designer was to empower XQuery development without making users learn the ins and outs of the XML Query Language. For a peek at the XQuery Designer in action see Figure 1.

Figure 1 The XQuery Designer in Action (Close Figure 1)

Up to this point, I've talked about the new .NET functionality and XML functionality of Yukon as separate entities. The combination of these technologies creates new opportunities to solve application scenarios in popular areas like e-commerce. SQL Server development is newly focused on distributed applications, where business workflow and logic is represented in the database application. Using the powerful support for Web-based applications that's built into Yukon, let's delve deeper into the new functionality I've presented here.

Meet the SQL Server Service Broker

Over the last 10 years, the proliferation of e-commerce applications has created a need for increased process management across database applications. If you've built an order entry system or made an online purchase, you are familiar with the workflow model. When a customer places an order for a book, a transaction must be committed into the inventory, the shipping, and the credit card systems, and an order confirmation must be sent through another Web application. Waiting for each of these processes to happen synchronously doesn't scale well. In the past, developers had to write complicated stored procedures and use remote procedure call code to queue messages. Yukon provides a new, scalable architecture for building asynchronous message routing.

The SQL Server Service Broker is a Yukon technology that allows internal or external processes to send and receive guaranteed, asynchronous messages using extensions to normal T-SQL DML. Messages can be sent to a queue in a database shared with the sender, to another database in the same SQL Server instance, or to another SQL Server instance either on the same server machine or on a remote server.

In traditional messaging systems, the application is responsible for ordering and coordinating messages. Message duplication, synchronization, and order are all difficult problems enterprise data systems must deal with.

SQL Server Service Broker solves these problems by automatically handling message order, unique delivery, and conversation identification. Once a conversation is established between two service broker endpoints, an application receives each message only once, in the order in which the message was sent. Your application can process messages exactly once, in order, without additional code. Service Broker automatically includes an identifier in every message. An application can always tell which conversation a particular message belongs to.

Service Broker queues messages for delivery. If the target service is not immediately available, the message remains with the sending service and delivery attempts are repeated until the message is sent successfully. This allows a conversation to continue reliably between two services, even if one service is temporarily unavailable at some point during the conversation.

SQL Server Service Broker provides loose coupling between the initiating service and the target service. A service can put a message on a queue and then continue with its application processing tasks, relying on SQL Server Service Broker to ensure that the message reaches its destination. This loose coupling enables scheduling flexibility. The initiator can send out multiple messages, and multiple target services can process them in parallel. Each target service processes messages at its own pace, depending on the system's current workload.

Queuing also allows systems to distribute processing more evenly, reducing the peak capacity required by a server. This can improve overall throughput and performance in database applications. For example, an order entry application might see an increase in requests at noon every day, resulting in high resource usage and slow response times. With Service Broker, the order entry application need not perform all of the processing for an order when the application receives it. Instead, the application can enter the order and submit requests for background processing such as billing, shipping, and inventory management. Background applications perform the processing reliably over a period of time, while the order entry application continues to receive new orders.

One of the most difficult things to accomplish in a messaging application is to allow multiple programs to read in parallel from the same queue. Situations like this can cause messages to be processed out of order, even if they're received in order.

Consider a traditional order processing application. Message A, containing instructions about creating the order header, and Message B, containing instructions about creating the order line items, are both received on the queue. If both of these messages are dequeued by separate service program instances and processed at the same time, it is possible that the order line item transaction will attempt to commit first, and fail because the order does not yet exist. The failure causes the transaction to roll back and the message to be requeued and processed again, wasting resources. Traditionally, this problem was solved by combining the information from Message A and Message B in a single message. While this approach is straightforward for two messages, it scales poorly to systems that involve coordinating dozens or hundreds of messages.

Service Broker solves this problem by automatically locking all messages related to the same task, so that these messages can only be received and processed by one service program instance. Meanwhile, other service program instances can continue to dequeue and process messages related to other tasks. This allows multiple parallel service programs to work reliably and efficiently.

One of the most useful features of Service Broker is activation. Activation automatically starts a service program to read messages from a queue as they arrive. If messages are arriving faster than they are being handled, additional instances of the service program are started, up to a configured maximum. If the message arrival rate is reduced and an active service program checks the queue and finds that there are no messages available for processing, the service program shuts down. This allows the number of service program instances to grow and shrink dynamically as the load on the service changes. If the system goes down or is rebooted, service programs are automatically started to read the messages in the queue when the system comes back up. Traditional messaging systems lack this behavior, and frequently end up having either too many or too few resources dedicated to a particular queue at any given time.

Database Integration

The integrated design of Service Broker provides benefits for application performance and administration. Integration with SQL Server allows transactional messaging; that is, each loop of a service program—receiving a message, processing the message, and sending a reply message—can be enclosed in a single database transaction. If the transaction fails, all work rolls back, and the received message is requeued so that another attempt can be made to process it. No actions take effect until the application commits the transaction. The application remains in a consistent state, without the added resource overhead and complexity of a distributed transaction coordinator.

Administration is easier when your data, messages, and application logic are all in the database. Only one item—instead of three or four separate components—requires maintenance for disaster recovery, clustering, security, backup, and so on. For instance, with traditional messaging systems, the message store and the database can become out of sync. For example, when one component is restored from a backup, the other component must also be restored from a backup taken at the same time, or the message store and the database may be inconsistent. With a single database for both messages and data, this is no longer an issue.

Using a common development environment is also a benefit. The messaging and data parts of an application can use the same languages and tools in a Service Broker application. This extends the developer's familiarity with database programming techniques to encompass message-based programming. Stored procedures that implement a Service Broker service can be written in either T-SQL or one of the CLR-targeted languages.

Additionally, database integration makes automatic resource management possible. Service Broker runs in the context of the SQL Server instance, so the broker can maintain an aggregate view of all messages ready to be transmitted from all databases in the instance. This allows each database to maintain its own queues while still maintaining fairness in resource usage across the entire SQL Server instance.

SQL Server Yukon Programmability Scenario

Let's see how these concepts come together in an order entry application where a customer places orders on a Web site. The messages sent between the services are stored in an XML column. For illustration purposes, I'll use an ASP.NET-based Web service that's registered and hosted by an instance of the CLR within SQL Server. This service communicates with a partner to send and receive purchase order information. Using an XML column for the messages allows for schema encapsulation and simplification.

The customer places an order on the Web site. When the order transaction commits, a message containing the order information is put in the Order Entry Service Queue. The order is sent to the Order Entry Service as an XML column. This allows all the order information to be consolidated into one column. For a full view of the process, see Figure 2.

Figure 2 An Order Entry Application (Close Figure 2)

The Order Entry Service starts a transaction, receives the message, and processes it. The Order Entry Service then sends a request on to the Credit Limit Service to verify credit status. If it is okay, the Order Entry Service moves to the next step.

The Order Entry Service checks the inventory for each item in the order. An XQuery processes the inventory check. At this point, if the item is available for shipping, a message is sent to the Shipping Service Queue.

The Shipping Service, a T-SQL stored procedure, uses XQuery to parse the XML message. The customer information is used to generate an order shipment. When the shipment is complete, a message is sent to the Billing Service Queue. When the Billing Service Queue receives the "ship complete" message, it updates the order status in the database to indicate the order has been shipped. All messages to and from the Order Entry Service are written to an audit table for later analysis and problem resolution.

You can see from this simple example how the new Yukon programmability features—XML, CLR, Service Broker, and T-SQL enhancements—can be used to build reliable, scalable, and complete database applications.

Conclusion

I've touched on just a few of the new and exciting architectural capabilities in Yukon. Database developers have never had so many choices for data access and storage. The SQL Server Service Broker example highlights only a fraction of the capabilities found when combining the new capabilities of SQL Server. Start working with the SQL Server Yukon beta and stay tuned to MSDN® Magazine for future coverage. Before you know it, you'll be using Yukon and never looking back.

Eric Brown joined the SQL Server team at Microsoft almost three years ago. Previously, he worked in data development at greatfood.com and other small companies.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


XML in Yukon: New Version Showcases Native XML Type and Advanced Data Handling

XML in Yukon

New Version Showcases Native XML Type and Advanced Data Handling

Bob Beauchemin
This article was based on Beta 1 of Microsoft SQL Server Code Named "Yukon" and all information contained herein is subject to change.
Code download available at: XMLinYukon.exe (114KB) Note: This document was developed prior to the product's release to manufacturing and, as such, we cannot guarantee that any details included herein will be exactly as the same as those found in the shipping product. The information represents the product at the time this document was printed and should be used for planning purposes only. Information subject to change at any time without prior notice.
SUMMARY

The next version of Microsoft SQL Server, code-named "Yukon," represents quite a few steps forward in the evolution of XML integration. Yukon supports native storage of XML data using the XML data type, which makes it possible to run native queries on XML data using the emerging industry standard XQuery language. Data integrity of the XML data type can be enforced through schema validation and XML-based check constraints, and special indexes can be defined that help speed up queries. In addition, Yukon has the built-in ability to expose its data through Web services. This article discusses these and other XML features of Yukon.

Contents
The XML Data Type
XML Indexes
The XML Data Type Functions
XQuery 1.0—An XML Query Language
Using The XQuery Functions
XQuery Extended Functions in SQL Server
XML DML—Updating XML Columns
FOR XML and OpenXML Enhancements
Native Web Service Support
Wrap-up

T
he next version of Microsoft® SQL Server™, code-named "Yukon," takes the groundbreaking XML support that was introduced in SQL Server 2000 and enhances it to include even more innovative functionality and ease of use. Yukon adds native XML data storage to the database management system (DBMS) through a new native XML data type. The introduction of this native XML data type, coupled with the emerging industry standard XQuery language, should spark a revolution in database application development.

Support for the native XML data type is extensive. It includes XML Schema-based validation and additional XML-based constraints, special XML Infoset-based indexes, and queries over XML content objects using XQuery. In addition to this radical new functionality, the existing SQL Server 2000 XML functionality has been fine-tuned for better performance and ease of use.

XML itself is a platform-independent data representation format that was used initially as a document format. As XML has gained wide acceptance, users have sought to solve tough business problems like data integration via XML. This has led to the evolution of XML as a data storage format. XML, like its structured counterpart, has its own type system based on the XML Schema Definition language (XSD). Both XML and XSD are W3C standards at the recommendation level. An XML Schema defines the format of an XML document, just as a SQL Server schema defines the layout of objects in a SQL Server database. The XML type system is quite rigorous; XSD can define almost all of the constructs available in relational databases, object-oriented type systems, and semi-structured and unstructured document formats. (See the sidebar "Standards Compliance".)

Figure 1 Hierarchical XML Returned from FOR XML
Figure 1 Hierarchical XML Returned from FOR XML

XML data is hierarchical, and XML documents used in business applications are most often comprised of data that is stored in multiple relational tables. For example, an XML order document might contain customer information, order information, and order detail information. In a relational database, this information would typically be stored in multiple tables. In SQL Server 2000, you had two choices if you wanted to store XML: store the entire document in a text field, or decompose the document into multiple relational tables. If you chose text field storage, you would use SQL Server only to store and retrieve the data; queries against the document content would occur in client-side libraries.

Figure 2 From OpenXML to Tables
Figure 2 From OpenXML to Tables

In SQL Server 2000, by using the FOR XML extension to the Transact-SQL (T-SQL) SELECT statement you could query multiple relational tables and compose the returned data into a hierarchical XML document, as shown in Figure 1. The system-defined OPENXML function accomplished document decomposition, as shown in Figure 2, and also permitted distributed relational queries between XML and relational data. In Yukon this support is enhanced to include composition, decomposition, and distributed relational queries. It also includes native XML queries and native XML storage, as shown in Figure 3.

Figure 3 XML Storage
Figure 3 XML Storage

The XML Data Type

Let's look at the XML data type and see how it differs from the conventional storage of an XML document in a text data type. The new data type is a first-class type; you can use it in most of the ways any other SQL Server data type can be used, including as a column in a table, a variable in T-SQL, a stored procedure or user-defined function parameter, or a user-defined function return value.

The XML type is quite similar to the concept of a distinct type defined by SQL-99 (the latest version of the ANSI SQL standard) and to the character-based large object (CLOB) types varchar(max) and nvarchar(max). You must convert it to and from a varchar or nvarchar type, although the T-SQL INSERT statement will do automatic conversion from a varchar and nvarchar value used in a VALUES list to an XML data type column value. You cannot CAST or CONVERT the XML type to any type other than varchar or nvarchar. Two instances of an XML data type cannot be compared. Like a SQL-99 distinct type, the XML type has its own methods; these methods enable you to use an alternate query language, XQuery. The data in an XML type does not follow the relational data model, but it does follow the XQuery 1.0 and XPath 2.0 data model.

The primary reason for having an XML data type is that when you define an XML data type column, the data in that column is stored in the database itself. The column is not a pointer to an XML document on the file system. This means that XML data is included in the backup and restore process, is subject to ordinary SQL Server security, and participates in transactions and logging. Having XML data inside a relational database may offend some relational purists, but it means that your data lives in a single repository for reasons having to do with administration, reliability, and control.

Here's an example of using an XML type in a table. The table stores invoice documents in XML format.

CREATE TABLE invoices (
    id  INTEGER IDENTITY PRIMARY KEY,
    xmlinvoice XML)

Note that you cannot use the XML column itself as a primary key. This is because instances of the XML data type are not directly comparable, though they can contain (and be tested for) a NULL value. Although it would not be difficult to perform a string comparison with XML data, it would be an order of magnitude more difficult to perform a comparison at the data level. For example, the following two XML documents are data-equivalent, but nowhere near lexically equivalent:

<!-- These two documents are equivalent -->
<doc1>
   <row au_id="111-11-1111"/>
</doc1>

<doc1>
   <row au_id='111-11-1111'></row>
</doc1>

You can store XML documents or XML fragments in an instance of the XML data type. XML fragments are data instances that follow the rules for well-formed XML, but do not necessarily contain a single root element. Well-formed means simply that the document conforms to the syntax defined in XML 1.0.

As an example, the output of the T-SQL FOR XML query is an XML fragment; it doesn't have a root element. Permitting fragments as well as documents is consistent with the XQuery 1.0 and XPath 2.0 data model. The data that you store in an XML data type must be well formed, however.

Because XML data types follow the XML XQuery 1.0 and XPath 2.0 data model, they are not constrained by relational constraints, but by an XML Schema. In addition to storing XML documents or fragments in an XML data type column, you can also use SQL Server to validate your XML data type column by associating the column with an XML Schema. Therefore, you can think of XML data types, which can contain documents or fragments, as being schema-valid—that is, conforming to a type defined in a schema or merely well formed.

XML Schemas define a set of data types that exist in a particular namespace. Schemas are themselves well formed, and conform to a schema called the schema for schemas. Schemas are valid XML documents, just as relational table definitions and other data definition language (DDL) statements are valid T-SQL. Although there can be more than one schema definition document for a particular namespace, a schema definition document defines types in only one namespace. XSD defines a standard set of base types that are supported as types in XML documents, in just the same manner as the SQL-99 standard defines a set of base types that relational databases must support.

In order to be able to validate an XML document with its corresponding schema, SQL Server Yukon stores XML Schemas in the database. Schema documents are catalogued in the SQL Server Engine by using the CREATE XMLSCHEMA DDL statement, as shown in the code in Figure 4.

Figure 4 CREATE XMLSCHEMA (Close Figure 4)
CREATE XMLSCHEMA'<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
         xmlns:tns="http://example.org/People"
         targetNamespace="http://example.org/People" >
   <xsd:simpleType name="personAge" >
     <xsd:restriction base="xsd:float" >
       <xsd:maxInclusive value="70" />
       <xsd:minExclusive value="0" />
     </xsd:restriction>
   </xsd:simpleType>
   <xsd:element name="age" type="tns:personAge" />
</xsd:schema>'

An XML Schema is dropped from the database by referring to its target namespace. For example, the schema I just catalogued would be dropped by the following statement:

DROP XMLSCHEMA NAMESPACE 'http://example.org/People'
Note that no matter how the database is configured, namespaces are case-sensitive.

XML data type columns, parameters, and variables may be typed or untyped—that is, they may conform to a schema or not. To specify that you're using a typed column (for example, in the XML column in Figure 4), you would specify the schema namespace in parentheses, like this:

CREATE TABLE xml_tab (
   the_id INTEGER,
   xml_col XML('http://example.org/People')
)
By doing this, you've just defined a series of integrity constraints with respect to what can appear in the XML that comprises that column. Typing the XML data in a column by using an XML Schema not only serves as an integrity constraint, but as an optimization for the SQL Server XQuery engine because you are using typed data in the query. This also allows the XQuery engine to know the data type of its intermediate and final results. If the XML data is not strongly typed, XQuery can either perform implicit type conversions based on the syntax of the query, as T-SQL does in some cases, or treat everything as a string data type.

When an XML Schema is stored in SQL Server, it is not stored directly as an XML document. Instead, it is shredded into a proprietary format that is useful for optimizing schema validation. Once you have stored your XML Schema, it's possible for SQL Server to reconstruct most of it using the xml_schema_namespace function. However, items such as annotations and comments are not reconstructed by this function, so you may need to keep track of the original schema document separately. A convenient way to keep a full-fidelity copy of a schema for later use would be to store it in a separate table as a varchar(max) column.

XML Indexes

The syntax for creating an index on an XML column is similar to what you'd use to create a SQL index:

CREATE XML INDEX xml_idx
  ON xml_tab (xml_col)
Although this looks like a normal index creation statement with an extra XML keyword, the actual effect of the statement is much different than what you get when you create a traditional index. What you are creating in the case of an XML column is an index over the internal representation of the column for the purpose of optimizing XQuery queries, not SQL queries. Remember that the SQL comparison operators cannot be used on an XML column. Because XML indexes and SQL indexes share the same value space in a database, you cannot create an XML index and a SQL index with the same index name. You can, however, also create a full-text index on the contents of an XML type column; this is unrelated to an XML index. (The details of full-text search are outside the scope of this article.)

The XML Data Type Functions

In addition to being able to use an XML data type as a column in a table or view, variable, or parameter in its entirety, the XML data type contains a variety of type-specific methods. These are invoked by using a double-colon variable::method syntax. The XML type functions encompass a couple of different types of functionality: determining if a node or nodes exist for a given XQuery path expression, and querying and modifying the value of the XML type through XQuery.

The XML data type is, by definition, a complex type—that is, a type that almost always contains more than one data value. You can use XML data type columns in two distinct ways. First, you can query the data and perform actions on it using T-SQL as though it were a simple data type. This is analogous to the way you would treat an XML file in the file system; you read the entire file or write the entire file. Second, you can run queries and actions on the XML data in the column, using the fact that the XQuery 1.0 and XPath 2.0 models allow the concrete data to be exposed as a series of data values or nodes. For this you need an XML query language.

XQuery 1.0—An XML Query Language

In addition to the SQL Server query functions, Yukon supports a set of native XQuery functions that are designed specifically for the XML data type. Although XQuery appears to be nominally SQL-like, it's designed especially to handle the semi-structured nature of XML data, including strong recursion capabilities and node navigation. In addition to the query capabilities of XPath, XQuery allows element and attribute construction via XSLT. A native XQuery 1.0 engine and an XPath 2.0 parser live inside of the Yukon query processor alongside the relational engine. Unlike XPath 1.0, XQuery 1.0 and XPath 2.0 are strongly typed query languages. This is quite evident in the wide range of constructors and casting functions for different data types. The reasoning behind this is that XQuery is built to be optimized and uses strong types rather than strings. This allows the query engine to do a better job in optimizing and executing the query. Just imagine if the T-SQL query parser in SQL Server had to deal with everything as a single data type!

XML data type columns, XML variables, and XML procedure parameters can be strongly typed by reference to an XML Schema, or they can be untyped. When typed XML is used in an XQuery, the query parser has more information to work with. If untyped XML is used, the query engine must start with the premise that every piece of data is a string type; the built-in data function can be used with strongly typed data to return the exact data type.

XQuery supports static type analysis, meaning that if the types of items that are input into an expression are known, the output type of an expression can be inferred at parse time. In addition, strong typing permits the XQuery language to syntax-check the input based on types at parse time, just as T-SQL can. This results in fewer implicit type conversions at run time. SQL Server supports XQuery on the XML data type directly. It does this through three XML data type methods:

  • xml::exist takes an XQuery as input and returns 0, 1, or NULL depending on whether the query returns an empty sequence (0), a sequence with at least one node (1), or the column is NULL.
  • xml::value takes an XQuery as input and returns a SQL type as a scalar value.
  • xml::query takes an XQuery as input and returns an XML data type as a result.
Standards Compliance

In addition to the prolific specifications for everything XML, guided by the W3C, the ANSI SQL committee has gotten into the act with an effort to standardize the integration of XML and relational databases. This series of specifications started under the auspices of the SQLX committee of database vendors, but has been subsumed as part 14 of the ANSI SQL 200n specification. A committee that includes representatives from Microsoft, Oracle, IBM, and other companies is working on this part of the SQL specification.

This specification touches on a number of subjects that I've discussed in this article. Although OpenXML and FOR XML predate the standard and are not included in it, the new XML features in SQL Server follow the standard closely. The XML data type in Yukon is patterned after the XML data type defined by the ANSI SQL specification. The data model of the XML data type defined by ANSI SQL part 14 is based on the XPath 2.0 and XQuery 1.0 data model. This data model permits an XML data type to contain XML documents, document fragments, and top-level text elements. The SQL Server XML data type adheres to this standard.

Because both the XPath and XQuery data model and ANSI SQL part 14 are specifications in progress, the specs may be out of sync at various points in the standardization process. SQL Server uses the standard way to encode SQL names as XML names (XML names have different escape character rules), and almost the exact rules for converting between SQL types and XML types.

The mapping of the database to a virtual XML document is implemented in the client libraries and will be enhanced in the next release of Visual Studio® .NET and the .NET Framework, code-named "Whidbey." This mapping is a superset of the single canonical mapping in the proposed spec.

The dialect of XQuery that's currently implemented using these functions in the prerelease version of SQL Server Yukon reflects the December 2001 version of the XQuery specification as a base. As SQL Server Yukon and the XQuery specifications move closer to production, the syntax will become more closely aligned to the current working specification.

Although the XQuery language itself can be used to construct complex documents, only the xml::query function takes full advantage of this feature. When using xml::value and xml::exist, you'll normally create simple XPath 2.0 expressions rather than complex XQuery with sequences, constructors, formatting information, and so on. This is because xml::value can only return a sequence containing a single scalar value (it cannot return an XML data type), and xml::exist is used to check for Boolean existence.

Using The XQuery Functions

The xml::exist function will search the specified XML column, looking up data that matches the given XQuery expression. For each row, if it contains matching data in the specified column, the function returns true (1). The XML functions can be used in check constraints on XML columns, either in addition to or instead of XML Schema constraints. Here's an example:

-- pdoc must have a person element
-- as a child of the people root
CREATE TABLE xmltab(
  id INTEGER PRIMARY KEY,
  pdoc XML CHECK (pdoc::exist('/people/person')=1)
)

The xml::value function takes a textual XQuery as a string and returns a single scalar value. The SQL Server type of the value returned is specified as the second argument, and the value is cast to that data type. The data type can be any SQL Server type except a SQL Server XML data type, a user-defined type, or a SQL Server timestamp data type.

The xml::value function must return a single scalar value, or else an error is thrown. You don't need to have xml::value to return a single XML data type—that's what xml::query does.

Using a simple document and table values such as this

CREATE TABLE xml_tab(
  the_id INTEGER PRIMARY KEY IDENTITY,
  xml_col XML)
GO

INSERT xml_tab values('<people><person name="curly"/></people>')
INSERT xml_tab values('<people><person name="larry"/></people>')
INSERT xml_tab values('<people><person name="moe"/></people>')
GO
the following SQL query using xml::value would produce a single row containing the ID and the name for each row.
SELECT the_id,
       xml_col::value('/people/person/@name', 'varchar(50)') AS name
  FROM xml_tab
The result would look like this:
 the_id      name
--------  ------------------------
   1         curly
   2         larry
   3         moe

Although all of the functions of the XML data type invoke the internal query processor, the function that actually executes a complete XQuery on an instance of the XML data type is xml:query. This function takes the XQuery text, which can be as simple or complex as you like, as an input string and returns an instance of the XML data type. The result is always an XQuery sequence as the spec indicates, except when your XQuery instance is NULL. If the input instance is NULL, the result is NULL. The resulting instance of an XML data type can be returned as a SQL variable, used as input to an INSERT statement that expects an XML type, or in can be returned as an output parameter from a stored procedure.

In SQL Server Yukon, remember that the xml::query function refers to a specific column (or instance of the XML data type), and input will come from that column or instance's data—this is the context document. The simplest XQuery might consist of nothing but an XPath 2.0 expression. This would return a sequence containing zero or more nodes or scalar values. Given the previous document, a simple XQuery

SELECT xml_col::query('/people/person') AS persons
  FROM xml_tab
would return a sequence containing a single node for each row:
persons
-----------------------------------
<person name="curly"/>
<person name="larry"/>
<person name="moe"/>

XQuery differs from XPath in that, in addition to querying a document, it may be used to compose a document utilizing what are known as node constructors. This is analogous to XSLT, but implemented differently. Figure 5 contains an example that constructs a Payment XML document from an Invoice document. The invoice document used and the payment document produced are included in the code download for this article (see the link at the top of this article).

Figure 5 Constructing XML (Close Figure 5)
SELECT invoice::query('
    {-- XQuery program --}
    namespace inv="urn:www-develop-com:invoices"
    namespace pmt="urn:www-develop-com:payments"
    for $invitem in //inv:Invoice
    return
      <pmt:Payment>
        <pmt:InvoiceID> { data($invitem/inv:InvoiceID) } </pmt:InvoiceID>
        <pmt:CustomerName> { data($invitem/inv:CustomerName) }
            </pmt:CustomerName>
        <pmt:PayAmt> { data(sum($invitem/inv:LineItems/inv:LineItem/
            inv:Price)) }</pmt:PayAmt>
      </pmt:Payment>
  ') as xmldoc
  FROM xmlinvoice

XQuery Extended Functions in SQL Server

Because the XQuery engine is executing in the context of a relational database, Yukon provides two convenient, standard ways to use non-XML data (that is, relational data) inside the XQuery itself. The extension functions sql:column and sql:variable serve this purpose. These functions allow you to refer to relational columns in tables and to T-SQL variables from inside an XQuery.

These functions can be used anywhere an XQuery can be used, namely xml:exist, xml::value, xml::query, and also xml::modify (which is discussed in the following section of this article). Figure 6 contains a user-defined function that returns all overpriced items in a particular invoice.

Figure 6 UDF to Return Items (Close Figure 6)
CREATE FUNCTION OverPricedItems(@price money, @theid int)
RETURNS xml
AS
BEGIN
 declare @x as xml
 select @x = invoice::query('
    {-- XQuery program --}
    namespace inv="urn:www-develop-com:invoices"

    for $invitem in //inv:Invoice/inv:LineItems/inv:LineItem
    where $invitem/inv:Price > sql:variable("@price")
    return
      <OverPriced>
        <Sku> { data($invitem/inv:Sku) } </Sku>
        <Price> { data($invitem/inv:Price) } </Price>
      </OverPriced>
  ')
 from xmlinvoice
 where id = @TheID
 return @x
END

XML DML—Updating XML Columns

One thing that was left out of the XQuery 1.0 specification was a definition of an XQuery syntax for mutating XML instances or documents in place. Currently there is no announced plan to add a data manipulation language (DML) to the XQuery specification, although a "standard extension to the standard" is being discussed by some vendors. Because Yukon will use XQuery as the native mechanism for querying the XML data type inside the server, it is required to have some sort of manipulation language. The alternative would be to give users only the ability to replace the instance of the XML type as an entire entity. Since changes to XML data type instances should participate in the current transaction context, this would be equivalent to using SELECT and INSERT in SQL without having a corresponding UPDATE statement. Therefore Yukon introduces a nonstandard implementation of XML DML.

XML DML is implemented using XQuery-like syntax with SQL-like extensions. This emphasizes the fact that manipulating an XML instance inside SQL Server is equivalent to manipulating a complex type or, more accurately, a graph of complex types. You invoke XML DML by using the xml:modify function on a single XML data type column, variable, or procedure parameter. You use XML DML within the context of a SQL SET statement, using either UPDATE...SET on an XML data type column, or by using SET on an XML variable or parameter. A simple example of the UPDATE...SET syntax would look like this:

-- change the value of XML data type column instance
UPDATE some_table
  SET xml_col::modify('some XML DML')
  WHERE id = 1

-- change the value of an XML variable
DECLARE @x XML
-- initialize it
SET @x = '<some>initial XML</some>'
-- now, mutate it
SET @x = @x::modify('some XML DML')
The xml::modify function supports the inserting, updating, and deleting of nodesets in an existing instance of an XML type. An example that inserts the InvoiceDate element as the first child of an Invoice element is shown in the following:
SET @x::modify('
  {-- XQuery program --}
  namespace inv="urn:www-develop-com:invoices"
  insert <inv:InvoiceDate>2002-06-15</inv:InvoiceDate>
                 as first
                 into /inv:Invoice ')

FOR XML and OpenXML Enhancements

The composition and decomposition functionality in SQL Server 2000 have been improved in Yukon. Although the major change to OpenXML is support of the XML data type (as an output column and through its related XML parsing stored procedure, sp_xml_preparedocument), the FOR XML syntax contains a number of exciting enhancements.

SQL Server 2000 provides an enhancement to T-SQL that permits composition of XML document fragments using SQL queries against relational tables. There are three dialects of FOR XML queries: FOR XML RAW, FOR XML AUTO, and FOR XML EXPLICIT. FOR XML RAW produces one XML element for each row in the result, no matter how many tables participate in the query. There is an attribute for each column, and the names of attributes reflect the column names or aliases. FOR XML RAW has been enhanced in Yukon to allow XML in element-normal form. FOR XML AUTO produces one XML element by row in the result, but produces nested XML elements if there is more than one table in the query. The order of nesting is defined by the order of the columns in the select statement. FOR XML EXPLICIT produces XML via SQL UNION queries. Each arm of the SQL UNION queries produces a different level of XML. This is by far the most flexible dialect and can produce element- or attribute-normal form and nested XML exactly as you like. This is also by far the most complex dialect to program.

SQL Server Yukon offers quite a few refinements and enhancements to FOR XML queries:

  • FOR XML can produce an instance of an XML type.
  • FOR XML AUTO and FOR XML RAW are able to preface the XML result with an inline schema in XSD schema format. The previous version of FOR XML could only preface an inline DTD or an XDR (XML Data Reduced) schema.
  • You can nest FOR XML queries.
  • You can produce element-centric XML using FOR XML RAW.
  • You can choose to generate xsi:nil for NULL database values rather than leaving that element out of the XML result entirely.
  • There is improved handling of white space by encoding some white space characters as entities.
  • There are subtle improvements to the algorithm for determining nesting in FOR XML AUTO.

The most interesting of these enhancements is that FOR XML can produce an instance of the XML data type. This is an XML fragment that can be wrapped in a root element and returned to the caller, queried, or mutated with the XML data type functions, and stored in an XML column. An example of using FOR XML queries to produce input to a table using the SELECT INTO syntax is shown in the following:

-- create a table
CREATE TABLE xml_tab(id INT IDENTITY PRIMARY KEY, xml_col XML)

-- populate it with a FOR XML query
DECLARE @x XML
SET @x = (SELECT * FROM pubs.dbo.authors FOR XML AUTO)
INSERT INTO xml_tab VALUES(@x)
GO

Native Web Service Support

Web services expose a standard mechanism for communication that uses standard protocols and a common message format. The network protocol most often used is HTTP, with SOAP employed as the message format. XML Web services can be produced and consumed by any platform that can produce XML-style text messages and send them across HTTP, making it a popular method of communication across unlike systems, and may displace proprietary protocols over time.

SQL Server 2000 allowed communication over HTTP by using IIS and an ISAPI DLL. This DLL allowed users to issue HTTP requests (subject to security checks, of course) to well-known endpoints exposed by XML-based files known as templates. SQLXML Web Release 3.0 expanded the capability of the ISAPI DLL to produce SOAP packets, allowing programmers to expose SQL Server through IIS as a Web service.

Any stored procedure, user-defined function, or template query can be exposed as a SOAP endpoint. Output is available in a variety of formats, some optimized for .NET based consumers, but all using the SOAP protocol. The ISAPI application was also expanded to produce Web Services Description Language (WSDL), a standardized dialect of XML that describes the format and location of a Web service. This allowed any program that could consume WSDL to know how to communicate with SQL Server.

If you've installed SQL Server Yukon on Windows Server™ 2003, you can get even better performance and functionality due to improvements in the HTTP handlers at the OS level, including the ability to serve HTTP from multiple applications on the same port (like SQL Server and IIS at the same time).

The new functionality is built directly into Yukon. Rather than storing configuration information in the registry or the IIS metabase, the information is stored as SQL Server metadata, which can be defined with T-SQL. The relevant DDL statements are CREATE HTTP ENDPOINT, ALTER HTTP ENDPOINT, and DELETE HTTP ENDPOINT. With HTTP ENDPOINT statements, you can define all of the properties of the HTTP support and define stored procedures and user-defined functions as Web methods. You can also allow ad hoc SQL queries using your endpoint. Figure 7 shows an example endpoint definition against the pubs database that permits ad hoc queries and defines the byroyalty stored procedure as a Web method.

Figure 7 Endpoint Definition (Close Figure 7)
CREATE HTTP ENDPOINT pubs_endpoint
AS
    SITE = '*',
    PATH = '/pubs/',
    AUTHENTICATION = (INTEGRATED),
    PORTS = (CLEAR),
    STATE = STARTED
FOR SOAP
(

  WEBMETHOD 'urn:www-develop-com:invoices'.'byroyalty'
  (
    name=pubs.dbo.byroyalty,
    SCHEMA = STANDARD
  )

  WITH
  BATCHES = ENABLED,  -- allow ad-hoc queries
  WSDL = ENABLED,
  DATABASE = 'pubs'
)
GO

Wrap-up

This article has provided a brief overview of the XML functionality enhancements found in the prerelease version of SQL Server Yukon. I haven't covered the client-side enhancements, only those that actually live in the SQL Server engine, and I haven't discussed the details of XQuery. To cover these would require another article or two, although SQL Workbench comes with a graphic XQuery tool to get you started. I hope that this article whets your appetite to try out the XML support in Yukon for yourself.

Bob Beauchemin is an instructor, course author, and database curriculum course liaison for DevelopMentor. He's written articles on ADO.NET, OLE DB and SQL Server. Bob is the author of Essential ADO.NET (Addison-Wesley, 2002) and coauthor of the upcoming book, A First Look at Microsoft SQL Server "Yukon" Beta for Developers, from which this article is excerpted.


T-SQL in Yukon: Powerful New T-SQL Syntax Gives SQL Server a Programmability Boost

T-SQL in Yukon

Powerful New T-SQL Syntax Gives SQL Server a Programmability Boost

Itzik Ben-Gan
This article was based on Beta 1 of Microsoft SQL Server Code Name "Yukon" and all information contained herein is subject to change.
Code download available at: TSQLinYukon.exe (117KB) Note: This document was developed prior to the product's release to manufacturing and, as such, we cannot guarantee that any details included herein will be exactly the same as those found in the shipping product. The information represents the product at the time this document was printed and should be used for planning purposes only. Information subject to change at any time without prior notice.
SUMMARY

The T-SQL language in the upcoming version of SQL Server will provide more power and flexibility than previous versions. Additions and enhancements include error handling with the TRY/CATCH construct, SNAPSHOT isolation, and WAITFOR enhancements. Also important are the BULK rowset provider, common table expressions, recursive queries, PIVOT and UNPIVOT operators, and much more. This article introduces these features so readers will be ready for the next version of SQL Server.

Contents
Error Handling
SNAPSHOT Isolation
The WAITFOR Command
BULK Rowset Provider
The TOP Option
The APPLY Operator
Common Table Expressions and Recursive Queries
The PIVOT Operator
Triggers and Notifications
Wrapping It Up

T
he Beta 1 release of the next version of SQL Server™ code-named "Yukon" introduces an array of T-SQL enhancements and new features that can improve your expressive powers, error management, and performance. In this article I will discuss some of the important enhancements and new features including error handling, recursive queries, and isolation. I will focus on error management and relational improvements and briefly describe the other enhancements.

In addition to the features I will describe here, there are many other important enhancements to T-SQL that I will not cover in depth because some are not new conceptually while others deserve independent coverage. These include messaging and the Service Broker platform, partitioning, and XML enhancements. (For more information about XML enhancements in Yukon, see "XML in Yukon: New Version Showcases Native XML Type and Advanced Data Handling" by Bob Beauchemin in this issue.

Error Handling

SQL Server Yukon Beta 1 introduces the new TRY/CATCH construct for error handling in T-SQL. This construct allows you to trap transaction abort errors, even those that would have caused a batch to terminate in previous versions of SQL Server (conversion errors, deadlocks, and so on). The types of errors that the new construct cannot handle are those that cause the termination of your session (usually errors with severity 21 and above, such as hardware errors). Typically, your error-handling code looks like Figure 1.

Figure 1 Error Handling Code (Close Figure 1)
SET XACT_ABORT ON

BEGIN TRY

  BEGIN TRAN
    -- your code goes here
  COMMIT TRAN

END TRY
BEGIN CATCH TRAN_ABORT

  DECLARE @err AS INT
  SET @err = @@error

  IF @err = <error id 1>
  BEGIN
    -- Handle error 1
  END
  ELSE IF @err = <error id 2>
  BEGIN
    -- Handle error 2
  END
  -- ... other errors ...
  ELSE
  BEGIN
    -- Handle unexpected error
  END

  IF @@trancount > 0 ROLLBACK

END CATCH

The XACT_ABORT setting is turned on so that SQL Server will treat any error as a transaction abort error, thus allowing it to be trapped and handled. Within the TRY block any error that occurs within an explicit transaction causes control to be passed to the CATCH block that immediately follows the TRY block. If no error occurs, the CATCH block is skipped. If you want to investigate the type of error that was raised and react accordingly, you must save the return value of @@error to a variable at the beginning of the CATCH block and then start investigating it. Otherwise the value returned by @@error may not be correct since any statement besides a DECLARE would change it.

When a transaction abort error occurs within a transaction located in the TRY block and control is passed to the CATCH block, the transaction enters a doomed state. Locks are not released, and persisted work is not reversed until you explicitly issue a ROLLBACK command. Until you issue a ROLLBACK, you are not allowed to initiate any activity that requires opening an implicit or explicit transaction. You can examine the content of resources that were changed in the transaction that caused the error, and by doing so you see the changes, but you have to issue a ROLLBACK in order to take any corrective measures that require a transaction to occur. Note that in order to trap errors that take place within the CATCH block, you must write your code within a nested TRY/CATCH construct. To see a more detailed example, first create an ErrorLog table in which the error-handling code audits comments, and then create the T1 and T2 tables against which queries will be issued, as I've done with the code in Figure 2.

Figure 2 ErrorLog Table (Close Figure 2)
CREATE TABLE ErrorLog
(
  lsn       INT          NOT NULL IDENTITY PRIMARY KEY,
  dt        DATETIME     NOT NULL DEFAULT(GETDATE()),
  errid     INT          NOT NULL,
  comments  VARCHAR(100) NOT NULL
)

CREATE TABLE T1(col1 INT NOT NULL PRIMARY KEY)
CREATE TABLE T2(col1 INT NOT NULL PRIMARY KEY)
INSERT INTO T1 VALUES(1)
INSERT INTO T2 VALUES(1)
GO

Next, run the script in Figure 3 (call it Script 1) in a new connection (call it Connection 1). Script 1 sets the lock timeout to 30 seconds and sets the deadlock priority to low, volunteering to be the deadlock victim in a deadlock scenario with a process that runs with a normal priority. The code in the TRY block updates T1, waits 10 seconds, then selects from T2. If the transaction completes with no errors, a row is inserted into the ErrorLog table with a comment saying that it completed successfully.

Figure 3 Script 1 (Close Figure 3)
SET XACT_ABORT ON
SET LOCK_TIMEOUT 30000
SET DEADLOCK_PRIORITY LOW

DECLARE
  @err        AS INT,
  @comments   AS VARCHAR(100),
  @retry      AS INT,
  @numretries AS INT

SELECT @retry = 0, @numretries = 2

RETRY:
BEGIN TRY

  BEGIN TRAN
    UPDATE T1 SET col1 = 2
    WAITFOR DELAY '00:00:10'
    SELECT * FROM T2
  COMMIT

  IF @@error = 0
    INSERT INTO ErrorLog(errid, comments)
      VALUES(0, 'Transaction completed successfully')

END TRY
BEGIN CATCH TRAN_ABORT

  SET @err = @@error

  IF @err = 2627
  BEGIN
    SELECT @retry = 0, @comments = 'PK violation'
  END
  ELSE IF @err = 1222
  BEGIN
    SELECT @retry = 0, @comments = 'Lock timeout expired'
  END
  ELSE IF @err = 1205
  BEGIN
    SELECT
      @retry = CASE WHEN @retry < @numretries THEN @retry + 1
                 ELSE 0 END,
      @comments = 'Deadlock;'
        + CASE WHEN @retry > 0
                 THEN ' attempting retry #' +
                   CAST(@retry AS VARCHAR(10))
               ELSE ' aborting'
          END
  END
  ELSE
  BEGIN
    SELECT @retry = 0, @comments = 'Unhandled error'
  END

  IF @@trancount > 0 ROLLBACK

  INSERT INTO ErrorLog(errid, comments)
    VALUES(@err, @comments)

  IF @retry > 0
  BEGIN
    WAITFOR DELAY '00:00:05'
    GOTO RETRY
  END

END CATCH

The CATCH block is designed to trap primary key violation errors, lock timeout expiration, and deadlock errors with retry logic. You can set the desired number of retries by changing the value assigned to the variable @retry at the beginning of the code; it is currently set to 2.

After running the code in Figure 3 for the first time, examine the contents of ErrorLog. Note that the transaction completed successfully. To test a primary key violation error, open a new connection (call it Connection 2) and run the following code:

INSERT INTO T1 VALUES(3)
Go back to Connection 1 and run Script 1 again. If you examine the contents of ErrorLog, you should see that a primary key violation error was logged. Go to Connection 2 and delete the row you just inserted by running this command:
DELETE FROM T1 WHERE col1 = 3

To test a lock expiration timeout, run the following in Connection 2:

BEGIN TRAN
  UPDATE T1 SET col1 = 1
Go back to Connection 1 and run Script 1 again. After about 30 seconds, you should get an error. Examine the contents of ErrorLog and notice that a lock timeout expiration was logged. Go to Connection 2 and issue a ROLLBACK command to roll back the transaction.

To test a deadlock, go to Connection 2 and paste in the following code, but don't run it yet:

DECLARE @i AS INT
BEGIN TRAN
  SET @i = 1
  WHILE @i <= 2
  BEGIN
    UPDATE T2 SET col1 = 2
    WAITFOR DELAY '00:00:10'
    SELECT * FROM T1
  WAITFOR DELAY '00:00:05'
  SET @i = @i + 1
END
ROLLBACK
Go to Connection 1, run the code in Script 1, and immediately afterward run the code in Connection 2. After about a minute you should see the error in Connection 1. Examine the contents of ErrorLog and notice that there were two retry attempts after deadlock errors and a third successful attempt that raised no error. Query the ErrorLog table and examine its contents.

Finally, if you want to raise your own transaction abort error within the TRY block, you can invoke the RAISERROR command with the TRAN_ABORT option.

SNAPSHOT Isolation

Yukon introduces a new isolation level called SNAPSHOT which allows you to work in a mode where writers do not block readers, providing readers with a committed version of the data they request. SQL Server Yukon maintains a linked list in tempdb that tracks changes to rows and constructs an older, committed version of data for readers. This isolation is useful for optimistic locking, where UPDATE conflicts are not common. If Process 1 retrieves data and later attempts to modify it, and if Process 2 has modified the same data between the retrieval and modification, then SQL Server produces an error upon Process 1's modification attempt because of the conflict. Then, Process 1 can try to reissue the transaction. This mode can be efficient in situations where update conflicts are not common.

In order to allow working in a SNAPSHOT isolation level, you must turn on the database option ALLOW_SNAPSHOT_ISOLATION, as you'll see shortly. To simulate a scenario in which writers do not block readers, create the testdb database, turn on the appropriate database option, and create the T1 table with the value "Version1" in the datacol by running the following code:

CREATE DATABASE testdb
GO
USE testdb
ALTER DATABASE testdb SET ALLOW_SNAPSHOT_ISOLATION ON

CREATE TABLE T1
(
  keycol  INT         NOT NULL PRIMARY KEY,
  datacol VARCHAR(10) NOT NULL
)
INSERT INTO T1 VALUES(1, 'Version1')
Issue the following code from Connection 1, which opens a transaction and changes the value in the datacol to "Version2":
USE testdb
BEGIN TRAN
  UPDATE T1 SET datacol = 'Version2' WHERE keycol = 1
  SELECT * FROM T1
Go to Connection 2 and run the following code, which sets the session's isolation to SNAPSHOT and retrieves the contents of T1:
USE testdb
SET TRANSACTION ISOLATION LEVEL SNAPSHOT
SELECT * FROM T1
Notice that you get "Version1" back, even though Connection 2 changed it to "Version2" (but had not committed that change yet).

Now go to Connection 1 and commit the transaction and close all connections with a COMMIT command. To try optimistic locking, open two new connections, go to Connection 1, and run the following code, which sets the session's isolation to SNAPSHOT, opens a transaction, and retrieves data from T1:

SET TRANSACTION ISOLATION LEVEL SNAPSHOT
BEGIN TRAN
  SELECT * FROM T1
Go to Connection 2 and issue an UPDATE:
UPDATE T1 SET datacol = 'Version3' WHERE keycol = 1
Go back to Connection 1 and try to update the same data you previously retrieved, which was modified by Connection 2:
UPDATE T1 SET datacol = 'Version4' WHERE keycol = 1
You should get an error notifying you that SQL Server cannot use snapshot isolation to access table T1 in database testdb and that you should retry your transaction.

The WAITFOR Command

The WAITFOR command has been enhanced in several ways in Yukon. Besides waiting for a specified duration or until a certain datetime value, you can now request to wait for a T-SQL statement to affect at least one row. You can specify that the command wait on one of the following statements: SELECT, INSERT, UPDATE, DELETE, or RECEIVE. The first four are self-explanatory; RECEIVE refers to receiving a message from a queue. You can optionally specify a timeout if you want to stop waiting after your specified number of milliseconds. The syntax of the WAITFOR command is:

WAITFOR(<statement>) [,TIMEOUT <timeout_value>]

Another T-SQL enhancement in Yukon allows you to return output from Data Manipulation Language (DML) statements other than SELECT (INSERT, UPDATE, DELETE). A new OUTPUT clause allows you to request that the old/new images of the columns be returned by referring to the INSERTED and DELETED tables, similar to the way you refer to them in triggers. You can even specify an INTO clause and direct the output to a table variable. Another enhancement lets you specify the READPAST hint with modifying statements, allowing you to skip locked rows.

An example of using the aforementioned enhancements is to have several processes waiting for a DELETE statement to delete at least one row from a table, directing the output to a table variable, with each processing different portions of the data in parallel. To witness this, create the following MsgQueue table:

USE tempdb
CREATE TABLE MsgQueue
(
  msgid   INT         NOT NULL IDENTITY PRIMARY KEY,
  msgdata VARCHAR(15) NOT NULL
)
Open one or more connections and run the following code in each to periodically insert new messages into the table:
SET NOCOUNT ON
USE tempdb

WHILE 1 = 1
BEGIN
  INSERT INTO MsgQueue VALUES('Msg' +
    CAST(CAST(RAND()*1000000000 AS INT) AS VARCHAR(10)))
  WAITFOR DELAY '00:00:01'
END
Next you need to open several other new connections and run the code in Figure 4 in each connection to simulate processing of the newly arriving messages.

Figure 4 Processing Queued Messages (Close Figure 4)
SET NOCOUNT ON
USE tempdb

DECLARE @MyQueue TABLE
(
  msgid   INT         NOT NULL PRIMARY KEY,
  msgdata VARCHAR(15) NOT NULL
)

WHILE 1 = 1
BEGIN
  BEGIN TRAN
    WAITFOR
    (
      DELETE FROM MsgQueue WITH (READPAST)
      OUTPUT DELETED.* INTO @MyQueue
    )

    PRINT 'Processing messages ...'
    SELECT * FROM @MyQueue
    -- Message processing goes here

    DELETE FROM @MyQueue
  COMMIT TRAN
END

BULK Rowset Provider

Yukon introduces the new BULK rowset provider, which you can specify in the OPENROWSET function to efficiently access files in a relational fashion. You can use the BULK provider in a manner similar to the way you use a BULK INSERT statement, but you don't have to send the output to a table. You do have to specify a format file, which is the same format file you use with bcp.exe or the BULK INSERT statement. The following code shows how to access a file called c:\temp\textfile1.txt using the format file c:\temp\textfile1.fmt, providing the alias C to the result table, and the aliases col1, col2, and col3 to the result columns:

SELECT col1, col2, col3
FROM OPENROWSET(BULK 'c:\temp\textfile1.txt',
       FORMATFILE = 'c:\temp\textfile1.fmt') AS C(col1, col2, col3)

In addition to the FORMATFILE option, you can also specify the following options within the parentheses of the OPENROWSET function: CODEPAGE, DATAFILETYPE, FIELDTERMINATOR, FIRSTROW, LASTROW, and ROWTERMINATOR. You can also use an INSERT SELECT to load the data into a table efficiently and can specify table hints for load options:

INSERT INTO MyTable WITH (BULK_CHECK_CONSTRAINTS)
  SELECT col1, col2, col3
  FROM OPENROWSET(BULK 'c:\temp\textfile1.txt',
         FORMATFILE = 'c:\temp\textfile1.fmt') AS C(col1, col2, col3)
Other options you can specify as a table hint include: BULK_BATCHSIZE, BULK_FIRE_TRIGGERS, BULK_KEEPIDENTITY, BULK_KEEPNULLS, BULK_KILOBYTES_PER_BATCH, BULK_MAXERRORS, and ROWS_PER_BATCH.

Using the BULK rowset provider you can load a file into a table's column more easily than ever before using regular DML. Now, instead of being limited to the TEXT, NTEXT, and IMAGE datatypes for large objects, you can also use the VARCHAR(MAX), NVARCHAR(MAX), and VARBINARY(MAX) datatypes. The new MAX option allows you to manipulate large objects the same way you manipulate regular datatypes. For example, the following UPDATE statement stores a text file in a table column that is defined as VARCHAR(MAX):

UPDATE LOBs
  SET clob_col =  (SELECT clob_data
                   FROM OPENROWSET(BULK 'c:\temp\textfile1.txt',
                          SINGLE_CLOB) AS C(clob_data))
WHERE keycol = 1
The SINGLE_NCLOB option tells SQL Server that the large object is in character format. Similarly, SINGLE_CLOB specifies a large object with regular character format, and SINGLE_BLOB specifies a binary format. The name of the returned column is BulkColumn, but as the previous code snippet shows, you can specify your own alias for this.

The TOP Option

The T-SQL TOP option in Yukon has two significant enhancements. You can now specify as an argument to TOP an expression that can contain variables or even a self-contained query. You can also use the TOP option with modifying DML (INSERT, UPDATE, DELETE).

To specify an expression, you must enclose it in parentheses. The expression should be of the BIGINT datatype when not using the PERCENT option, and a float value in the range 0 through 100 when using the PERCENT option. The following code shows how to use an expression with a variable to return the requested number of earliest orders from the SalesOrderHeader in the AdventureWorks database:

USE AdventureWorks

DECLARE @n AS BIGINT
SET @n = 5

SELECT TOP (@n) *
FROM SalesOrderHeader AS SOH
ORDER BY OrderDate, SalesOrderID

The SalesOrderID is used as a tiebreaker. Similarly, the following example demonstrates how to use the PERCENT option to return the requested percentage of earliest orders:

DECLARE @p AS FLOAT
SET @p = 0.01

SELECT TOP (@p) PERCENT *
FROM SalesOrderHeader AS SOH
ORDER BY OrderDate, SalesOrderID
You should get 4 rows as a result since the SalesOrderHeader table contains 31,519 rows, and 31,519 x .0001 rounded up equals 4.

The main motivation for allowing TOP with modifying DML was to substitute the SET ROWCOUNT option, which SQL Server could not optimize very well. The SET ROWCOUNT option was often used with modifications to process large numbers of rows in batches, preventing the transaction log from exploding and keeping the individual locks from escalating to a full table lock. To see how to delete rows in batches using the new TOP functionality, first copy the contents of the SalesOrderHeader table to MySalesOrderHeader, and create an index on the OrderDate, and SalesOrderID columns by running the following code:

SELECT *
INTO MySalesOrderHeader
FROM SalesOrderHeader

CREATE UNIQUE CLUSTERED INDEX idx_uc_OrderDate_SalesOrderID
  ON MySalesOrderHeader(OrderDate, SalesOrderID)
To delete all rows with an order year earlier than 2003 in batches of 1,000, use the following code:
WHILE 1 = 1
BEGIN
  DELETE TOP (1000)
  FROM MySalesOrderHeader WHERE OrderDate < '20030101'

  IF @@rowcount < 1000 BREAK
END
SQL Server optimizes such code much more efficiently than it can with the SET ROWCOUNT option. You can now drop the MySalesOrderHeader table:
DROP TABLE MySalesOrderHeader

The APPLY Operator

APPLY is a new relation operator that you specify in the FROM clause of a query. It allows you to invoke a table-valued function for each row of an outer table, optionally using the outer table's columns as the function's arguments. The APPLY operator comes in two forms: CROSS APPLY and OUTER APPLY. The former doesn't return the outer table's row if the table-valued function returns an empty set for it, while the latter returns a row with NULLs instead of the function's columns. To use the APPLY operator, first create the following Arrays table, which stores arrays of comma-separated values:

CREATE TABLE Arrays
(
  arrid INT NOT NULL IDENTITY PRIMARY KEY,
  array VARCHAR(7999) NOT NULL
)

INSERT INTO Arrays VALUES('')
INSERT INTO Arrays VALUES('10')
INSERT INTO Arrays VALUES('20,40,30')
INSERT INTO Arrays VALUES('-1,-3,-5')
Next, create the fn_splitarr table-valued function, which accepts an array as an argument and returns a table containing the individual elements and their positions (see Figure 5). To test the function, run this code:
SELECT * FROM fn_splitarr('20,40,30')
The output should look like the following lines:
pos         value
---         -----
1           20
2           40
3           30
Now use the CROSS APPLY operator to invoke the function for each row in Arrays:
SELECT A.arrid, F.*
FROM Arrays AS A
  CROSS APPLY fn_splitarr(array) AS F
Then check the values in your output against these:
arrid       pos         value
-----       ---         -----
2           1           10
3           1           20
3           2           40
3           3           30
4           1           -1
4           2           -3
4           3           -5
Note that the row with arrid 1 from Arrays was not returned, as the function returned an empty set for it. To return all rows from Arrays regardless of whether the function returns rows for them or not, use OUTER APPLY.

Figure 5 fn_splitarr Table-valued Function (Close Figure 5)
CREATE FUNCTION fn_splitarr(@arr AS VARCHAR(7999))
  RETURNS @t TABLE(pos INT NOT NULL, value INT NOT NULL)
AS
BEGIN
  DECLARE @end AS INT, @start AS INT, @pos AS INT

  SELECT @arr = @arr + ',', @pos = 1,
    @start = 1, @end = CHARINDEX(',', @arr, @start)

  WHILE @end > 1
  BEGIN
    INSERT INTO @t VALUES(@pos, SUBSTRING(@arr, @start, @end - @start))

    SELECT @pos = @pos + 1,
      @start = @end + 1, @end = CHARINDEX(',', @arr, @start)
  END

  RETURN
END
GO

A bonus feature of support for the APPLY operator is that you can now refer to table-valued functions and specify the outer table's columns as arguments in your subqueries. For example, the following code returns all of the Arrays whose sum of its elements is less than or equal to 10:

SELECT *
FROM Arrays
WHERE (SELECT SUM(value) FROM fn_splitarr(array)) <= 10

Common Table Expressions and Recursive Queries

Common table expressions (CTEs) allow you to write named table expressions that persist for only the duration of a query. In their simple form, they provide a mixed functionality of views and derived tables. Like views, the CTE can be referred to more than once in the outer query, and like a derived table, it persists for only the duration of the query. In their more complex form, you can write recursive CTEs, allowing you to manipulate trees and graphs more easily and efficiently.

You define a CTE using a WITH clause followed by the CTE's name and optionally provide a list of result column aliases in parentheses. You follow with the AS clause and parentheses containing the CTE's query expression. You finish by providing an outer query which refers to the CTE's result. Within the CTE's query expression, you can refer to variables if you like.

The code in Figure 6 shows a simple example for writing a non-recursive CTE that returns yearly customer's sales order values. Obviously you could achieve the same results without using a CTE, but suppose with each row you also want to return the previous year's total value, and the difference from the current year. If you choose to use derived tables, you have to specify the current year's query in one derived table, and the previous year's query in another, and have the outer query join the two. With CTEs you can write a single query returning yearly totals, and have the outer query refer to it twice (see Figure 7).

Figure 7 Single Query Returning Yearly Totals (Close Figure 7)
DECLARE @CustomerID AS INT
SET @CustomerID = 3

WITH YearlyCustOrders(OrderYear, TotalValue)
AS
(
  SELECT YEAR(OrderDate), SUM(OrderQty*UnitPrice)
  FROM SalesOrderHeader AS H
    JOIN SalesOrderDetail AS D
      ON H.SalesOrderID = D.SalesOrderID
  WHERE CustomerID = @CustomerID
  GROUP BY YEAR(OrderDate)
)
SELECT Cur.*, Prev.TotalValue AS PrevTotalValue,
  (Cur.TotalValue/Prev.TotalValue - 1) * 100 AS Diff
FROM YearlyCustOrders AS Cur
  LEFT OUTER JOIN YearlyCustOrders AS Prev
    ON Cur.OrderYear = Prev.OrderYear + 1;
Figure 6 A Non-recursive CTE (Close Figure 6)
USE AdventureWorks

DECLARE @CustomerID AS INT
SET @CustomerID = 3

WITH YearlyCustOrders(OrderYear, TotalValue)
AS
(
  SELECT YEAR(OrderDate), SUM(OrderQty*UnitPrice)
  FROM SalesOrderHeader AS H
    JOIN SalesOrderDetail AS D
      ON H.SalesOrderID = D.SalesOrderID
  WHERE CustomerID = @CustomerID
  GROUP BY YEAR(OrderDate)
)
SELECT * FROM YearlyCustOrders;

The true power of CTEs, though, is in their recursive form. Within the CTE's parentheses you define queries that are either independent or refer back to the CTE. The independent queries (those without a reference to the CTE's name) are called anchor members and are invoked only once. The queries that refer back to the CTE's name are called recursive members and are invoked repeatedly until the query returns no rows. Anchor members can be appended to each other using either a UNION or a UNION ALL operator, depending on whether you want to eliminate duplicates or not. You must append recursive members using a UNION ALL operator.

As an exmaple of a scenario in which recursive CTEs are useful, consider the BillOfMaterials table in the AdventureWorks database. This table represents a typical bill of materials scenario with product assemblies that form an acyclic directed graph. Each product is assembled from other products, which might in turn be assembled from other products as well, and there are no cyclic relationships. The assembly product-contained product relationship is expressed by the AssemblyID and ComponentID columns. The PerAssemblyQty contains the quantity of the product expressed in ComponentID for each individual product expressed in AssemblyID. Relationships that become obsolete have a date specified in the ObsoleteDate column. If you're interested only in non-obsolete data, you should test this column for NULLs. There is other useful information in the table including the unit measure, but for the sake of this illustration I'll ignore all other columns.

The code in Figure 8 yields the data for an exploded view of ProductID 210. Figure 9 shows an excerpt of this view; and depicts the containment relationships between the products. Within the CTE's body, the first query doesn't have a reference to the CTE's name, so it is an anchor member and as such is invoked only once. Notice that the query looks for a row where the component ID is 210, with an assembly ID of NULL, meaning that it's a top-level product. The query makes sure that the relationship is not obsolete, and returns the component ID and quantity. The recursive member returns the products contained within the assemblies returned from the previous step by issuing a join between the CTE's name and the BillOfMaterials table. The first time the recursive member is invoked, the previous step is the result returned by the anchor member. The second time it is invoked, the previous step is the result returned by the first invocation of the recursive member, and so on until the recursive member returns an empty set.

Figure 9 Exploded View of ProductID 210 (Close Figure 9)

Figure 8 Details on ProductID 210 (Close Figure 8)
WITH BOMCTE(ProductID, Qty)
AS
(
  SELECT ComponentID, PerAssemblyQty
  FROM BillOfMaterials
  WHERE ComponentID = 210
    AND AssemblyID IS NULL
    AND ObsoleteDate IS NULL

  UNION ALL

  SELECT BOM.ComponentID, BOMCTE.Qty * BOM.PerAssemblyQty
  FROM BOMCTE
    JOIN BillOfMaterials AS BOM
      ON BOM.ObsoleteDate IS NULL
      AND BOM.AssemblyID = BOMCTE.ProductID
)
SELECT B.ProductID, P.Name, B.Qty
FROM BOMCTE AS B
  JOIN Product AS P
    ON P.ProductID = B.ProductID
ORDER BY P.ProductID;

The recursive member calculates the cumulative quantity of the component by multiplying the previous step's quantity by the component's quantity. The reference of the outer query to the CTE's name gets the unified results of all of the invocations of the anchor and recursive members. The outer query joins the CTE to the Products table to get the product names and produces the 90 rows in Figure 10 (abbreviated). Each component can appear more than once in the output, for example product 835, since it might participate in different assemblies. You can revise the outer query to group the results by product ID and name to get the total quantities per product. The code looks like Figure 8, but the outer query will look like this:

SELECT B.ProductID, P.Name,
   SUM(B.Qty) AS TotalQty
FROM BOMCTE AS B
  JOIN Product AS P
    ON P.ProductID = B.ProductID
GROUP BY B.ProductID, P.Name
ORDER BY B.ProductID;

Figure 10 CTE Joined to the Products Table (Close Figure 10)
ProductID   Name                            Qty
---------- ------------------------------ ------------------
1           Adjustable Race                 1
2           Bearing Ball                    100
3           LL Bottom Bracket               1
...
835         Metal Sheet 5                   2
835         Metal Sheet 5                   1
835         Metal Sheet 5                   2
...

(90 row(s) affected)

If you suspect cycles and want to limit the number of recursive invocations, you can specify the MAXRECURSION option right after the outer query:

WITH...
outer_query
OPTION(MAXRECURSION 30)
This option causes SQL Server to raise an error when the CTE exceeds the specified limit. The default in SQL Server is 100 when not specifying this option. If you don't want to have a limit, you must specify 0. Note that you can write custom code that detects cyclic relationships, but that is beyond the scope of this article.

The PIVOT Operator

The new PIVOT operator in SQL Server Yukon allows you to write crosstab queries that rotate rows into columns. The UNPIVOT operator does the opposite—working on pivoted data to rotate columns into rows. Figure 11 shows the result of using the PIVOT operator in the database when you want to return yearly total sales order values for each salesperson, with each year's value appearing in a different column.

Figure 11 Result of a PIVOT (Close Figure 11)
SalesPersonID 2001            2002            2003          2004
------------ -------------- -------------------- --------------
20            NULL            NULL            197303.2486   31680.0348
33            569966.4019     1475303.3042    1693905.5061  685729.9303
29            1665446.9384    3104537.8510    2720479.1674  1311261.2956
•••

(18 row(s) affected)

The important thing to note when using the PIVOT operator is that you need to provide it with a query expression that returns only the columns of interest by using a view, derived table, or a CTE. The reason is that PIVOT makes an implicit GROUP BY operation behind the scenes on all columns that are not explicitly referenced by the operator. In this case, you need the salesperson ID, the order year, and the order value:

USE AdventureWorks

SELECT
  SOH.SalesPersonID,
  YEAR(SOH.OrderDate) AS OrderYear,
  SOD.OrderQty * SOD.UnitPrice AS OrderValue
FROM SalesOrderHeader AS SOH
  JOIN SalesOrderDetail AS SOD
    ON SOD.SalesOrderID = SOH.SalesOrderID

SQL Server figures out that the "GROUP BY" column list should be the list of columns in the input table that were not referred to explicitly by the PIVOT operator in the aggregate function or in the IN clause. So that you don't get undesired columns in the implicit GROUP BY column list, you need to provide the PIVOT operator with an input table containing only the columns of interest for the aggregate function, IN clause, and implicit GROUP BY. This can be achieved by using a CTE or a derived table containing the previous query that returns only the columns of interest.

The code in Figure 12 shows how you use this query within a CTE, and have the outer query issue a PIVOT operation on the CTE's results. SUM(OrderValue) tells PIVOT which aggregation to calculate to populate the cells of the pivoted column. The FOR clause tells PIVOT which source column contains the values to rotate into result columns. The IN clause contains the list of values that are going to appear as result column names.

Figure 12 PIVOT on CTE (Close Figure 12)
WITH OrdersCTE(SalesPersonID, OrderYear, OrderValue)
AS
(
  SELECT
    SOH.SalesPersonID,
    YEAR(SOH.OrderDate),
    SOD.OrderQty * SOD.UnitPrice
  FROM SalesOrderHeader AS SOH
    JOIN SalesOrderDetail AS SOD
      ON SOD.SalesOrderID = SOH.SalesOrderID
)
SELECT *
FROM OrdersCTE
  PIVOT(SUM(OrderValue)
    FOR OrderYear IN([2001], [2002], [2003], [2004])) AS P

SQL Server requires you to specify explicitly the list of values in the IN clause to rotate to result columns. You can't use a static query and have SQL Server figure out all distinct values in OrderYear. To achieve this, you have to use dynamic execution to construct the query string dynamically, as the code in Figure 13 shows.

Figure 13 Constructing a Dynamic Query String (Close Figure 13)
CREATE INDEX idx_nc_OrderDate ON SalesOrderHeader(OrderDate)

DECLARE @sql AS NVARCHAR(4000), @dt AS DATETIME

SET @sql =
  N'WITH OrdersCTE(SalesPersonID, OrderYear, OrderValue)
AS
(
  SELECT
    SOH.SalesPersonID,
    YEAR(SOH.OrderDate),
    SOD.OrderQty * SOD.UnitPrice
  FROM SalesOrderHeader AS SOH
    JOIN SalesOrderDetail AS SOD
      ON SOD.SalesOrderID = SOH.SalesOrderID
)
SELECT *
FROM OrdersCTE
  PIVOT(SUM(OrderValue) FOR OrderYear IN('

SET @dt =
  (SELECT CONVERT(CHAR(4), YEAR(MIN(OrderDate))) + '0101'
   FROM SalesOrderHeader)

WHILE @dt IS NOT NULL
BEGIN
  SET @sql = @sql + N'[' + CAST(YEAR(@dt) AS NCHAR(4)) + N']'

  SET @dt =
    (SELECT CAST(YEAR(MIN(OrderDate)) AS CHAR(4)) + '0101'
     FROM SalesOrderHeader
     WHERE OrderDate >= DATEADD(year, 1, @dt))

  IF @dt IS NOT NULL SET @sql = @sql + N','
END

SET @sql = @sql + N')) AS P'

EXEC sp_executesql @sql

To see the UNPIVOT operator in action, first create the SalesPivoted table by running the query in Figure 12 with the addition of "SELECT INTO SalesPivoted" before the FROM clause (see Figure 14). The UNPIVOT operator's arguments are very similar to PIVOT's arguments. But this time you specify a name of a result column that will contain the pivoted cell's values all in one column. Following the FOR clause, you specify the name of the result column that will store the names of the pivoted columns as column values. In the parentheses following the IN clause, specify the list of pivoted columns you want to unpivot:

SELECT *
FROM SalesPivoted
  UNPIVOT(OrderValue
    FOR OrderYear IN([2001], [2002], [2003], [2004])) AS U

Figure 14 Setting up for UNPIVOT (Close Figure 14)
WITH OrdersCTE(SalesPersonID, OrderYear, OrderValue)
AS
(
  SELECT
    SOH.SalesPersonID,
    YEAR(SOH.OrderDate),
    SOD.OrderQty * SOD.UnitPrice
  FROM SalesOrderHeader AS SOH
    JOIN SalesOrderDetail AS SOD
      ON SOD.SalesOrderID = SOH.SalesOrderID
)
SELECT *
INTO SalesPivoted
FROM OrdersCTE
  PIVOT(SUM(OrderValue) FOR OrderYear
    IN([2001], [2002], [2003], [2004])) AS P

UNPIVOT does not return rows for cells that contained NULL values. To clean up the extra table and index I've built in the database, run the following code:

DROP INDEX SalesOrderHeader.idx_nc_OrderDate
DROP TABLE SalesPivoted

Triggers and Notifications

SQL Server Yukon Beta 1 introduces support for Data Definition Language (DDL) triggers, allowing you to trap DDL operations and react to them, optionally rolling back the activity. DDL triggers work synchronously, right after the triggering event, similar to the way triggers from previous versions of SQL Server worked. SQL Server also supports an asynchronous event consumption mechanism using notifications, which lets you subscribe to be notified when certain events occur.

The following trigger is created at the database level and captures DROP TABLE attempts:

CREATE TRIGGER prevent_drop_table ON DATABASE FOR DROP_TABLE
AS
RAISERROR('Not allowed to drop tables.', 10, 1)
ROLLBACK

-- For debug
PRINT 'DROP TABLE attempt in database ' + DB_NAME() + '.'
PRINT EventData()
GO
You can define the trigger to fire for specific DDL events such as CREATE_TABLE, DROP_TABLE, ALTER_TABLE, CREATE_VIEW, and so on, or you can specify DDL_DATABASE_LEVEL_EVENTS if you want the trigger to fire for all DDL events in the database. Within the trigger you can invoke the EventData function to return information about the process and activity that fired the trigger. You can investigate the XML returned by the function and react accordingly.

To test the trigger, first create the table TestDrop and insert a row into it by running the following code:

CREATE TABLE TestDROP(col1 INT)
INSERT INTO TestDROP VALUES(1)
Next, attempt to drop the table:
DROP TABLE TestDROP
The DROP attempt is trapped and a message is printed indicating that it is not allowed to drop the table. Also, the return value of the EventData function is printed in XML format for debug purposes. (In practice, within the trigger you can examine the XML data, which contains a lot of useful information, in order to determine which course of action best suits your needs. For example, you can prevent dropping certain tables at specific times of the day.) The trigger rolled back the activity, so the table is not dropped from the database. To drop the trigger, issue the following code statement:
DROP TRIGGER prevent_drop_table ON DATABASE

You can also create a trigger that captures server-level events. For example, the following trigger captures login manipulation events such as creating, altering, or dropping a login:

CREATE TRIGGER audit_ddl_logins ON ALL SERVER
  FOR CREATE_LOGIN, ALTER_LOGIN, DROP_LOGIN
AS
PRINT 'DDL LOGIN took place.'
PRINT EventData()
GO
In this case, the trigger merely prints a notice that the event took place along with details of the event. But, of course, you can investigate the event details and react accordingly. To test the trigger, run the following code and then take a look at the result:
CREATE LOGIN login1 WITH PASSWORD = '123'
ALTER LOGIN login1 WITH PASSWORD = 'xyz'
DROP LOGIN login1
A DDL login event was identified and the event data was produced in XML format. You could examine the event data and audit the information that is important to you, if you'd like.

If you want to drop the trigger, run the following code:

DROP TRIGGER audit_ddl_logins ON ALL SERVER

Wrapping It Up

The T-SQL enhancements and new features provided with Yukon allow you to manipulate data more efficiently, develop applications more easily, and increase your error handling capabilities. When dealing with data manipulation, T-SQL is still the best development choice in SQL Server, and now you have a richer environment for your development. To make it easier for you to experiment with this new feature set, all of the samples described in this article are available for download from the link at the top of this article.

For background information see:
SQL Server
http://www.sqlmag.com

Itzik Ben-Gan is a Principal Mentor with Solid Quality Learning. He teaches, lectures, writes, and consults about SQL Server internationally. Itzik coauthored the book Advanced Transact-SQL for SQL Server 2000 (APress, 2000). He manages the Israeli SQL Server Users Group and is a SQL Server MVP. Contact him at itzik@solidqualitylearning.com.


Types in Yukon: Managed UDTs Let You Extend the SQL Server Type System

[ Editor's Update - 11/24/2004: Some of the TSQL syntax used in this article has changed since Beta 1 of SQL Server 2005. Please see the SQL Server Developer Center for general information on this feature in SQL Server 2005 Beta 2 and subsequent releases.]
Types in Yukon

Managed UDTs Let You Extend the SQL Server Type System

Peter W. DeBetta
This article was based on Beta 1 of Microsoft SQL Server Code Name "Yukon" and all information contained herein is subject to change.
Code download available at: UDTsinYukon.exe (112KB) Note: This document was developed prior to the product's release to manufacturing and, as such, we cannot guarantee that any details included herein will be exactly the same as those found in the shipping product. The information represents the product at the time this document was printed and should be used for planning purposes only. Information subject to change at any time without prior notice.
SUMMARY

The next version of SQL Server, code-named "Yukon," will offer enhanced support for user-defined types (UDTs). Because UDTs can be managed by the CLR, you can represent a wide variety of data structures to create types not possible with previous versions of SQL Server. With UDTs you can more precisely control the kind of data your type accepts, resulting in better data management. This article explores UDTs in "Yukon" and covers their design and implementation.

Contents
Opaque or Abstract Types
Creating the Assembly
Class Attributes
Comparing UDT Values
Nullability
Supporting Conversion
Other Requirements
Using a UDT in the SQL Server Create Assembly
Create Type
Using a UDT
Dropping and Altering UDTs and Assemblies
Conclusion

U
ser-defined types (UDTs) are one of the new common language runtime (CLR) integration features in the next version of SQL Server™, code-named "Yukon." UDTs in Yukon represent a significant advance from previous versions of SQL Server. SQL Server 2000, for example, supports alias types, which provide a simple way for users to redefine native types. Alias types are created using an existing data type that was defined the same way in multiple locations (tables, procedures, and so on). For example, a ZIP code could be used in multiple tables and as a parameter in multiple stored procedures. You could create a type named ZIP in the following manner:
EXEC sp_addtype ZIP, 'CHAR(5)', 'NOT NULL'

You can then use ZIP instead of char(5) NOT NULL wherever a ZIP code was needed and not have to worry about making it consistent with other fields in other tables that also store ZIP codes. Note that ZIP has an additional constraint on it—NOT NULL. This forces the field to require a value, so wherever this type is used, it will not allow null values. You can create rules and default objects and bind them to the alias type, further enforcing and maintaining data integrity. For example, you can create a rule that only allows numeric values and then bind it to the ZIP type. This technique has some great design and implementation benefits, such as more organized and consistent data structures that will result in more consistent data.

The sp_addtype system stored procedure is being replaced in Yukon by the new Data Description Language (DDL) syntax CREATE TYPE. The example I showed earlier might be written to look something like this:

CREATE TYPE ZIP FROM char(5) NOT NULL
Yukon UDTs allow users to write Microsoft® .NET Framework classes that can be registered to act as scalar types within the SQL language-type system. This puts Yukon UDTs on the same footing as any other SQL native type. Yukon UDTs are compiled into .NET assemblies that can be registered and stored inside the database. Once the assembly is stored in the database, a user can define a class in that assembly as a type using an extension of the CREATE TYPE statement, which I'll cover in more detail later in this article. For a .NET Framework class to act as a type in the SQL type system, it must implement a contract—a set of interfaces and methods—which is verified by Yukon at CREATE TYPE time. Defining and implementing UDTs defined in .NET assemblies allows for more flexible type design that can not only store structured types, but that can also allow custom methods, properties, and serializations on the data.

There are several benefits to this functionality. It's a powerful way to extend the scalar type system of the database. This extensibility mechanism is pretty robust; you can use it to store instances of the type in the database and operate on them in many contexts—as variables, parameters to stored procedures, and values to functions—and use them in almost all the places in which you can use a native type, including in advanced scenarios such as replication, bulk copy, distributed queries, and cross-database operations. In addition, you can invoke behaviors defined on the type as part of your query and even index the results of calling such behaviors to speed up query execution.

Another benefit of UDTs is encapsulation. The state for the type and the operations on the type are always available as a single unit. Methods on the type control access to the state, which frees you to think about your types using familiar OO paradigms and produce reusable components that can be consumed by other application developers. Later in this article I'll discuss the various aspects of implementing UDTs in .NET code and using the resulting classes as data types in Yukon.

Opaque or Abstract Types

Yukon UDTs can be regarded as opaque or abstract data types since users of the type remain unaware of the internal details of the type's implementation. They interact with the type only through its public interface. This new technique of defining UDTs opens new doors in database design. Since UDTs can be managed types with properties, methods, and so on, you can now create types to represent a myriad of data structures previously not possible with SQL Server. Using UDTs, you can create data structures to represent geospatial (location and plat type data), custom binary data, encoded data, and encrypted data.

You can extend the SQL type system by creating a class in managed code that implements the contract for UDTs. You then load the compiled assembly that contains your UDT into a database on the server using the CREATE ASSEMBLY statement and create a type in the database using the CREATE TYPE statement that exposes the managed code UDT.

At this point, you can use the type in a table definition or in variable declarations. Let's start out by examining the managed code requirements for a UDT.

Creating the Assembly

In order to create an assembly in .NET that will be usable in Yukon, there are a few requirements. I will not cover the basic aspects of creating a .NET assembly here. For more information on that, see one of the Microsoft .NET tutorials.

You will be utilizing several namespaces used in UDT development. In order to make your development efforts easier, add directives for the following:

using System.Data.Sql;
using System.Data.SqlTypes;
The System.Data.SqlTypes namespace contains classes representing the SQL Server native data types that can be used in your assembly. The System.Data.Sql namespace contains the objects needed for various attributes required in an assembly that will be used as a UDT. All of these types live in System.Data.dll.

Let's look at CLR specifics for creating an assembly that is used by Yukon. In the following sections, I will examine various implementation requirements, some of which will be familiar to you if you have been programming in .NET, and some of which are particular to the new namespaces I just mentioned.

Class Attributes

For Beta 1, two attributes must be specified in order for a class to be used as a UDT: Serializable and SqlUserDefinedType (shown in Figure 1). The Serializable attribute enables the class's data to be serialized by the CLR into a storable format, while SqlUserDefinedType defines UDT features such as the serialization format and the maximum size of the stored structure. The SqlUserDefinedType attribute has four properties (see Figure 2), of which only one is required. In future betas, marking the UDT class as Serializable will no longer be required.

Figure 2 The SQLUserDefinedType Attribute (Close Figure 2)

Property | Description
Format.<SerializationFormat> | The only required property. Format determines the method of serialization that will be used to store the data (listed below).
MaxByteSize | This optional property sets the maximum size of the instance in bytes.
IsFixedLength | This optional property determines if the total byte-length of the type is fixed or variable. The default value for this property is false.
IsByteOrdered | This optional property, meaningful only for Format.Native and Format.UserDefined, determines if the binary representation of the UDT is ordered. Binary ordering affects various aspects of the UDT, including index, comparison, and ordering abilities. The default for this property is false.
Figure 1 Class Attributes
Figure 1 Class Attributes

The three serialization formats available in Format.SerializationFormat are: UserDefined, Native, and SerializedDataWithMetadata. As its name suggests, UserDefined serialization is user defined and must be implemented in assembly code. For example, an assembly could do some custom binary serialization. In order to do this, the class must implement the IBinarySerialize interface, specifically its Write and Read methods. This is the most flexible serialization method and can be just as fast as the Native format (depending on the implementation of the Write and Read methods).

The Native format uses native SQL Server binary serialization. It's the fastest, but also the least flexible. You can only use this serialization format if the public properties of the class are fixed length, value type data types. For example, if the properties are of numeric or date/time types, Native can be used. If, however, you expose a string value property, you cannot use the Native format.

SerializedDataWithMetadata uses .NET serialization. It's very flexible because the serialization is handled automatically and is available for more data types, including strings and reference types. Unfortunately, it's also the slowest, performing at an order of magnitude slower than the Native format. More importantly, SerializedDataWithMetadata will not be supported in future betas.

As such, Microsoft recommends that SerializedDataWithMetadata not be used for any new code. If you're already using it, it should only be used in development and never in production. Classes created with SerializedDataWithMetadata should be converted to UserDefined format and should implement IBinarySerialize, using the Read and Write methods to define the custom serialization of the public class members.

Here is an example of these class attributes that would satisfy the minimal requirements for a UDT:

// C#
[Serializable]
[SqlUserDefinedType (Format.Native)]

' Visual Basic .NET
<Serializable(), SqlUserDefinedType(Format.Native)>
As you can see, I am using the two required attributes, and for the SqlUserDefinedType attribute, I am setting its required property.

Comparing UDT Values

The IsByteOrdered property affects how SQL Server uses the instance of a UDT when performing binary comparisons. In other words, if IsByteOrdered is false, the binary representation of the UDT is not comparable with another value. This means that the field defined as this UDT cannot participate in operations that require comparison, such as sorting and indexing. Because it cannot be used in an index, the field cannot be a primary or unique key, which in turn means it cannot participate in referential integrity. It also cannot be used in an ORDER BY, GROUP BY, or PARTITION BY clause.

When IsByteOrdered is true, SQL Server uses the binary representation of the UDT for all comparisons without the need to create a managed instance of the UDT. This allows for the features I just mentioned as well as others including indexability, primary and foreign keys, check and unique constraints, ORDER BY, GROUP BY, and PARTITION BY clauses, ordered comparisons, and comparison operators.

If you need any of these abilities for your UDT, you must set IsByteOrdered equal to true. The one caveat, however, is that IsByteOrdered must only be set to true if the UDT author can guarantee that the serialized binary data is exactly the same as the semantic ordering of the information. Let's take a look at an example to better understand this.

If the Point class represents a point in space, with X and Y (both defined as Int32) as its only properties, would it be a candidate for binary ordering? In other words, can any one Point be thought of as less than or greater than another Point? For example, in comparing PointA (4, -1) to PointB (2, 3), which of these is greater than the other? There is no simple answer to this, just as there is no simple way to store binary data to reflect this ordering. Thus, Point is not a class that would implement binary ordering.

Yukon supports comparison operators (and related features that rely on comparison support like GROUP BY and ORDER BY) on values of a UDT only if the serialized representation of the value is binary ordered. The UDT author indicates that the serialized representation has this property by setting IsByteOrdered to true in the type definition. The ordering of the binary representation of the data must also be equivalent to the semantic order for the type. In other words, performing a comparison operation on the serialized bytes will produce the same result as performing the comparison in managed code.

You should note that in the Native format, SQL Server ensures that the binary representation is comparable if this property is set. In the UserDefined format, it is the responsibility of the developer to pick a normalization algorithm that has this property. Also, all three serialization formats support the equal to (=) and not equal to (!=) comparison operators. The Native and UserDefined serialization formats additionally support the greater than (>), less than (<), greater than or equal to (>=), and less than or equal to (<=) comparison operators.

Nullability

All Yukon UDTs are null-aware, but in order for the UDT to recognize null as a valid value, the class must implement the INullable interface. This is done as follows:

// C#
public class MyClass: INullable

' Visual Basic .NET
Public Class MyClass
    Implements INullable

INullable contains a single property, IsNull, that the class must implement, like so:

// C#
public bool IsNull
{
    get {return is_Null;}
}

' Visual Basic .NET
Public ReadOnly Property IsNull() As Boolean Implements INullable.IsNull
    Get
        Return (is_Null)
    End Get
End Property
The is_Null variable is private and holds the state of null for the instance. Finally, the class must have a static property called Null that returns a null value instance of the UDT. This allows the UDT to return a null value if the instance is indeed null, as shown in the code in Figure 3.

Figure 3 Null Values (Close Figure 3)
// C#
public static MyClass Null
{
    get
    {
        MyClass MC = new MyClass();
        MC.is_Null = true;
        return (MC);
    }
}

' Visual Basic .NET
Public Shared ReadOnly Property Null() As MyClass
    Get
        Dim MC As New MyClass
        MC.is_Null = True
        Return (MC)
    End Get
End Property

Supporting Conversion

The UDT must support conversion to and from a string. To do this, the class needs two public methods defined in its interface. The first is a static method, Parse, which allows a string value to be converted into a UDT instance. Here is the function definition:

// C#
public static MyClass Parse(SqlString s)

' Visual Basic .NET
Public Shared Function Parse(ByVal s As SqlString) As MyClass
The second method is an override method of the instance method ToString, as shown here:
// C#
public override string ToString()

' Visual Basic .NET
Public Overrides Function ToString() As String
This method allows a UDT to be converted into a string value. Although not enforced, a proper implementation of these two should be inverses of each other. That is, these two methods should allow a string to be converted into a UDT using Parse and then back to the original string value when converted back using the ToString method.

Other Requirements

In addition to the features I've just mentioned, there are a few other requirements you should be aware of. Since a class needs a constructor, you need to implement one in a UDT. A UDT class requires a zero-argument public constructor, like this:

// C#
public MyClass () { }
Additionally, you can create other public constructors. This is the only time when a public member of a type can be overloaded. The following is a list of other limitations you should be aware of when creating your UDT:
  • No modifiable static data members are allowed in your .NET code.
  • Fixed-length string and fixed-length binary data types are not supported.
  • All class, method, and property names must conform to the Yukon sysname system data type, meaning that your public names can be no longer than 128 characters.
  • Inheritance is not supported; Yukon cannot utilize inheritance, although it can still exist in the .NET code.

Note that the SQL type system is not aware of the inheritance hierarchy among UDTs. However, UDT authors can use inheritance as an implementation mechanism in the way they structure their classes. You cannot call inherited methods from T-SQL, although you can call such methods in the managed implementation of the type.

You should also be forewarned that if you do create an overloaded method, it will not be caught when you register your assembly or even create your type in SQL Server. This is because the detection of an overloaded method occurs at run time, not when the type is created. Thus, overloaded methods can exist in the class as long as it is not invoked. If the class is invoked, an error will be raised that states: "More than one method, property, or field was found with name method_name in class class_name in assembly assembly_name." Overloaded methods, properties, or fields are not supported. Also, note that there is an exception to this rule, as I already mentioned in relation to constructors.

The SqlMethod Attribute is required when you need to mark a method as deterministic or allow a method to be a mutator. There are four parameters for this attribute:

Deterministic This marks a method as deterministic (true) or nondeterministic (false). It is false by default.

OnNullCall If OnNullCall is false, the method returns NULL without evaluating the method when at least one of its input arguments is NULL. If OnNullCall is true (which is the default), the resulting value is determined by evaluating the method, regardless of the argument inputs.

IsMutator This property, which is false by default, dictates if a method can be a mutator of the instance. By default, method calls cannot modify the state of the instance. If you create a property that needs to modify instance state, you need to mark it as a mutator by using this property. Please note that a property is considered a mutator by default and does not require the use of this property.

DataAccess This property states whether a function or a method includes SQL SELECT statements. It can be set to either DataAccessKind.None or DataAccessKind.Read.

Let's review two examples of a CLR-based UDT implementation. The first is a simple Point class, shown in Figure 4. This type has two properties, X and Y, and a public method, DistanceTo, that returns the distance of the current Point to another Point that is passed in as an argument. You will see its usage in Yukon later in this article. Note that this could also easily be defined as a struct.

Figure 4 Point Class in Visual Basic .NET (Close Figure 4)
Imports System
Imports System.Data.SqlServer
Imports System.Data.SqlTypes
Imports System.Runtime.Serialization

<Serializable(), SqlUserDefinedType(Format.Native)> Public Class Point
    Implements INullable

    Private is_Null As Boolean
    Private m_x As Decimal
    Private m_y As Decimal

    Public ReadOnly Property IsNull() As Boolean _
       Implements INullable.IsNull
        Get
            Return (is_Null)
        End Get
    End Property

    Public Overrides Function ToString() As String
        If Me.IsNull Then
            Return "NULL"
        Else
            Return Me.m_x & ":" & Me.m_y
        End If
    End Function

    Public Shared Function Parse(ByVal s As SqlString) As Point
        If s.IsNull Then
            Return Nothing
        Else
        'Parse input string here to separate out points
        Dim pt as new Point()
        Dim str as String = Convert.ToString(s)
        Dim xy() as String = str.Split(":")
        pt.x = xy(0)
        pt.y = xy(1)
            Return (pt)
        End If
    End Function

    Public Shared ReadOnly Property Null() As Point
        Get
            Dim pt As New Point
            pt.is_Null = True
            Return (pt)
        End Get
    End Property

    Public Property X() As Decimal
        Get
            Return (Me.m_x)
        End Get
        Set(ByVal Value As Decimal)
            m_x = Value
        End Set
    End Property

    Public Property Y() As Decimal
        Get
            Return (Me.m_y)
        End Get
        Set(ByVal Value As Decimal)
                m_y = Value
        End Set
    End Property

    Public Function DistanceTo (ByVal other AS Object)
        As Decimal
        Dim pt2 As New Point
        Dim pt1 As Point
        pt1 = me
        Return System.Convert.ToDecimal(System.Math.Sqrt( _
                System.Convert.ToDouble( _
                (pt1.x - pt2.x) * (pt1.x - pt2.x) +
                (pt1.y - pt2.y) * (pt1.y - pt2.y))))
    End Function

End Class

The second example is an Address class (see Figure 5), which stores address, city, state, and ZIP code data. This class demonstrates the UserDefined serialization format by implementing IBinarySerialize and the Write and Read methods.

Figure 5 Address Class in C# (Close Figure 5)
using System;
using System.Data.SqlTypes;
using System.Data.SqlServer;
using System.Data.Sql;

[Serializable]
[SqlUserDefinedType (Format.UserDefined, MaxByteSize = 150)]
public class Address: INullable, IComparable, IBinarySerialize {
    private bool is_Null;
    private string m_address1;
    private string m_address2;
    private string m_address3;
    private string m_city;
    private string m_state;
    private string m_zip;

    #region INullable Members

    public bool IsNull {
        get {return (is_Null);}
    }

    public static Address Null {
        get {
            Address addr = new Address();
            addr.is_Null = true;
            return (addr);
        }
    }

    #endregion

    #region Constructor
    public Address() {
        this.is_Null = true;
        this.m_address1 = "";
        this.m_address2 = "";
        this.m_address3 = "";
        this.m_city = "";
        this.m_state = "";
        this.m_zip = "";

    }
    #endregion

    #region String Conversion Members
    public override string ToString ()
    {
        if (this.IsNull)
            return "null";
        else
        {
            string delim = new string( (new char[] {';'}));
            return (this.m_address1 + delim + this.m_address2
                    + delim + this.m_address3 + delim
                    + this.m_city + delim + this.m_state
                    + delim + this.m_zip);
        }
    }
    public static Address Parse(SqlString s)
    {
        if(s.IsNull)
            return null;
        else
        {
            Address addr = null;
            string str = Convert.ToString(s);
            string[] a = null;
            a = str.Split(new char[] {';'} );
            addr.m_address1 = a[0] == null ? "" : a[0];
            addr.m_address2 = a[1];
            addr.m_address3 = a[2];
            addr.m_city = a[3];
            addr.m_state = a[4];
            addr.m_zip = a[5];
            addr.is_Null = false;
            return (addr);
        }
    }
    #endregion

    #region Class Properties
    public string address1
    {
        get {return (this.m_address1);}
        set
        {
            this.m_address1 = value;
            this.is_Null = false;
        }
    }
    public string address2
    {
        get {return (this.m_address2);}
        set
        {
            this.m_address2 = value;
            this.is_Null = false;
        }
    }
    public string address3
    {
        get {return (this.m_address3);}
        set
        {
            this.m_address3 = value;
            this.is_Null = false;
        }
    }
    public string city
    {
        get {return (this.m_city);}
        set
        {
            this.m_city = value;
            this.is_Null = false;
        }
    }
    public string state
    {
        get {return (this.m_state);}
        set
        {
            this.m_state = value;
            this.is_Null = false;
        }
    }
    public string zip
    {
        get {return (this.m_zip);}
        set
        {
            this.m_zip = value;
            this.is_Null = false;
        }
    }

    public string cityStateZip()
    {
        return this.m_city + ", " + this.m_state + " "
                           + this.m_zip;
    }
    #endregion

    #region IComparable Members

    //Override the Equals method
    public override bool Equals (object other)
    {
        return this.CompareTo (other) == 0;
    }

    //Override the GetHashCode method
    public override int GetHashCode ()
    {
        if (this.IsNull)
            return 0;

        return this.ToString ().GetHashCode ();
    }

    public int CompareTo (object other)
    {
        if (other == null)
            return 1; //by definition

        Address addr = other as Address;

        if (addr == null)
            throw new ArgumentException ("the argument to
                compare is not a Address");

        if (this.IsNull)
        {
            if (addr.IsNull)
                return 0;

            return -1;
        }

        if (addr.IsNull)
            return 1;

        return this.ToString().CompareTo(addr.ToString());
    }

    #endregion

    #region IBinarySerialize Members
    public void Write (System.IO.BinaryWriter w)
    {
        byte header = (byte)(this.IsNull ? 1 : 0);

        w.Write (header);
        if (header == 1)
        {
            return;
        }

        w.Write(this.address1);
        w.Write(this.address2);
        w.Write(this.address3);
        w.Write(this.city);
        w.Write(this.state);
        w.Write(this.zip);

    }

    public void Read (System.IO.BinaryReader r)
    {
        byte header = r.ReadByte();

        if (header == 1)
        {
            this.is_Null = true;
            return;
        }

        this.is_Null = false;
        this.m_address1 = r.ReadString();
        this.m_address2 = r.ReadString();
        this.m_address3 = r.ReadString();
        this.m_city = r.ReadString();
        this.m_state = r.ReadString();
        this.m_zip = r.ReadString();
    }
    #endregion

}

//}

Using a UDT in the SQL Server Create Assembly

Once you have compiled this class into an assembly, the next step is to register it within SQL Server Yukon. To do this, you first need to load the assembly with the CREATE ASSEMBLY statement, which registers the assembly in the current database. Keep in mind that if you would like to use an assembly in multiple databases, each database must separately register the assembly. The syntax is as follows:

CREATE ASSEMBLY ASSEMBLY_NAME FROM 'PATH\ASSEMBLY_NAME.dll'

Note that the SQL assembly name must match the CLR assembly name, or you will not be able to register it. Here's an example:

CREATE Assembly YukonCLR FROM
'C:\Projects\"Yukon"\UDTs\Point\"Yukon"CLR.dll'
The registered assembly exists only in the context of the current database and is stored in several system objects. The implication here is that the DLL in the file system is only referenced once to create the internal representation of the DLL. After registering an assembly, its DLL can be deleted from the file system and your UDT in Yukon will continue to function normally. Since the assembly is loaded into the current database, it cannot be used by other databases. Each database that needs the assembly reference must make its own call to the CREATE ASSEMBLY statement. Although this may sound obvious, you should also note that only the compiled DLL is loaded into the database, whereas the source code is not. Like all objects in SQL Server, the name of the assembly needs to conform to sysname conventions (with a maximum length of 128 characters).

Create Type

Take a look at the syntax for the CREATE TYPE statement:

CREATE TYPE [TYPE_SCHEMA_NAME.]TYPE_NAME EXTERNAL NAME
ASSEMBLY_NAME:CLASS_NAME
Once the assembly is loaded you use the CREATE TYPE statement to add the type to the list of available types for that database. Like an assembly, it will be created in the current database only, and also like an assembly it too is bound by sysname conventions. The following example will create the Point UDT from the Visual Basic® .NET CLR code I just showed:
CREATE TYPE Point EXTERNAL NAME YukonCLR:Point

The name of the UDT (in this case, Point) must be unique for the given schema name in the database context. This means that if you have two different assemblies registered in a database that have a common class name, only one of those classes can be created as a type in that database in the same schema. In the following code example, the Point class exists in both the YukonCLR and YukonCLR2 assemblies. Once the Point class of YukonCLR is created as a UDT in the current database, the Point class of YukonCLR2 cannot be created unless it is created in a different schema. For now, let's assume the following examples are created in the MySchema schema:

CREATE Assembly YukonCLR FROM 'C:\Projects\Yukon\UDTs\Point\YukonCLR.dll'
CREATE Assembly YukonCLR2 FROM 'C:\Projects\Yukon\UDTs\Point\YukonCLR2.dll'
CREATE TYPE Point EXTERNAL NAME YukonCLR:Point                --OK
CREATE TYPE Point EXTERNAL NAME YukonCLR2:Point
--BAD - created in MySchema
CREATE TYPE DifferentSchema.Point EXTERNAL NAME YukonCLR2:Point
--OK  - created in DifferentSchema
CREATE TYPE Point EXTERNAL NAME YukonCLR2:Triangle            --OK

Using a UDT

After the type is registered as one of the available types for the database, you can begin to use it in the creation of objects such as tables, as shown here:

CREATE TABLE Points
(
    PointID int NOT NULL,
    Pnt Point NOT NULL
)
Using a UDT in a table definition requires no special coding. You define the table the same way you would if you were using one of the intrinsic data types, such as an int or nchar.

Before I go any further, let's take a look at the syntax for both property and method usage of a user-defined type in Yukon. It is very similar to C# and Visual Basic .NET syntax, except here the property or method is preceded by a double colon (::) instead of a period. By Beta2, the :: notation should disappear in favor of the period notation. The following shows this syntax for both a property and a method:

Property Use Syntax: <implemented_type>::<property_name> =
    <scalar_expression>
Method Use Syntax: <implemented_type>::<method_name>([arguments])
In order to populate the table with data using T-SQL, you can run the following script:
DECLARE @startPoint Point
DECLARE @endPoint Point
SET @startPoint = CAST('10:10' AS Point)
SET @endPoint::X = 5
SET @endPoint::Y = 3

INSERT Points
VALUES(1, @startPoint)

INSERT Points
VALUES(2, @endPoint)

Much of this code should be familiar to anyone who has experience with T-SQL variables, DML statements, and SELECT statements from previous versions of SQL Server. There are, however, some elements of the T-SQL code that stand out. The first is the use of the CAST function to assign the X and Y values to @startPoint. When you use the CAST function in this manner, the Parse method is called on the UDT to populate the UDT instance with data. Next, the X and Y properties of the @endPoint variable are set individually, thus allowing you to explicitly pass values to properties of the UDT.

The next portion of the code inserts the instance of the UDT type into the Points table using the standard INSERT statement. Selecting the appropriate values from the UDT stored in the table is a process of specifying the property or method you want to call by using the syntax ColumnName::Property or ColumnName::Method. Even if the method takes no arguments, parentheses are required. In the following example, you will select the X and Y values from the entire set of points:

SELECT Pnt::X AS XValue, Pnt::Y AS YValue FROM Points
Here I will select a point and store it in the variable @pt, and then use it to check the distance to another point in the table using the DistanceTo method:
DECLARE @pt Point
SELECT @pt = Pnt FROM Points WHERE PointID = 2

SELECT Pnt::DistanceTo(@pt) AS Distance
FROM Points
WHERE PointID = 1

The next example, Figure 6, shows how to use the Address class in Yukon (assuming it is part of the YukonCLR assembly and you have created its type already). First you create three variables of type Address. Next, you assign the address information and fetch the formatted city, state, and ZIP code using the cityStateZip method. The second part of the code makes a copy of the first address (@addr) into the second (@addr2) using the CAST function. This creates a copy, not a reference to the first variable. Any subsequent changes to @addr will not be reflected in @addr2. Finally, you check the variable @addr3 to see if it is null. Since it was only declared and never assigned any value, it will indeed be null, as you can see.

Figure 6 Using the Address Class (Close Figure 6)
DECLARE @addr Address, @addr2 Address, @addr3 Address
SET @addr::address1 = '1 Main Street'
SET @addr::city = 'Dallas'
SET @addr::state = 'TX'
SET @addr::zip = '75321'
PRINT @addr::cityStateZip()

SET @addr2 = CAST(@addr AS Address)
PRINT @addr2:: cityStateZip()

IF @addr3 IS NULL
    PRINT 'YES'
ELSE
    PRINT 'NO'

Dropping and Altering UDTs and Assemblies

UDTs are easily dropped by using the DROP TYPE statement, as seen in the following example, which drops the Point UDT:

DROP TYPE Point
But there is a catch—what if the UDT is currently in use (for example, in a column's definition) like in the Point example? Obviously, you cannot remove the type from the database if another object depends on it. So you must be sure to drop all objects that use the type before dropping the type itself. The same holds true for dropping assemblies. All UDTs must be dropped before you can drop an assembly. Based on this latest example, the following would have to be done to remove my YukonCLR assembly:
DROP TABLE Points
DROP TYPE POINT
DROP TYPE Address
DROP ASSEMBLY YukonCLR

As you can see, the syntax for dropping any of these objects is simple. When dropping an assembly, you remove the code base from your current database and must have a compiled version available if you want to add it back to your database.

ALTER TYPE is not valid to use due to a variety of factors such as storage representation, ordering, and interfaces. You will have to drop and recreate these objects if a change needs to be made. ALTER ASSEMBLY, however, is valid for making bug fixes to existing assembly code.

In case you're wondering, there is deep support for UDTs both in the managed ADO.NET stack and in the unmanaged client APIs—OLE DB, ODBC, and ADO.

Conclusion

You now know how to implement user-defined types in Yukon and .NET and how to use these implementations in Yukon. Of course, there are other features of UDTs that I did not explore here that will likely be covered in future issues of MSDN® Magazine.

For related articles see:
Creating User-Defined Data Types

Peter W. DeBetta is a trainer at Wintellect, a consultant, and developer who helps clients develop enterprise-level software solutions using Visual Basic, ASP, C#, ASP.NET, and SQL Server. Peter has coauthored several books and is currently working on a book on SQL Server "Yukon" to be published by Microsoft Press.


Standard I/O: Console Appplications in .NET, or Teaching a New Dog Old Tricks

Standard I/O

Console Appplications in .NET, or Teaching a New Dog Old Tricks

Michael Brook
This article assumes you're familiar with C#
Code download available at: NETConsoleApps.exe (228KB)
SUMMARY

The Microsoft .NET Framework is not just about Windows Forms and Web services. This article discusses the simplest kind of Framework-based application—the console app—along with the frequently overlooked constructs of standard input/output and the pipe. When designed carefully, console applications offer a surprisingly powerful way of solving complex programming problems. One of the more exciting aspects of this approach is that while each application in the pipe is fairly simple, the result of their interaction can be a relatively complex task. Here the author explores the ins and outs of writing these console apps.

Contents
The Command Prompt
Building Console Apps in .NET
A Design Pattern for Console Apps
Controlling Console Apps with the Process Class
Yes, But Can it Read an RSS Feed?
Conclusion

T
he Microsoft® .NET Framework is more than Windows® Forms and Web services. In this article, I'll explore the simplest kind of Framework-based application—the console application—along with the frequently overlooked constructs of standard input/output and the pipe. If you are anything like me, you'll find yourself full of new ideas once you become familiar with these venerable concepts.

So, what is a console app? It's tempting to say that console applications are applications invoked from the Windows command prompt (cmd.exe), though this wouldn't be strictly true. Simply put, a console app is any program with access to three basic data streams: standard input, standard output, and standard error. As the names suggest, standard input represents data flowing into the program, standard output represents data flowing out, and standard error represents a special kind of data flowing out (namely, error messages). Along with its command-line arguments, these data streams represent the runtime context of the console app.

It's easy to spot a console application running in Windows; it's the familiar black box with gray text. This is the way Windows makes the standard data streams available to you. Whatever you type (or paste) into the black window becomes standard input to the console application and whatever text you see displayed is actually the standard output and/or standard error (it can be difficult to tell them apart sometimes). Here's an example. If you select Run from the Windows Start menu and type the following, a console window pops up:

at.exe /delete
The console window states: "This operation will delete all scheduled jobs. Do you want to continue this operation?" This is actually the standard output of the program, which is paused while it waits for your response to appear on standard input. By typing "n" and pressing Enter, you are sending at.exe some standard input and causing it to perform some actions (in this case, you are saying "no," causing the program to end). I chose this example because the at.exe /delete command waits for standard input from the user. Other commands (at.exe by itself, for instance), simply send something to standard output and exit before you can see what it was.

The Command Prompt

Console applications, then, are just programs with access to the three basic data streams. Windows makes the data streams available to you through the console window. So far, this is probably more nostalgic than useful, but Windows comes with a special console application called cmd.exe that allows you to do some more interesting things with the three standard data streams. For example, open a command prompt and type the following ("C:\>" of course represents the actual command prompt and should not be typed into the box):

C:\> at.exe /?
You should see some reference information for the at.exe program. Now type the following:
C:\> at.exe /? >atref.txt
Although nothing appears on the screen, the atref.txt file now contains the reference information for the at.exe program. Cmd.exe provides the special > (greater-than) operator for sending standard output to a file. The important point is that the > operator is not part of the operating system, but part of cmd.exe. If you need proof, try typing the same command line into the Run box of your Windows Start menu (thus bypassing cmd.exe altogether). You will see the Windows console briefly appear, but atref.txt is nowhere to be found.

Cmd.exe isn't a magical window on the guts of the operating system—it is just a particularly useful console application that allows you to do some convenient (and powerful) things with the standard data streams of other console applications. This becomes very interesting when you consider that standard input and output are really compatible ideas and that standard output from one program can serve as standard input to another, and so on and on. This idea is nearly as old as the idea of standard data streams themselves and dates back to a version of Unix that first appeared in 1972. The mechanism for linking together console applications in such a way is called the "pipe." Its command-line representation (on Windows and Unix) is the vertical bar character (|). Since its inception, the pipe has had a profound influence on Unix software, which is famous for being comprised of many simple, single-purpose, "pipeable" programs.

Cmd.exe supports pipes too. For example, the following command line sends the reference information for the at.exe program to a program called findstr.exe:

C:\> at.exe /? | findstr.exe "\/delete"
Findstr.exe is a pattern-matching text searcher that comes with Windows. In this case, you are telling it to output only the lines with the word "/delete" in them. The findstr.exe program is highly specific in its purpose. So are most of the basic Windows commands, for that matter. The pipe allows you to string them together to form complex commands. If cmd.exe didn't allow use of the pipe, you would end up with a proliferation of console applications that differed only in some small detail, such as the ability to sort or filter their output. Or, you would end up with prohibitively complex commands with lengthy and confusing command-line switches. Without the pipe, the command I just showed might have been written like so:
C:\> at.exe /? /filter:\/delete /write:atref.txt
This would require the developer of at.exe to understand string pattern matching and file writing in addition to what the at.exe program actually does.

The world of console applications is like an ecosystem, in which programs forge symbiotic relationships and thus become useful. This is especially true on Unix, where that ecosystem has existed much longer. Some of the most useful programs on Unix ("sort," for example) are utterly useless on their own. For some reason, interoperability via the pipe and standard I/O is a relative rarity in the world of Windows. This might be because Windows has historically been more graphically oriented than Unix, or perhaps it is because Visual Basic® (through version 6.0) didn't make it easy to write console applications. Regardless, the .NET Framework takes a great leap forward by looking back. It is easier than ever to write applications that take advantage of standard I/O and the pipe. The rest of this article looks at two sides of console applications—first, how to build them using the .NET Framework, and second, how to interoperate with them from the .NET Framework.

Building Console Apps in .NET

Different programming environments have different semantics for producing console applications. In VBScript, for example, the script engine starts executing code at the top of the file and continues until there is no more code to execute. In .NET, the common language runtime (CLR) looks for a specific entry point in the compiled executable. Namely, it looks for a static method marked with the .entrypoint IL directive (often the Main method). The CLR calls the entry method and the program proceeds from there. When you create a new C# console application in Visual Studio® .NET, you are given a simple class with just a static Main method:

using System;
namespace ConsoleApplication1
{
    class Class1
    {
        [STAThread]
        static void Main(string[] args)
        {
        }
    }
}

When the C# compiler compiles this code, it marks Main with the .entrypoint IL directive, making it the application's starting method. This is the simplest possible console application—it compiles but doesn't actually do anything useful when you run it. Notice how the Main method takes an array of strings as its only argument. This represents the command-line parameters passed to the console application (unlike with standard C applications, the name of the executable is not passed to the application as the first argument). Also notice how the Main method doesn't return anything. Although console apps can return an int (which can then be queried via ERRORLEVEL), return values aren't needed for pipe applications.

All communication with the outside world is done through the standard data streams. Using the System.Console.In, System.Console.Out, and System.Console.Error static properties, the .NET Framework makes the standard data streams available to the console application. System.Console.In is an instance of System.IO.TextReader; Out and Error are instances of System.IO.TextWriter. This proves to be a big advantage since you can work with the standard data streams just as you would a file or network stream.

Figure 1 shows a simple example of working with standard input and output. This is the code for a program called Passthrough, a console application that simply passes standard input through to standard output. You can compile it through Visual Studio .NET or using the command-line tools. For your reference, this and all other examples in this article will compile with a command-line call to the C# compiler, such as:

C:\> csc.exe /t:exe /out:Passthrough.exe Passthrough.cs
If you were to run this program as follows, it would appear as though you ran the dir command by itself:
C:\> dir | Passthrough.exe

Figure 1 Passthrough.exe (Close Figure 1)
using System;
namespace ConsoleApps
{
    class Passthrough
    {
        [STAThread]
        static void Main(string[] args)
        {
            string currentLine = Console.In.ReadLine();
            while(currentLine != null)
            {
                Console.Out.WriteLine(currentLine);
                currentLine = Console.In.ReadLine();
            }
        }
    }
}

An interesting property of console applications that read from standard input is that they can be used (and tested) interactively. For example, if you run Passthrough by itself (Passthrough.exe), it appears as though the program is hung—cmd.exe doesn't give you back the command prompt. If you type something and press Enter, the program echoes it back to you. Passthrough is running just as before, except that it is getting standard input from the keyboard instead of from the dir command. If you were to check your Windows Task Manager, you would see a task for Passthrough.exe. This is to be expected—Passthrough is a normal executing program that Windows identifies with a process ID. Try running the following command:

C:\> Passthrough.exe | Passthrough.exe
Here, the output of your interactive session with Passthrough routes to another instance of Passthrough, whose output routes to the console. If you check Task Manager again, you will see two instances of Passthrough.exe. Why? Because both instances of Passthrough are running simultaneously. In fact, you would see an instance of every program that you included in a pipeline. What does this mean? Remember, pipelines are the domain of cmd.exe—they have no meaning to the underlying operating system. When you construct a pipeline, as in the example I just showed, cmd.exe does the work of launching the required applications and routing their standard data streams according to the specifications of the pipeline. To the operating system, they are separate processes; you can see this through Task Manager. To the user, however, cmd.exe makes it look as though there were a single executable at work.

Most console applications process one line of standard input at a time and send one line to standard output at a time. As soon as a program that is upstream in a pipeline is finished with a line of input, it is available to be processed by a downstream program. The upstream program might have moved well beyond the line that the downstream program is currently working on. While this design will not necessarily improve the overall performance of the entire pipeline, it will tend to result in a lower overall memory requirement as well as a shorter time until the first line of output is finished being processed by the whole pipeline. There are circumstances where line-at-a-time processing is impossible (such as a sorting program or a program that needs to work with an in-memory XML tree). In these cases, standard input needs to be read in its entirety before anything can be sent to standard output. Meanwhile, downstream programs in the pipeline sit idle, waiting for something to appear on their standard input.

Just about anything can be accomplished in the Main routine of a console application. For example, the Main routine in Figure 2 builds on the idea of the Passthrough example but performs a regular expressions-based replacement on each line of standard input. If you were to run the compiled application as follows, you would see the normal output of dir, but with all instances of the string "<DIR>" replaced with "****":

C:\> dir | Replace.exe "<DIR>" "****"

Figure 2 Main (Close Figure 2)
static void Main(string[] args)
{
    if(args.Length < 2)
    {
        Console.Error.WriteLine("Usage:");
        Console.Error.WriteLine("Replace \"search\" \"replace\"");
        return;
    }

    System.Text.RegularExpressions.Regex re =
        new System.Text.RegularExpressions.Regex(args[0]);
    string currentLine = Console.In.ReadLine();
    while(currentLine != null)
    {
        Console.Out.WriteLine(re.Replace(currentLine, args[1]));
        currentLine = Console.In.ReadLine();
    }
}

A Design Pattern for Console Apps

You could continue to put all processing logic in the Main routine of your console application, but eventually you would get tired of always typing the same basic code. All of your programs would start to look pretty similar to Passthrough and Replace. It would be nice if you could inherit some functionality and simply change the parts that you need to change. You can achieve such an effect with a design pattern known as the Template Method. In it, you create a skeleton algorithm in a base class and then allow child classes to fill in the details. Expressed in pseudocode, the skeleton algorithm might look like the following pseudocode snippet:

If the arguments to the program are valid then
    Do necessary pre-processing
    For every line in the input
        Do necessary input processing
    Do necessary post-processing
Otherwise
    Show the user a friendly usage message

An example of subclass responsibility is to define what pre-processing, post-processing, and input processing actually mean to that particular subclass. You could put logic like this in the static Main method, effectively making it the skeleton algorithm, but this presents two problems. First, console apps compile to executables. .NET executables can be used as class libraries, but Visual Studio .NET makes that difficult to do. Thus, it's hard to allow subclasses to fill in the details because it's difficult to create subclasses.

The second problem is that Main is a static method. Static methods don't belong to instances of a class; they belong to the class itself. This means that all the steps of the algorithm (pre-process, post-process, and so on) would have to be implemented as static methods as well, preventing them from being overridden. Only instance methods can be overridden and, as such, the methods representing the steps of the skeleton algorithm need to be instance methods of a class.

The best way to get around these limitations is to separate each console application into two classes. The first class is just the normal console application class with the static Main entry point (call it the "chassis"). Instead of containing processing logic, however, the Main routine creates an instance of the second class (call it the "engine") and delegates processing to it. For the Passthrough example, the chassis would look like the code in Figure 3. Notice how the engine class has a similar entry point, Main, but that it is an instance method rather than a static method. The Main method of the engine class will become your skeleton algorithm, allowing you to employ the Template Method design pattern. ConsoleEngineBase is the name of the class that will hold the implementation of the skeleton algorithm and all engines will inherit from it, supplying their own versions of the algorithm's steps along the way. The complete code for ConsoleEngineBase is given in Figure 4.

Figure 4 ConsoleEngineBase (Close Figure 4)
using System;

namespace Engines
{
    public abstract class ConsoleEngineBase
    {
        private string[] m_args;
        protected bool ReadInput = true;
        protected System.IO.TextReader In = null;
        protected System.IO.TextWriter Out = null;
        protected System.IO.TextWriter Error = null;

        public ConsoleEngineBase()
        {
            //by default, read from/write to standard streams
            this.In = System.Console.In;
            this.Out = System.Console.Out;
            this.Error = System.Console.Error;
        }

        public void Main(string[] args)
        {
            this.m_args = args;

            if(this.ValidateArguments())
            {
                this.PreProcess();
                if(this.ReadInput)
                {
                    string currentLine = this.In.ReadLine();
                    while(currentLine != null)
                    {
                        this.ProcessLine(currentLine);
                        currentLine = this.In.ReadLine();
                    }
                }
                this.PostProcess();
            }
            else
                this.Error.Write("Usage: " + this.Usage());
        }

        public void Main(string[] args,
            System.IO.TextReader In,
            System.IO.TextWriter Out,
            System.IO.TextWriter Error)
        {
            //this version of Main allows alternate streams
            this.In = In;
            this.Out = Out;
            this.Error = Error;
            this.Main(args);
        }

        protected virtual bool ValidateArguments()
        {
            //override this to add custom argument checking
            return true;
        }

        protected virtual string Usage()
        {
            //override this to add custom usage statement
            return "";
        }

        protected virtual void PreProcess()
        {
            //override this to add custom logic that
            //executes just before standard in is processed
            return;
        }

        protected virtual void PostProcess()
        {
            //override this to add custom logic that
            //executes just after standard in is processed
            return;
        }

        protected virtual void ProcessLine(string line)
        {
            //override this to add custom processing
            //on each line of input
            return;
        }

        protected string[] Arguments
        {
            get {return this.m_args;}
        }
    }
}
Figure 3 PassthroughChassis (Close Figure 3)
using System;
using Engines;

namespace ConsoleApps
{
    class PassthroughChassis
    {
        [STAThread]
        static void Main(string[] args)
        {
            Engines.ConsoleEngineBase engine = new PassthroughEngine();
            engine.Main(args);
        }
    }
}

Since PassthroughEngine inherits from ConsoleEngineBase, it can be implemented very simply. The complete PassthroughEngine class is shown here:

public class PassthroughEngine : ConsoleEngineBase
{
    public override void ProcessLine(string line)
    {
        this.Out.WriteLine(line);
    }
}
Passthrough doesn't need to bother with any special pre- or post-processing or checking of arguments, so it doesn't override these steps of the algorithm. It needs to do one thing—echo every line of input to output—and it does this by providing its own implementation of ProcessLine. Figure 5 summarizes the interactions between the classes in this model.

Figure 5 Class Relationships
Figure 5 Class Relationships

The benefit of employing the Template Method design pattern is obvious: inherited classes only need to change the parts of the algorithm that they require, resulting in less overall coding. But what are the consequences of breaking every console application into chassis and engine classes? There are at least two negative consequences: first, you end up with twice as many classes as before, and second, you end up with a bunch of really simple, similar chassis classes. I think these drawbacks, however, are more than outweighed by the benefits.

The first, most important benefit is that engine classes are now free to inherit from one another. PassthroughEngine can be implemented with one line of code. A second, more subtle benefit is that the engine can be chosen at run time—they are pluggable. Since all engine classes derive from ConsoleEngineBase, the chassis doesn't really care which one is used. In fact, you could generalize PassthroughChassis to be configurable at run time. Figure 6 shows the code for a chassis class that requires an assembly name and class name as command-line arguments. It uses this information to create an instance of a particular engine class. It then calls its Main method, effectively turning over processing to it. For Passthrough, the command would look like this:

C:\> GenericChassis "Engines" "Engines.PassthroughEngine"
In this case, the PassthroughEngine class is a member of the Engines namespace and resides in an assembly called Engines.dll (note that the DLL extension must be omitted from the argument in this example).

Figure 6 GenericChassis (Close Figure 6)
using System;
using Engines;

namespace ConsoleApps
{
    class GenericChassis
    {
        [STAThread]
        static void Main(string[] args)
        {
            if(args.Length < 2)
            {
                Console.Error.WriteLine("Two arguments required:");
                Console.Error.WriteLine("assembly and class name.");
                return;
            }

            Engines.ConsoleEngineBase engine = null;
            string assemName = args[0];
            string className = args[1];
            string[] newargs = new string[args.Length - 2];

            for(int i = 2; i < args.Length; i++)
            {
                newargs[i - 2] = args[i];
            }

            try
            {
                Runtime.Remoting.ObjectHandle engineHandle =
                    Activator.CreateInstance(assemName, className);
                engine = (ConsoleEngineBase)engineHandle.Unwrap();
            }
            catch(Exception e)
            {
                Console.Error.WriteLine(e.Message);
                return;
            }
            engine.Main(newargs);
        }
    }
}

This approach—separating the chassis from the engine—is usually called the Strategy design pattern (though I prefer the car metaphor). It allows an algorithm to change without having to create an entire subclass of its associated class. What's so bad about subclassing just to change one method? Having lots of classes that differ only in their implementation of a single algorithm can be confusing and hard to maintain. Families of algorithms are generally thought to be more intuitive than families of classes. In Figure 5, ConcreteEngine1 and ConcreteEngine2 are members of this family. Physically, they are implemented as classes; logically, they exist only to help the chassis implement its Main method. (This might have been more intuitively accomplished with delegates, though it would have complicated the use of the Template Method pattern.)

You may have noticed that there isn't anything console-specific about these engine classes. They simply know how to read and write System.IO.Streams. This is another benefit of this approach—the same logic that you use from the console can be used from (for example) a network server class. Just as you can swap out the engine, you can swap out the chassis, albeit with a little more work.

Even engines that are more complex than Passthrough are similarly straightforward. Think back to the Replace example. Replace is similar to Passthrough except that it performs a regular expressions-based replacement on each line of input. The complete listing for ReplaceEngine is shown in Figure 7. Notice how the PreProcess method sets up a single instance of the System.Text.RegularExpressions.Regex class and the ProcessLine method uses it on each line of input. I'll build further on these later in the article. For now, however, let's look at the other side of console apps: how to control them from within a .NET Framework-based app.

Figure 7 ReplaceEngine (Close Figure 7)
using System;
using System.Text.RegularExpressions;

namespace Engines
{
    public class ReplaceEngine : ConsoleEngineBase
    {
        private Regex replacer = null;

        protected override void PreProcess()
        {
            this.replacer = new Regex(this.Arguments[0]);
        }

        protected override void ProcessLine(string line)
        {
            this.Out.WriteLine(
                this.replacer.Replace(line, this.Arguments[1]));
        }

        protected override string Usage()
        {
            return "Replace \"findexp\" \"replaceexp\"";
        }

        protected override bool ValidateArguments()
        {
            if(this.Arguments.Length == 2)
                return true;
            return false;
        }
    }
}

Controlling Console Apps with the Process Class

By now, hopefully you are thinking of console applications as building blocks that can be strung together with pipes. Indeed, this is typically how they are used. They can, however, be used in other ways. The .NET Framework class library's Process class allows .NET Framework-based programs to interoperate with console applications, regardless of their origin. The Process class communicates with the console application by way of—you guessed it—the standard input and output streams.

One occasion when this might be useful is when a console application already exists to perform a desired task. Rather than recode it in a CLR-compliant language, it might be easier to interoperate with it via the Process class. Or perhaps using the .NET Framework isn't the best way to implement a particular bit of functionality. Consider this example. Suppose you were asked to create a graphing calculator application. How would you do it using the .NET Framework? Aside from the expected tasks of laying out forms and working with the System.Drawing library, you would find yourself faced with having to figure out how to evaluate expressions. That is, how can you find out the value of (sin(x) + 3)/16 for all values of x?

It isn't an easy problem. You would probably find yourself having to write an expression parser or attempting to compile the expression to Microsoft intermediate language (MSIL) on the fly. Or, you could take the easy way out and use the venerable VBScript Eval function. It isn't part of the .NET Framework, but it sure is easy. It turns out that it takes about 20 lines of VBScript to implement a simple calculator that works on standard input and output. If you were to run this script in a console window, you would find that you can type expressions like "1+1" or "sin(23) + 232.23 / 17" into standard input and see results like "2" or "12.8143678311189" appear on standard output. Since it loops forever, you must type Ctrl-C to end the program when using it in interactive mode.

Meanwhile, back in the .NET world, you need a way of communicating with the VBScript calculator. This is where the Process class comes in. The following code snippet starts up an instance of the VBScript calculator:

System.Diagnostics.Process calcProc = new System.Diagnostics.Process();
System.Diagnostics.ProcessStartInfo i =
    new System.Diagnostics.ProcessStartInfo();
i.FileName = "cscript.exe";
i.Arguments = "//NoLogo calc.vbs";
i.RedirectStandardOutput = true;
i.RedirectStandardInput = true;
i.RedirectStandardError = true;
i.CreateNoWindow = true;
i.UseShellExecute = false;
calcProc.StartInfo = i;
calcProc.Start();

Let's look at what's going on here. The first step is to create an instance of the Process class, called calcProc. In order to specify detailed start-up parameters for Process, the .NET class library provides a helper class, ProcessStartInfo. The next seven lines create and populate an instance of this class. The properties are described in Figure 8. Finally, the instance of the Process class is handed the instance of the ProcessStartInfo class, and the Process is started.

Figure 8 System.Diagnostics.ProcessStartInfo Properties (Close Figure 8)

Property | Description
FileName | The name of the executable to start when the Start method is called on the Process class.
Arguments | Command-line arguments to pass to the executable specified in FileName.
RedirectStandardOutput | Indicates whether the standard output for the target application (that is calc.vbs) should be routed to the instance of the Process class. The alternative is to have it go to the default location for standard output—the console.
RedirectStandardInput | Same as RedirectStandardOutput, but for standard input instead.
CreateNoWindow | Causes the target application to execute invisibly. Not setting this property causes a command window to appear.
UseShellExecute | Indicates whether the Windows shell or the Process class should be used to start the process. This property must be set to false in order to redirect the input and output streams.

Thereafter, the Framework-based application can communicate with the VBScript calculator via the StandardInput and StandardOutput properties of calcProc. For example, the following code sends an expression to the calculator and reads the result:

calcProc.StandardInput.WriteLine("1+1");
string result = calcProc.StandardOutput.ReadLine();

To turn the expression evaluator into a graphing calculator, you just need to send the expression multiple times with different values of x. Remember, the VBScript calculator stays running for the lifetime of the Process class. You just keep sending it standard input and it keeps returning standard output.

Figure 9 Graphing Calculator
Figure 9 Graphing Calculator

Figure 9 shows the finished graphing calculator. The majority of the code deals with drawing the graph and is tangential to this article (it's included in the code download at the link at the top of this article). There is, however, one more important point to make: the VBScript calculator will continue to run even after you close the Framework-based application that invoked it. To ensure that you don't have an orphaned process, you can send Ctrl-Z in input, or explicitly end it by invoking the Kill method:

calcProc.Kill();

I'll bet you never thought you'd see interoperability between the .NET Framework and VBScript! While this example is contrived, I don't think it is entirely far-fetched. There are many useful VBScript programs (not to mention Perl, Python, and C programs) already written for systems management tasks. While these will eventually get ported to managed code, it won't be for a long time. Moreover, when they do get ported, it won't be all at once. Interoperability with legacy code will always be important and standard input and output are simple, effective ways to achieve it.

Yes, But Can it Read an RSS Feed?

In case you aren't familiar with it, RSS is a way of syndicating information on the Internet in a machine-readable format. It takes advantage of the addressing and transport aspects of the Web (URLs and HTTP) while eschewing its markup language (HTML) in favor of something more structured (the RSS dialect of XML). Programs that read RSS feeds have become commonplace and are frequently considered archetypal examples of how to create XML and network-savvy applications. Figure 10 shows the XML code for a shortened RSS feed.

Figure 10 RSS Feed (Close Figure 10)
<rss version="2.0">
    <channel>
        <title>MSDN: .NET Framework and CLR</title>
        <link>http://msdn.microsoft.com/netframework/</link>
        <description>
            The latest information for developers on the Microsoft
            .NET Framework and Common Language Runtime (CLR).
        </description>
        <language>en-us</language>
        <ttl>1440</ttl>
        <item>
            <title>
                Improving Web Application Security:
                Threats and Countermeasures
            </title>
            <pubDate>Thu, 12 Jun 2003 07:00:00 GMT</pubDate>
            <description>
                Bake security into your application lifecycle.
                You'll get guidance that is task-based and modular,
                with tons of implementation steps.
            </description>
            <link>...</link>
        </item>
    </channel>
</rss>

Reading an RSS feed is really a matter of fetching a URL, transforming its contents from machine-readable to human-readable form, and displaying it to the user. Usually there is a caching step between steps one and two so that the whole thing can work offline as well. This pattern of fetch, cache, transform, and display is repeated over and over for every RSS feed in which a particular user is interested. Fetch, cache, transform, and display. It starts to sound like a pipeline. Can the world's first command-line RSS reader be far behind?

Like any good pipeline, the steps are not RSS specific. Figure 11 provides a description of how the first three steps should work. I have left out the "display" step because the console takes care of it in this case. These three console applications fit together in a pipeline, like the one shown in Figure 12.

Figure 11 Reading an RSS Feed (Close Figure 11)

Step | Description
Fetch URL | Retrieves the contents of the URL given in the first argument. Writes contents to standard out. If an error is encountered, writes error information to standard error and writes nothing to standard out.
Cache file name | Takes contents of standard in, writes it to the file given in the first argument, and passes data through to standard out. If standard in is empty, attempts to read the file given in the first argument and sends its contents to standard out. File access errors are written to standard error.
Transform stylesheet | Expects valid XML on standard in. If XML is valid, transforms it with the XSLT stylesheet given in the first argument and sends the results to standard out. Writes errors to standard error.
Figure 12 An Application Pipeline
Figure 12 An Application Pipeline

When implementing fetch, cache, and transform, you should continue with your design patterns as I've described them. To refresh your memory, all engine classes should be implemented as subclasses of the ConsoleEngineBase class. Figure 13 shows the class library so far. Code for all engines is included in the download for this article.

Figure 13 Class Library
Figure 13 Class Library

As I've shown, it isn't very difficult to add functionality to the engine classes. In all cases, it is primarily a matter of overriding PreProcess, PostProcess, and ProcessLine. All three of these new engines take advantage of PreProcess to create instance-level private resources: in the case of FetchEngine, an instance of the WebClient class; in the case of CacheEngine, a file handle; in the case of TransformEngine, an instance of the XslTransform class. The code for all three engines is included in the download for this article.

Now that the engines are implemented, you can test them. The following command line uses the generic chassis class to put the RSS reader through its paces:

C:\> GenericChassis "Engines" "Engines.FetchEngine" "http://
     msdn.microsoft.com/netframework/rss.xml" | GenericChassis "Engines"
     "Engines.CacheEngine" "netframework.xml" | GenericChassis "Engines"
     "Engines.TransformEngine" "FormatForConsole.xslt"
That's a lot of typing to view an RSS feed. You might consider creating a batch file with that in it, like this:
GenericChassis "Engines" "Engines.FetchEngine" "%1" | GenericChassis
   "Engines" "Engines.CacheEngine"
   "%2" | GenericChassis "Engines"
   "Engines.TransformEngine"
   "FormatForConsole.xslt"

Assuming you called your batch file rss.bat, you could execute it by running:

C:\> rss.bat "http://
  msdn.microsoft.com/netframework/
  rss.xml" "netframework.xml"
If you run it while connected to the Internet, you should see the contents of the RSS feed scroll past your eyes in a somewhat readable format. If you disconnect, the cache engine should kick in, allowing you to still see the feed.

Now let's say you want to take advantage of Windows Forms to improve the user interface. At first, this sounds straightforward: create an instance of the Process class, feed it the command text for the pipeline, read standard output, and show it to the user. Unfortunately, it doesn't prove to be that simple. Unlike the calculator example, this one involves three distinct processes: fetch, cache, and transform. Remember, when you run these via cmd.exe they actually appear in Task Manager as three distinct processes. Cmd.exe takes care of routing the standard data streams among them. The operating system itself has no notion of pipes. Consequently, the Process class has no notion of pipes. Trying to send a pipeline to a Process instance will not get you what you want.

There are two different ways to get around this. First, you could invoke cmd.exe via the Process class and send commands to its standard input. Cmd.exe is, after all, just another console application. It differs from other console applications only in that it expects commands rather than data on standard input. In this scenario, you would let cmd.exe do all the work of parsing the command lines and routing the standard data streams. You could monitor cmd.exe's standard output to get the results.

The second option is to do the work of routing the standard streams yourself. That is, you could create a class called Pipe that joins one Process to another. The Pipe class would basically have the job of looking for standard output of one Process and immediately sending it to standard input of the other Process. You could use a combination of processes and pipes to build complex pipelines in your code.

I tried both options and can confirm that they both work. For the first option, I created a class called CommandRunner that references an instance of Process internally. It has a single method, called RunCommand, that takes a command string and returns the output of the command as a string. The internal Process class serves as a proxy to a running instance of cmd.exe. The RunCommand method sends commands to it using standard input and reads the results on standard output.

Figure 14 Reading Dir Listing
Figure 14 Reading Dir Listing

But there's a problem here. The standard output stream of cmd.exe never ends, though it frequently contains no data to read. What does this mean? Basically, that read operations on standard output will block indefinitely when the end of the output is reached, effectively locking up the program. Figure 14 shows a typical interaction with cmd.exe via the standard streams. Cmd.exe is sent a command (dir in this case) and responds, as expected, with a directory listing. Since a directory listing can contain any number of lines, the client program doesn't know when to stop reading. Eventually, the client program will attempt to read a line that doesn't exist and will wait for it, blocking execution of the entire program.

For this reason, the code shown in Figure 15 will not work. One way around this is to run an output "monitor" on another thread, but that still leaves ambiguity as to whether the current command is finished or is just in the middle of a long operation. If you run cmd with the /c option, it will execute and exit immediately. However, you can also check this manually. To find out if cmd.exe is finished with the current command, Figure 16 shows an effective, if ugly, solution. The trick here is to insert a known string into the output of the command and then look for it. When it is found, you know you have reached the end of the current command's output.

Figure 16 Looking for End String (Close Figure 16)
public string RunCommand(string cmd)
{
    this.InnerProcess.StandardInput.WriteLine(cmd);
    this.InnerProcess.StandardInput.WriteLine("echo ---end---");
    StringBuilder output = new StringBuilder();
    string currentLine = this.InnerProcess.StandardOutput.ReadLine();
    while(currentLine != "---end---")
    {
        output.Append(currentLine);
        output.Append("\r\n");
        currentLine = this.InnerProcess.StandardOutput.ReadLine();
    }
    return output.ToString();
}
Figure 15 Reading Output (Close Figure 15)
public string RunCommand(string cmd)
{
    this.InnerProcess.StandardInput.WriteLine(cmd);
    StringBuilder output = new StringBuilder();
    string currentLine = this.InnerProcess.StandardOutput.ReadLine();
    while(currentLine != null)
    {
        output.Append(currentLine);
        output.Append("\r\n");
        //the next line will block when there is no more
        //output from the command
        currentLine = this.InnerProcess.StandardOutput.ReadLine();
    }
    return output.ToString();
}

Because of all the subterfuge required to get cmd.exe to cooperate, it's tempting to try to implement your own piping functionality. To accomplish this, I created two new classes. The first class, Pipe, connects two Processes. It continuously reads the output from the "left-side" Process and writes it to the input of the "right-side" Process. Because the Read operation will block as it waits for more input, the read/write loop executes on a separate thread. The following example connects the two commands via the Pipe class:

ConsoleProcess atProc = new ConsoleProcess("at.exe", "/?");
ConsoleProcess findProc = new ConsoleProcess("findstr.exe", "\/delete");
Pipe pipe = new Pipe(atProc, findProc);
atProc.Start();
findProc.Start();
pipe.Start();
string output = findProc.StandardOutput.ReadToEnd();
The complete code for the Pipe class is available as a part of the code download.

The second new class, Pipeline, automates some of these actions and allows an arbitrary number of Processes to be strung together. Internally, it keeps a list of Processes and Pipes. Each Pipe connects two of the Processes in the list. Processes are added to the Pipeline via the Add method. As you would expect, the Pipeline class exposes StandardInput and StandardOutput properties, which are just proxies for the StandardInput of the first Process and the StandardOutput of the last Process, respectively. The example I just showed could be recast using the Pipeline class as follows:

ConsoleProcess atProc = new ConsoleProcess("at.exe",
   "/?");
ConsoleProcess findProc = new ConsoleProcess("findstr.exe", "\/delete");
Pipeline pipeline = new Pipeline();
pipeline.Add(atProc);
pipeline.Add(findProc);
pipeline.Start(); //pipes are started automatically
string output = pipeline.StandardOutput.ReadToEnd();
The code for the Pipeline class is also available in the code download.

Figure 17 An RSS Reader
Figure 17 An RSS Reader

Either of these techniques (sending commands to cmd.exe or using the Pipe and Pipeline classes) could be used to hook the RSS pipeline into another .NET Framework-based application. The Pipe/Pipeline version is cleaner, but is slightly harder to implement. The cmd.exe version feels like a workaround, but you get all the features of cmd.exe for free. I built a Windows Forms user interface to the RSS pipeline that works both ways (shown in Figure 17 and included as a download), complete with support for clickable hyperlinks.

Conclusion

The reason I like console applications so much is that they reduce software to its bare essentials. Console applications only need to worry about four things: command-line arguments and the three standard data streams. The simplicity of this model looks like a severe limitation, but the pipe allows independent programs to behave symbiotically, enabling them to do more together than the sum of what they could do separately.

Implementing console applications with the .NET Framework is easy. With judicious use of some design patterns, it becomes easier still. The Process class makes accessing console applications from .NET straightforward, as long as you understand the limitations. Windows doesn't understand pipes; cmd.exe does. As such, to get pipe functionality in a Framework-based program, you need to either route all commands through cmd.exe or build your own piping capability. Both ways work, but each has trade-offs.

Console applications are currently an overlooked class of applications on Windows. This, however, is planned to change in the Longhorn timeframe. Microsoft is completely redesigning the console as a managed application for the next major release of its operating system. For more information on the Longhorn Command Shell (MSH), see the presentation from this year's PDC session ARC334, available at PDC 2003 Sessions.

For related articles see:
Design Patterns: Solidify Your C# Application Architecture with Design Patterns
Using command redirection operators

For background information see:
The Unix Programming Environment by Brian W. Kernighan and Rob Pike (Prentice-Hall, 1984)
Design Patterns by Gamma, Helm, Johnson, and Vlissides (Addison-Wesley, 1995)

Michael Brook is an independent consultant based in Pasadena, CA. He spends his time mining the depths of the .NET Class Library in search of the simplest solutions. He can be reached at horst@alumni.princeton.edu.


Timers: Comparing the Timer Classes in the .NET Framework Class Library

Timers

Comparing the Timer Classes in the .NET Framework Class Library

Alex Calvo
This article assumes you're familiar with C#
Code download available at: TimersinNET.exe (126KB)
SUMMARY

Timers often play an important role in both client applications and server-based components (including Windows services). Writing effective timer-driven managed code requires a clear understanding of program flow and the subtleties of the .NET threading model. The .NET Framework Class Library provides three different timer classes: System.Windows.Forms.Timer, System.Timers.Timer, and System.Threading.Timer. Each of these classes has been designed and optimized for use in different situations. This article examines the three timer classes and helps you gain an understanding of how and when each class should be used.

Contents
System.Windows.Forms.Timer
System.Timers.Timer
System.Threading.Timer
Thread-safe Programming with Timers
Dealing with Timer Event Reentrance
Conclusion

T
imer objects in Microsoft® Windows® allow you to control when actions take place. Some of the most common uses of timers are to start a process at a regularly scheduled time, to set intervals between events, and to maintain consistent animation speeds (regardless of processor speed) when working with graphics. In the past, timers have even allowed developers who use Visual Basic® to simulate multitasking.

As you would expect, the Microsoft .NET Framework arms you with the tools you need to tackle each of these scenarios. There are three different timer classes in the .NET Framework Class Library: System.Windows.Forms.Timer, System.Timers.Timer, and System.Threading.Timer. The first two classes appear in the Visual Studio® .NET toolbox window, allowing you to drag and drop both of these timer controls directly onto a Windows Forms designer or a component class designer. If you're not careful, this is where trouble can begin.

The Visual Studio .NET toolbox has a timer control on both the Windows Forms tab and the Components tab (see Figure 1). It is very easy to use the wrong one, or worse yet, to not even realize that they are different. Use the timer control that is located under the Windows Forms tab only if the target is a Windows Forms designer. This control will place an instance of the System.Windows.Forms.Timer class on your form. Like all other controls in the toolbox, you can either let Visual Studio .NET handle the plumbing or you can manually instantiate and initialize the class yourself.

Figure 1 Timer Controls
Figure 1 Timer Controls

The timer control that is located on the Components tab can be safely used in any class. This control creates an instance of the System.Timers.Timer class. If you're using the Visual Studio .NET toolbox, you can safely use this timer with either the Windows Forms designer or the component class designer. The component class designer is used by Visual Studio .NET when you're working on a class that derives from System.ComponentModel.Component (as is the case when you're working with Windows services). The System.Threading.Timer class does not appear on the Visual Studio .NET toolbox window. It is somewhat more complicated but provides a higher level of control, as you will see later in this article.

Figure 2 Sample Application
Figure 2 Sample Application

Let's first examine the System.Windows.Forms.Timer and System.Timers.Timer classes. These two classes have a very similar object model. Later I'll explore the more advanced System.Threading.Timer class. Figure 2 shows a screenshot of the sample application that I will be referring to throughout this article. This application will help you gain a clear understanding of each of the timer classes. You can download the complete code from the link at the top of this article and experiment with it.

System.Windows.Forms.Timer

If you're looking for a metronome, you've come to the wrong place. The timer events raised by this timer class are synchronous with respect to the rest of the code in your Windows Forms app. This means that application code that is executing will never be preempted by an instance of this timer class (assuming you don't call Application.DoEvents). Just like the rest of the code in a typical Windows Forms application, any code that resides inside a timer event handler (for this type of timer class) is executed using the application's UI thread. During idle time, the UI thread is also responsible for processing all messages in the application's Windows message queue. This includes Windows API messages as well as the Tick events raised by this timer class. The UI thread processes these messages whenever your application isn't busy doing something else.

If you wrote Visual Basic code prior to Visual Studio .NET, you probably know that in a Windows-based application the only way to allow the UI thread to respond to Windows messages while executing in an event handler is to call the Application.DoEvents method. Just like with Visual Basic, calling Application.DoEvents from the .NET Framework can cause many problems. Application.DoEvents yields control to the UI message pump, allowing all pending events to be processed. This can alter the expected path of execution that I just mentioned. If Application.DoEvents is called from your code, your program flow may be interrupted in order to process the timer events generated by an instance of this class. This can cause unexpected behaviors and make debugging difficult.

How this timer class behaves becomes evident when running the sample application. Clicking the application's Start button, then its Sleep button, and finally the Stop button, will yield the following output:

System.Windows.Forms.Timer Started @ 4:09:28 PM
--> Timer Event 1 @ 4:09:29 PM on Thread: UIThread
--> Timer Event 2 @ 4:09:30 PM on Thread: UIThread
--> Timer Event 3 @ 4:09:31 PM on Thread: UIThread
Sleeping for 5000 ms...
--> Timer Event 4 @ 4:09:36 PM on Thread: UIThread
System.Windows.Forms.Timer Stopped @ 4:09:37 PM

The sample application sets the Interval property of the System.Windows.Forms.Timer class to 1000 milliseconds. As you can see, if the timer event handler had continued to capture timer events while the main UI thread was sleeping (for five seconds), there would have been five timer events displayed as soon as the UI thread woke up again—one for each second the UI thread was sleeping. Instead, the timer remained in a suspended state while the UI thread was sleeping.

Programming the System.Windows.Forms.Timer class couldn't be easier—it has a very simple and intuitive programmatic interface. The Start and Stop methods essentially provide an alternate way of setting the Enabled property (which itself is a thin wrapper around Win32® SetTimer/ KillTimer functions). The Interval property, which I just mentioned, is self-explanatory. Even though you can technically set the Interval property as low as one millisecond, you should be aware that the .NET Framework documentation states that this property is only accurate to approximately 55 milliseconds (provided that the UI thread is available for processing).

Capturing the events raised by an instance of the System.Windows.Forms.Timer class is handled by wiring the Tick event to a standard EventHandler delegate, as shown by the code snippet in the following example:

System.Windows.Forms.Timer tmrWindowsFormsTimer = new
    System.Windows.Forms.Timer();
tmrWindowsFormsTimer.Interval = 1000;
tmrWindowsFormsTimer.Tick += new
    EventHandler(tmrWindowsFormsTimer_Tick);
tmrWindowsFormsTimer.Start();
•••
private void tmrWindowsFormsTimer_Tick(object sender,
    System.EventArgs e) {
  //Do something on the UI thread...
}

System.Timers.Timer

The .NET Framework documentation refers to the System.Timers.Timer class as a server-based timer that was designed and optimized for use in multithreaded environments. Instances of this timer class can be safely accessed from multiple threads. Unlike the System.Windows.Forms.Timer, the System.Timers.Timer class will, by default, call your timer event handler on a worker thread obtained from the common language runtime (CLR) thread pool. This means that the code inside your Elapsed event handler must conform to a golden rule of Win32 programming: an instance of a control should never be accessed from any thread other than the thread that was used to instantiate it.

The System.Timers.Timer class provides an easy way to deal with this dilemma—it exposes a public SynchronizingObject property. Setting this property to an instance of a Windows Form (or a control on a Windows Form) will ensure that the code in your Elapsed event handler runs on the same thread on which the SynchronizingObject was instantiated.

If you use the Visual Studio .NET toolbox, Visual Studio .NET will automatically set the SynchronizingObject property to the current form instance. At first it may seem as though using this timer class with the SynchronizingObject property makes it functionally equivalent to using the System.Windows.Forms.Timer. For the most part, it is. When the operating system notifies the System.Timers.Timer class that the enabled timer has elapsed, the timer uses the SynchronizingObject.Begin.Invoke method to execute the Elapsed event delegate on the thread on which the SynchronizingObject's underlying handle was created. The event handler will be blocked until the UI thread is able to process it. However, unlike with System.Windows.Forms.Timer, the event will still eventually be raised. As you saw in Figure 2, System.Windows.Forms.Timer will not raise events that occur while the UI thread is unable to process them, whereas System.Timers.Timer will queue them to be processes when the UI thread is available.

Figure 3 shows an example of how to use the SynchronizingObject property. You can use the sample application to analyze this class by selecting the System.Timers.Timer radio button and running through the same sequence of events as you did with the System.Windows.Forms.Timer. Doing so will produce the output shown in Figure 4.

Figure 4 Output (Close Figure 4)
System.Timers.Timer Started @ 5:15:01 PM
--> Timer Event 1 @ 5:15:02 PM on Thread: WorkerThread
--> Timer Event 2 @ 5:15:03 PM on Thread: WorkerThread
--> Timer Event 3 @ 5:15:04 PM on Thread: WorkerThread
Sleeping for 5000 ms...
--> Timer Event 4 @ 5:15:05 PM on Thread: WorkerThread
--> Timer Event 5 @ 5:15:06 PM on Thread: WorkerThread
--> Timer Event 6 @ 5:15:07 PM on Thread: WorkerThread
--> Timer Event 7 @ 5:15:08 PM on Thread: WorkerThread
--> Timer Event 8 @ 5:15:09 PM on Thread: WorkerThread
System.Timers.Timer Stopped @ 5:15:10 PM
Figure 3 Using the SynchronizingObject Property (Close Figure 3)
System.Timers.Timer tmrTimersTimer = new System.Timers.Timer();
tmrTimersTimer.Interval = 1000;
tmrTimersTimer.Elapsed += new
    ElapsedEventHandler(tmrTimersTimer_Elapsed);
tmrTimersTimer.SynchronizingObject = this; //Synchronize with
                                           //the current form...
tmrTimersTimer.Start();
•••
private void tmrTimersTimer_Elapsed(object sender,
    System.Timers.ElapsedEventArgs e) {
  // Do something on the UI thread (same thread the form was
  // created on)...
  // If we didn't set SynchronizingObject we would be on a
  // worker thread...
}

As you can see, it didn't skip a beat—even when the UI thread was sleeping. An Elapsed event handler was queued to be executed at each event interval. Because the UI thread was sleeping, however, the sample application displays the five timer events (4 through 8) all at once when the UI thread wakes up again and is able to process the handlers.

As I mentioned earlier, the System.Timers.Timer class members are very similar to those of the System.Windows.Forms.Timer class. The biggest difference is that System.Timers.Timer is a wrapper around Win32 waitable timer objects and raises an Elapsed event on a worker thread rather than a Tick event on the UI thread. The Elapsed event must be connected to an event handler that matches the ElapsedEventHandler delegate. The event handler receives an argument of type ElapsedEventArgs.

Above and beyond the standard EventArgs members, the ElapsedEventArgs class exposes a public SignalTime property, which contains the exact time the timer elapsed. Because this class supports access from different threads, it is conceivable that the Stop method may be called on a thread other than the thread that is used for the Elapsed event. This could potentially result in the Elapsed event firing even after the Stop method has been called. You can deal with this by comparing the SignalTime property to the time the Stop method was called.

The System.Timers.Timer class also provides an AutoReset property that determines if the Elapsed event should fire continuously or just once. Keep in mind that resetting the Interval property after the timer has started will reset the current count back to zero. For example, if the interval is set to 5 seconds and 3 seconds have already elapsed before the interval is changed to 10 seconds, the next timer event will be 13 seconds from the last timer event.

System.Threading.Timer

The third timer class comes from the System.Threading namespace. I'd like to say that this is the best of all timer classes, but that would be misleading. For one thing, I was surprised to find that instances of this class are not inherently thread safe, given that it resides in the System.Threading namespace. (Obviously, this doesn't mean it can't be used in a thread-safe manner.) The programmatic interface of this class is not consistent with the other two timer classes and it's also a bit more cumbersome.

Unlike the previous two timer classes that I have just covered, System.Threading.Timer has four overloaded constructors. The following shows what they look like:

public Timer(TimerCallback callback, object state, long dueTime,
             long period);
public Timer(TimerCallback callback, object state, UInt32 dueTime,
             UInt32 period);
public Timer(TimerCallback callback, object state, int dueTime,
             int period);
public Timer(TimerCallback callback, object state, TimeSpan dueTime,
             TimeSpan period);
The first parameter (callback) requires a TimerCallback delegate that points to a method with the following signature:
public void TimerCallback(object state);

The second parameter (state) can either be null or an object containing application-specific information. This state object is passed to your timer callback function during each timer event invocation. Keep in mind that the timer callback function is executed on a worker thread, so you'll want to ensure that you have thread-safe access to the state object.

The third parameter (dueTime) allows you to specify when the initial timer event should be fired. You can specify 0 to start the timer immediately or to prevent the timer from automatically starting, you can use the System.Threading.Timeout.Infinite constant.

The fourth parameter (period) allows you to specify the interval (in milliseconds) at which the callback function should be called. Specifying either 0 or Timeout.Infinite for this parameter will disable subsequent timer event invocations.

Once the constructor has been called, you can still alter the dueTime and period settings by using the Change method. This method has the following four overloads:

public bool Change(int dueTime, int period);
public bool Change(uint dueTime, uint period);
public bool Change(long dueTime, long period);
public bool Change(TimeSpan dueTime, TimeSpan period);
Here's the code I used in the sample application to start and stop this timer:
//Initialize the timer to not start automatically...
System.Threading.Timer tmrThreadingTimer = new
System.Threading.Timer(new
                       TimerCallback(tmrThreadingTimer_TimerCallback),
                       null, System.Threading.Timeout.Infinite, 1000);
//Manually start the timer...
tmrThreadingTimer.Change(0, 1000);
//Manually stop the timer...
tmrThreadingTimer.Change(Timeout.Infinite, Timeout.Infinite);

As you might expect, running the sample application with the System.Threading.Timer class selected results in the same output you just saw with the System.Timers.Timer class. Because the TimerCallback function gets called on a worker thread, there are no skipped beats (assuming the availability of worker threads). Figure 5 shows the output from the sample application.

Figure 5 Sample App Output (Close Figure 5)
System.Threading.Timer Started @ 7:17:11 AM
--> Timer Event 1 @ 7:17:12 AM on Thread: WorkerThread
--> Timer Event 2 @ 7:17:13 AM on Thread: WorkerThread
--> Timer Event 3 @ 7:17:14 AM on Thread: WorkerThread
Sleeping for 5000 ms...
--> Timer Event 4 @ 7:17:15 AM on Thread: WorkerThread
--> Timer Event 5 @ 7:17:16 AM on Thread: WorkerThread
--> Timer Event 6 @ 7:17:17 AM on Thread: WorkerThread
--> Timer Event 7 @ 7:17:18 AM on Thread: WorkerThread
--> Timer Event 8 @ 7:17:19 AM on Thread: WorkerThread
System.Threading.Timer Stopped @ 7:17:20 AM

Unlike the System.Timers.Timer class, there is no counterpart to the SynchronizingObject property that was offered by the System.Timers.Timer class. Any operations that require access to UI controls should be marshaled using the Control's Invoke or BeginInvoke methods.

Thread-safe Programming with Timers

To maximize code reuse, the sample application calls the same ShowTimerEventFired method from all three different types of timer events. Here are the three timer event handlers:

private void tmrWindowsFormsTimer_Tick(object sender,
    System.EventArgs e) {
    ShowTimerEventFired(DateTime.Now, GetThreadName());
}
private void tmrTimersTimer_Elapsed(object sender,
    System.Timers.ElapsedEventArgs e) {
    ShowTimerEventFired(DateTime.Now, GetThreadName());
}
private void tmrThreadingTimer_TimerCallback(object state) {
    ShowTimerEventFired(DateTime.Now, GetThreadName());
}
As you can see, the ShowTimerEventFired method takes the current time and current thread name as arguments. In order to distinguish worker threads from the UI thread, the main entry point for the sample application sets the Name property of the CurrentThread object to "UIThread." The GetThreadName helper method returns either the value of Thread.CurrentThread.Name or "WorkerThread" if the Thread.CurrentThread.IsThreadPoolThread property is true.

Because the timer events for System.Timers.Timer and System.Threading.Timer execute on worker threads, it is imperative that any user interface code within these event handlers be marshaled back onto the UI thread for processing. To do this, I created a delegate called ShowTimerEventFiredDelegate:

private delegate void
    ShowTimerEventFiredDelegate
    (DateTime eventTime,
    string threadName);
ShowTimerEventFiredDelegate allows the ShowTimerEventFired method to call itself back on the UI thread. Figure 6 shows the code that makes all this happen.

Figure 6 ShowTimerEventFired (Close Figure 6)
private void ShowTimerEventFired(DateTime eventTime,
    string threadName) {
    //InvokeRequired will be true when using
    //System.Threading.Timer or System.Timers.Timer (without a
    //SynchronizationObject)...
    if (lstTimerEvents.InvokeRequired) {
        //Marshal this call back to the UI thread (via the form
        //instance)...
        BeginInvoke(new
              ShowTimerEventFiredDelegate(ShowTimerEventFired),
              new object[] {eventTime, threadName});
    }
    else
        lstTimerEvents.TopIndex = lstTimerEvents.Items.Add(
            String.Format("—> Timer Event {0} @ {1} on Thread:
            {2}",
            ++_tickEventCounter, eventTime.ToLongTimeString(),
            threadName));
}

It's very easy to determine whether you can safely access a Windows Forms control from the current thread by querying its InvokeRequired property. In this example, if the ListBox's InvokeRequired property is true, the form's BeginInvoke method can be used to call the ShowTimerEventFired method again via ShowTimerEventFiredDelegate. This will ensure that the ListBox Add method executes on the UI thread.

As you can see, there are many issues you need to be aware of when programming with asynchronous timer events. I recommend that you read Ian Griffith's article "Windows Forms: Give Your .NET-based Application a Fast and Responsive UI with Multiple Threads," from the February 2003 issue of MSDN Magazine before using either System.Timers.Timer or System.Threading.Timer.

Dealing with Timer Event Reentrance

There's another subtle issue you'll need to consider when working with asynchronous timer events, such as the events generated by System.Timers.Timer and System.Threading.Timer. The problem has to do with code reentrance. If the code in your timer event handler takes longer to execute than the interval at which the timer is raising events, and you haven't taken the necessary precautions to guard against multithreaded access to your objects and variables, then you could be in for some difficult debugging sessions. Take a look at the following code snippet:

private int tickCounter = 0;

private void tmrTimersTimer_Elapsed(object sender,
    System.Timers.ElapsedEventArgs e) {
    System.Threading.Interlocked.Increment(ref tickCounter);
    Thread.Sleep(5000);
    MessageBox.Show(tickCounter.ToString());
}

Assuming your timer Interval property is set to 1000 milliseconds, you may be surprised to find out that the first message box that pops up will show a value of 5. This is because during the five seconds that the first timer event was sleeping, the timer kept on generating Elapsed events on different worker threads. Hence, the value of the tickCounter variable was incremented five times before the processing of the first timer event was completed. Notice how I used the Interlocked.Increment method to increment the tickCounter variable in a thread-safe manner. There are other ways to do this, but the Interlocked.Increment method was specifically designed for this kind of operation.

One easy way to solve this type of reentrance problem is to sandwich your timer event handler in a block of code that temporarily disables and then reenables the timer, as shown in the following example:

private void tmrTimersTimer_Elapsed(object sender,
    System.Timers.ElapsedEventArgs e) {
    tmrTimersTimer.Enabled = false;
    System.Threading.Interlocked.Increment(ref tickCounter);
    Thread.Sleep(5000);
    MessageBox.Show(tickCounter.ToString());
    tmrTimersTimer.Enabled = true;
}
With this code in place, message boxes will show up every five seconds and, as you would expect, the value of tickCounter will be incremented by one each time. Another option would be to use a synchronization primitive such as Monitor or a mutex to ensure that all future events are queued until the current handler has finished executing.

Conclusion

For a quick recap of my look at the three timer classes available in the .NET Framework, see the table in Figure 7 which compares these three classes. A point you may want to consider when working with timers is whether your problem can be solved more simply by using the Windows Scheduler (or the AT command for that matter) to run a standard executable periodically.

Figure 7 Timer Classes in the .NET FCL (Close Figure 7)

 | System.Windows.Forms | System.Timers | System.Threading
Timer event runs on what thread? | UI thread | UI or worker thread | Worker thread
Instances are thread safe? | No | Yes | No
Familiar/intuitive object model? | Yes | Yes | No
Requires Windows Forms? | Yes | No | No
Metronome-quality beat? | No | Yes* | Yes*
Timer event supports state object? | No | No | Yes
Initial timer event can be scheduled? | No | No | Yes
Class supports inheritance? | Yes | Yes | No
* Depending on the availability of system resources (for example, worker threads)

For related articles see:
Windows Forms: Give Your .NET-Based Application a Fast and Responsive UI with Multiple Threads

For background information see:
Programming the Thread Pool in the .NET Framework: Using Timers
.NET Framework Class Library

Alex Calvo is a Microsoft Certified Solutions Developer for .NET. When he's not reading, coding, or meditating, he's playing guitar. You can reach Alex at acalvo@hotmail.com.


Editor's Note: Yukon Bonanza

Editor's Note
Yukon Bonanza

Having endured 20 inches of snow in the past three days, there are a few things on our minds here. Warm beaches, swaying palm trees, and tropical drinks sipped slowly out of pineapples top the list. We're also trying to figure out where all that fluffy Manhattan snow magically goes by Monday morning. We had a foot and a half over the weekend, and all that remains on the street are a few slush puddles in the building crevices. Years ago, we'd have endless hedgerows of snow, taunting us for months, daring us to cross the street without being splashed by the crosstown bus. Is there some lot out in Canarsie that has a 50-story-tall mountain of gray slush sitting there, waiting until May to melt?

Instead of worrying about such globally important issues, the snow has brought us to our world-famous "lousy segue" as we look to the frozen north and Yukon. We're assuming you've looked at the cover of this issue, and have figured out that Yukon as we're using it is the code name for the next version of SQL Server, not the region of northwestern Canada. This issue contains our first coverage of some of the cool new features in Yukon.

SQL Server has always let you do some neat stuff. For instance, SQL Server 2000 included XML querying capabilities (FROM XML AUTO) that let you change the return type of a query into well-formed XML. This, in turn, led to some creative solutions, as developers could combine this XML with some clever stylesheets and come up with, say, GIS-based graphics. (We never get tired of pushing this page—SQL Server 2000: New XML Features Streamline Web-centric App Development.)

Very nice, if we do say so ourselves. Of course, the world of computing never stands still. The .NET Framework was announced and released after SQL Server 2000, and they've been first cousins until now. But Yukon changes everything. Arguably the biggest new feature of this release of SQL Server will be the integration of the CLR with the database engine itself. As you can imagine, this opens up a world of possibilities, the best of which is the ability for we application programmers to engineer a takeover of the world of database work! Are you with us here?

For too long, we've been sitting around waiting for some admin to set up our test database the way we want it. Oh no, they say, it's far too complicated for you to touch! But wait! Now the CLR is integrated into the database. We can design our own .NET-compliant data types. We can write stored procedures in C#. We can take over the world! And since we have a head start in obfuscating our C#, there's nothing they can do to read it once we get a full head of steam going!

Sorry—maybe we got a bit carried away there. Still, we are excited about Yukon because of all the enhancements planned for it. In this issue, we'll discuss some of the highlights of this new database programming platform. The major topics we'll present are the addition of CLR hosting (did we mention that already?), enhanced XML support (with an XML data type and XQuery support), and some major improvements to the T-SQL query language.

This issue is intended to give you a look ahead at Yukon. As the beta program and final release draw closer, we'll continue our coverage with topics relevant to developers. There are so many new features that it would be pointless to list all of them here. Lucky for us, there's now a Top 30 list of new features on Microsoft.com. It's an impressive list of features, even the weird-sounding databasey stuff that developers only hear about as they congregate around the Jolt cooler at work.

The product info page has lots of other resources if you're looking at Yukon for the first time. There are book chapters, guides, news and reviews, and at press time, there was even a 10-minute video demo of Yukon. We get the feeling this is going to be "must have" information in the years to come, so you should get ahead of the curve now.

Finally, we would like to officially introduce Stephen Toub, our new technical editor. Steve claims to be excited about joining the MSDN Magazine team. Most recently, he worked as a developer consultant with Microsoft Consulting Services, building enterprise apps for companies such as GE, McGraw-Hill, BankOne, and JetBlue. He was a developer for Microsoft Outlook as well as for the Microsoft Office Solution Accelerators. Prior to joining Microsoft, Steve taught computer science at Harvard, where he also received a degree in CS. Welcome!

—J.T.

Active Directory, IntelliSense, Microsoft, MSDN, MS-DOS, Visual Basic, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Server is a trademark of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Rapidly Build Installations
Multi-platform Database Administration
.NET E-mail Component
Tune Up 32-bit Windows Executables
Transport Legacy App Data Securely
Forums for ASP.NET Sites
Whois System Interface
Track Those Problems
Remove the Trash from PEs
Set Up a Testing Environment
The Bookshelf

Rapidly Build Installations

InstallShield Software Corporation has announced the availability of InstallShield Express 5, which allows you to rapidly build, test, and implement installations. Enhancements include the Project Assistant, which guides users through the installation-authoring process with a simplified set of views.

The upgrade also allows you to rebuild an installation without retrieving and compressing all the files again, and to see new redistributable technologies automatically with the option to download and include in your project. You can add an installation to a Microsoft® .NET Framework project by browsing to the project and running a wizard.

Migration of projects from previous versions of InstallShield Express is supported. InstallShield Express 5 can coexist with previous versions, allowing developers to work on new and previous projects at the same time. Express 5 now includes support for all 32-bit versions of Windows®, and has over 30 runtime languages so developers can broaden their deployment and tailor installations to specific regions.

http://www.installshield.com

Multi-platform Database Administration

AquaFold Inc. recently announced the latest version of Aqua Data Studio, a tool for building, managing, and maintaining enterprise relational databases. It includes support for all major database platforms and operating systems.

The new release introduces complete schema extraction and SQL scripting of database objects. The SQL DML for tables, views, stored procedures, and all database objects may be scripted to a query window. SQL history capabilities provide access to the history of all SQL statements and scripts executed. A new popup query window allows you to open multiple query windows side-by-side to compare scripts and query results.

Additional database support includes PostgreSQL 7.3 with new schema support and trusted connection support for SQL Server through an ODBC interface. Aqua Data Studio enables database administrators and developers to tackle multiple tasks simultaneously from one application. Its graphical browsing capabilities allow you to understand the structure and dependencies of the database schema, and to browse and select database scripts.

http://www.aquafold.com

.NET E-mail Component

ExclamationSoft Corporation has released Smtp.NET 3.0, which gives developers the ability to add e-mail functionality to an ASP.NET or Windows Forms application. Smtp.NET is a 100 percent managed code component that is written in C#.

Features include integration with ExclamationSoft's e-mail queuing application email Q.NET, which allows scheduled delivery of e-mail to manage server resources. It also includes built-in database features such as mail merge and personalization. Users can send a message containing both HTML and plain text, providing rich message formatting.

The component provides authentication and the ability to specify multiple mail servers for failover and redundancy. New features in Smtp.NET 3.0 include automated Zip archive compression of attachments and examples with help links that allow developers to click on keywords that lead directly to product documentation.

Smtp.NET supports Windows 98, Windows NT®, Windows 2000, Windows Me, and Windows XP with the .NET Framework installed.

http://www.exclamationsoft.com

Tune Up 32-bit Windows Executables

Heaventools Software has released Resource Tuner 1.94, an update of its advanced visual resource editor for 32-bit Windows executable files. Resource Tuner offers a thorough look at all of the resources (bitmaps, icons, strings, and so on) in the file, and allows you to make modifications without having to recompile the source code.

It also enables you to edit the properties of any controls on Delphi forms within the PE file. Resource Tuner allows legacy applications to take advantage of the new look for common control styles on Windows XP by enabling you to inject the application manifest directly into the EXE file.

Resource Tuner ships with UPX Unpacker, a start-up processing plug-in for unpacking files compressed with UPX. You can browse, view, extract, modify, translate, edit, or replace any selected executable file resources contained within 32-bit Windows executable files. Version 1.94 now supports editing accelerators and opening files with modified MS-DOS® stubs, and saves charset settings when customizing fonts. It includes additional startup processing options and supports all versions of UPX from 0.8 up to 1.90.

http://www.heaventools.com

Transport Legacy App Data Securely

HiT Software has announced a new product family, SafeConduct, which brings the benefits of Secure Sockets Layer (SSL) 3.0, including authentication and 128-bit data encryption, to any point-to-point application data traffic.

When applications move sensitive data across the Internet or Virtual Private Networks without authentication or data encryption, they are susceptible to unauthorized access or tampering, thereby creating information privacy and quality risks. SafeConduct can remove these risks by adding SSL security to shield sensitive data.

SafeConduct can be used with a variety of application architectures including client-server, host-slave, or multitier.

The SafeConduct product family includes the SafeConduct Windows Client, the SafeConduct Java Client, and the SafeConduct Server.

The SafeConduct Windows Client runs on Windows 98 and later client and server platforms as an application or as a service. The SafeConduct Java Client can be used on multiple client and server platforms including, but not limited to, Windows OS, OS/400, AIX, Linux, Solaris, Mac OS X, and OS/2. The SafeConduct Server similarly can be used on multiple client and server platforms.

http://www.hitsw.com

Forums for ASP.NET Sites

POP World Media LLC recently released CliqueSite POP Forums 7.0, forum software for ASP.NET Web sites. The latest version takes advantage of SQL Server and the ASP.NET Web application platform. Its discreet separation of logic and presentation makes the forum modifiable and helps you avoid writing database and data caching code.

The documentation and code samples show how a developer using .NET can use the included UI or build an entire site with POP Forums as the core engine for role-based member management. POP Forums ships with a UI that includes features that Web developers and their communities have come to expect, including user profiles, member photos, avatars, private messaging, and moderator functions.

http://popforums.cliquesite.com

Whois System Interface

Hexillion Technologies has announced the release of RegistryFusion 1.0, a set of Windows-based tools that provide a programmable interface to the Whois system. The software automatically finds the correct Whois server for a query and parses the output, enabling new Whois applications such as automatic domain registration tracking, security log analysis, and fraud detection.

The Whois protocol is the de facto standard means of obtaining contact information for domain names, IP addresses, and other network resources. Software applications have trouble accessing this information, however, because there are hundreds of scattered Whois servers, each returning a different output format that was not intended for machine consumption. Hexillion's RegistryFusion tools shield the user from the complexities and inconsistencies of Whois by tracking available Whois servers and parsing their various output formats. The user can specify a domain name or IP address and get results in a machine-readable format.

The software comes in three forms: a Web service, an XML proxy server, and a .NET Framework class library. The RegistryFusion tools require the .NET Framework 1.0 or 1.1 and Windows NT 4.0 SP6a, Windows 2000, Windows XP, or Windows Server™ 2003. The .NET Framework class library will also work on Windows 98 and Windows Me.

http://www.Hexillion.com/whois

Track Those Problems

NetResults is shipping ProblemTracker 5.5, along with support and help desk add-on options for this issue-tracking software. ProblemTracker is a Web-based collaboration tool that helps companies keep track of issues and automatically manage them to resolution. Its applications include bug, defect, and issue tracking; workflow management; help desk; and self-service automated support.

AutoReport reports user computer configuration information when an issue is submitted to ProblemTracker 5.5. This lets you automatically collect computer system information from non-technical users. In addition, ProblemTracker 5.5 offers several new add-on options for support, help desk, and integration. The Knowledge Base feature allows you to create a searchable knowledge base. The Submit via Email feature allows users to report issues via e-mail in addition to using the Web, and OpenAPI allows you to integrate your application with ProblemTracker. OpenAPI is based on SOAP and XML technologies.

http://www.problemtracker.com

Remove the Trash from PEs

PE Corrector, published by Gigamind Systems, corrects Windows Portable Executable (PE) files (EXE, DLL, SYS, BPL, DPL, CPL, OCX, ACM, AX, and SCR), removing extraneous information from the final file version. This information may include headers, strings with illegal characters, and export debugging functions.

Many compilers insert this binary trash into the compiled file, which significantly increases its final size. More dangerously, it gives hackers direct clues about how your program works and the easiest way to bypass its defense.

PE Corrector removes unnecessary debugging functions, clears binary trash in headers, organizes internal file structure, erases section names (.rsrc, .reloc, .pdata, and so on), and removes relocation tables.

In addition, PE Corrector allows programmers to insert their own data into the header (copyright information, for example), and examine the list of the export functions. The program can prepare Portable Executable files for EXE packers and reduce their size.

http://www.gigamindsystems.com

Set Up a Testing Environment

Software Prototype Technologies has introduced Direct-to-Test (DTT), a software testing environment that combines test-case design and automation into an integrated process. DTT automatically designs a complete suite of functional test cases from a model of the system requirements, generates all test documentation, and creates ready-to-run automated test script files. DTT eliminates the need for ad hoc specification of test requirements, the manual construction of test specifications and, in most cases, manual execution of the tests.

Direct-to-Test integrates several tools, including the RBT test design tool from Bender RBT and the TMX scripting tool from ScriptTech, plus Software Prototype Technologies' own test modeling and specification technology. The customer sees a single integrated test workbench rather than having to assemble and learn to use a variety of tools.

Direct-to-Test supports automated scripting for most of the major automated execution tools such as Seque Silk, Mercury Interactive's WinRunner, and generic XML and Unix scripts.

http://www.softprot.com

The Bookshelf

Addison-Wesley recently released ASP.NET SolutionsChar name="#151;"/>23 Case Studies: Best Practices for Developers by Rick Leinecker. This book explains how to take advantage of ASP.NET features using detailed, real-world case studies. Each case study illustrates one aspect of ASP.NET by building an application based on that technology. Extensive sample code can be used in the reader's own development projects.

Addison-Wesley has also published Practical Software Engineering: Analysis and Design for the .NET Platform by Enricos Manassis, which walks through a case study that specifies, analyzes, designs, implements, and tests a sample software system using the .NET Framework. The book presents an integrated vision of three dimensions in software development: process, techniques, and technology. Manassis provides a practical approach to software engineering, exemplified for the .NET Framework. The book covers UML, Rational XDE, Rational Unified Process, and Visual Studio .NET.

A First Look at ASP.NET v. 2.0 and A First Look at ADO.NET and System.Xml v. 2.0, by Alex Homer and Dave Sussman, are part of the Addison-Wesley Microsoft .NET Development Series. A First Look at ASP.NET v. 2.0, was written with Rob Howard, Program Manager of the Microsoft ASP.NET team. It offers readers a practical first look at changes to the new version of ASP.NET. The goal of the book is to enable programmers to develop applications now which will be ready for the next version of ASP.NET.

A First Look at ADO.NET and System.Xml v. 2.0 was written with Mark Fussell, a major contributor to the design and development of ADO.NET and System.Xml classes. It guides you through the functionality that will be available with the new generation of products, and provides a first look at changes to the new versions of ADO.NET and System.Xml.

http://www.awprofessional.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: ANSI Chars in XML, E-commerce Architecture, and More

Web Q&A
ANSI Chars in XML, E-commerce Architecture, and More
Edited by Nancy Michell

Q I am writing some code that writes strings from Registry keys, value names, and value data strings into an XML document that I operate on programmatically and sometimes display in an HTML report. I also sometimes need to regenerate the original data string from the XML document. My problem is that there are valid ANSI characters such as 0x07 (BEL) that appear in these Registry strings which are not valid XML characters and, according to the specification, are not valid to escape.

I am considering using a Base64 encoding of these strings, but this makes rendering the data in an XML page difficult. How can I automatically decode and display the Base64 string with the invalid character?

Another approach would be to use the rules for XML escaping to double encode these strings using "&x07;" notation. However, I then need to write my XML encoder (because it seems that these characters are invalid, so a standard encoder may not consistently encode them as I expect) and build a decoder that I can run on the strings twice to convert the final escaped sequence into the string as needed. However, both of these solutions seem like a hack to me. There must be enough of these legacy strings around that others have faced this problem as well.

A Escaping the characters would still be incorrect; they are disallowed for well-formed XML regardless of whether they are escaped or not. The only correct way to store bytes like 0x07 in XML is to use BinHex or Base64.

On the other hand, you are correct that this is a relatively common problem. SQL Server™ varchar columns, for example, and messages in Microsoft® Exchange Server both allow such characters. Support for these characters has been proposed for the XML 1.1 specification, but you should not make plans based on that. A workaround would be to use your own escaping mechanism (not XML's "&#007;"). Of course, you could write the characters out using XmlTextWriter and the invalid XML will load in Microsoft Internet Explorer (which supports these invalid characters for legacy reasons), but this is strongly discouraged because your XML will break in most other parsers.

Q My company sells tickets for events. Popular events produce heavy traffic on our ASP.NET system. We have six Web servers load balanced using hardware and an Oracle back-end database. During heavy load on the system, the Oracle Query/Lock manager pegs the memory and processor. Through testing, I found that if I can use a separate queue, I can alleviate the immediate problem.

I am considering the following architecture to solve the problem. First, I want to create two local Microsoft Message Queue Server (MSMQ) queues on each of the six Web servers. Let's call these queues ProcessQueue and ResponseQueue. Each Web server will have a custom .NET service. When a request comes in, the ASP.NET page will put a message in the ProcessQueue and then get into a loop to continuously read the ResponseQueue until a message is found or a predefined timeout occurs.

The service will read the message (which will be comprised of the data I need to run the stored procedure in Oracle) and call the Oracle stored procedure. This will be a synchronous process. So from the perspective of the Oracle database there will be only five requests in the queue at any one point in time, thus minimizing the load on the database server. The service can be made smart enough to spawn off multiple threads, providing the ability to find a sweet spot where the database is neither overloaded nor underutilized. Once the stored procedure is completed, the service will put a message in ResponseQueue for the ASP.NET request to pick it up. Will this architecture solve the problem?

A Does the long-running transaction have to be synchronous to the user? In other words, for asynchronous tracking, you can just drop a message on the queue, give the user a tracking ID so that they can manually track the result, and then send them a success or failure e-mail, as many e-commerce sites such as Amazon.com and Barnes & Noble.com do.

In this scenario, you drop a message on the queue and return to the user immediately. The user gets a tracking ID. The service pulls the message off the queue and sends the request to the database when it has the opportunity. When it gets the success or failure result from the database, it sends an e-mail indicating so. If it was a failure, the user is prompted to return to the site, enter a new credit card number, choose different seats, and so on. If it was a success, the mail just gives the customer a final confirmation number, expected ship dates, a link to print up tickets, and so on.

Q Does Microsoft have any solution for securing my network from people who walk into the office, plug in a laptop, and boot on the network? I know about Remote Access with Windows Server™ 2003, but is there any way besides physical security to secure the internal network from visitors and customers just plugging in and using the network? I want to avoid viruses that come in via portable devices.

A If you implement 802.1x for wired networks, it's possible to secure your internal network. But this solution can be costly because you need network hardware that supports it. You could use 802.1x certificate-based authentication with RADIUS server, but if you do not have Active Directory® (autoenrollment) and Windows Server, you'll incur an additional cost as well as facing a programming challenge.

Without Active Directory fully deployed, you might want to create a Web site where users can request a certificate. Once they get this certificate, it will direct them to the RADIUS server at which point you'll do your mandatory checks and put them in quarantine. So, how do you force all clients to get a certificate? You can refuse to route their IP traffic based on either their machine access code (MAC) or current IP (MAC would be better since it does not change as often). They will then be forced to call your help desk where you can direct them to the correct procedure. The beauty of this solution is that it works for wireless, too.

You can require MAC address reservations for ports on switches. But using MAC address or IP address restrictions does not scale well and is not secure, as both are easily spoofed. Plus, using MAC filtering in the switches means that every time someone gets a new PC, another human will have to update the switch table.

802.1x or IPSec are the recommended technologies to use here. 802.1x requires Windows® XP on all clients as well as a public key infrastructure (PKI) and RADIUS. IPSec works with Windows 2000 clients and doesn't require RADIUS, but it does require a PKI (please don't use pre-shared keys).

In addition, if you want to use a technology like Enterasys, which allows for blocking devices at the port level, you should go to http://www.enterasys.com/solutions/secure-networks to take a look at their User Personalized Network (UPN) offering.

There's also a great article on this exact topic by the IPSec team at Microsoft which describes how to configure IPSec for Windows 2000 and Windows XP to help secure an internal corporate network server against network-based attacks from untrusted computers. This paper describes the security threats to and the benefits of using IPSec on an internal corporate network server and uses a scenario to describe the process of IPSec policy design for an internal corporate network. Although the focus of this paper is IPSec for Windows 2000 and Windows XP, it also provides information about IPSec functionality enhancements in Windows 2000 service packs and in the Windows Server 2003 family. Microsoft and Foundstone Strategic Strategy coauthored "Using Microsoft Windows IPSec to Help Secure an Internal Corporate Network Server.".

IPSec is a good solution, but if you are implementing 802.1x for your wireless networks anyway, then using it as a starting point to a wired solution may make sense.

Q I'm looking for documentation to help me design an offline scenario in a traditional three-tier application. If the network connection becomes unavailable, what are the best practices for continuing work until the connection becomes available again?

Here's some background. Currently, I use a centralized database on the back end. There are clients at many different locations and multiple clients at each location. We're using COM+ for the COM components in the middle tier, and ADO and ODBC.

A There's a good tutorial on this subject—"Northwind Unplugged: Build a Semi-Connected Application in Visual Basic .NET". It's a .NET-compatible application and will give you some good ideas on how to handle the "sometimes-offline" scenario. It shows how to build a semi-connected application using Visual Basic® .NET. The application bases its behavior on the connection state. When working offline, changes to data are cached, and later, when reconnected, the data is automatically synchronized with the server.

Q I need to install IIS and some of its components in an unattended mode on more than 30 machines running domain controllers in order to run Microsoft Software Update Services (SUS). Is there a way to perform the installation outside of the Add/Remove programs console?

A Create a file named iis.txt containing the code shown in Figure 1 . Then copy the file to drive A, and issue the following command from the command line while you are on A:

sysocmgr /i:%windir%\inf\sysoc.inf /u:a:\ iis.txt
You'll need to tweak the iis.txt file according to your environment and your particular needs.

Figure 1 iis.txt (Close Figure 1)
[Components]
Iis_common = On
Iis_ftp = Off
Iis_nntp = Off
Iis_webdav = Off
Iis_serversideincludes = Off
Iis_smtp = Off
iis_asp =Off
Iis_inetmgr = On
iis_www=on
iis_webadmin=off
TSWebClient=off
indexsrv_system=Off
Aspnet = On
rootautoupdate = Off
[InternetServer]
PathWWWRoot = E:\WebSite

Q I need to search all files and folders on a test machine and have it look for a certain file or file type. How can I do this in script?

A That's easy—take a look at Figure 2. Just replace the following line with whatever thing you want to do to the file based on its path:

WScript.Echo File.Path
This recursively descends the C disk—it would be easy to make it enumerate the local disks and recursively descend each one.

Figure 2 Recursive Descent (Close Figure 2)
Option Explicit
Dim FSO
Set FSO = CreateObject("Scripting.FileSystemObject")
DoDir FSO.GetFolder("c:\")
Sub DoDir(Folder)
    On Error Resume Next
    Dim File, SubFolder
    For Each File In Folder.Files
        WScript.Echo File.Path
    Next
    For Each SubFolder in Folder.SubFolders
        DoDir SubFolder
    Next
End Sub

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Joshua Allen, Rhett Attwood, Jerry Carlson, Samuel Cavaliere, Len Childers, Mark Cleary, Brad Daniels, Scott Deacon, Joseph Elalouf, Murat Erenturk, Les Gainous, Denis Hawkins, Chris Hollander, Larry Kuhn, Eric Lippert, Maurizio Macagno, Richard Macdonald, Steve Riley (SBU), Taufiq Shamim, Jamie Sharp, Robert Steele, Chad Verbowski


Cutting Edge: Owner-Drawing in .NET

Cutting Edge
Owner-Drawing in .NET
Dino Esposito

Code download available at: CuttingEdge0402.exe (182KB)
Contents
Customizing Menu Rendering
Overriding the Menu of a Form
The MeasureItem Event
The DrawItem Event
Painting the Menu Item
Context Menu and TextBoxes
Using Graphical Menus Seamlessly

Each new major release of a Microsoft product like Office or Visual Studio® is characterized, among other things, by a new menu style. As new menu styles make their way into products, third-party vendors begin to emulate them with a new wave of custom controls and components. If you're taking advantage of one of these products, you only have to upgrade to the new release. Otherwise, your applications will keep on showing the classic menu user interface that was last updated when Windows® 95 shipped almost a decade ago.

While Microsoft periodically refreshes the menu style in key applications, the menu API available to custom applications hasn't changed much since Windows 3.x and File Manager. Since Win16, applications have been able to customize their menu by claiming the right to paint it personally, one item after the next. This technique is known as owner-drawing and applies to a number of other system components and controls, including listboxes, buttons, and comboboxes.

Figure 1 Owner-drawn Controls
Figure 1 Owner-drawn Controls

Customizing the drawing of controls is a rather boring task in Win16 and Win32®. It's not particularly complex in terms of functionality, but it's a little annoying to write and maintain. In this column, I'll delve into the owner-drawing mechanism exposed by the Microsoft® .NET Framework for window menus. The ultimate goal is to create a custom component that, once dropped onto the component tray of a Windows Form, allows you to customize the appearance of menus according to a given theme. As an example, I'll use a mix of the Visual Studio .NET and Office 2003 menu styles. Figure 1 gives you an idea of what you can accomplish once you understand owner-drawn controls in .NET.

Customizing Menu Rendering

In the .NET Framework, the classes involved with a window menu are MainMenu, ContextMenu, and MenuItem. All classes descend from a common base class—Menu. The top-level menu of a form is always an instance of the MainMenu class. The application's menu contains a list of popup menus, each composed of items and submenus. The ContextMenu class represents the context menu that any active visual object in the application can display. The context menu is an individual submenu and is in no way different from any of the submenus displayed by the top-level menu. Finally, a MenuItem object represents an individual item that is displayed within a MainMenu or ContextMenu object. The upshot is that any menu in the .NET Framework is a collection of MenuItem objects—no matter where or how it is displayed.

The underlying Win32 owner-draw functionality is conveniently exposed through a handful of properties and events on the MenuItem class. If you're an experienced Win32 programmer, you can appreciate the simplicity and the effectiveness of the .NET owner-draw programming model. If you've never worked with Win32 owner-drawn controls, you can't imagine how painful it really was. To turn on owner-draw for menus in .NET, you simply set the OwnerDraw property to true for each MenuItem object found in the MenuItems property. In addition, each MenuItem object can handle a couple of events, DrawItem and MeasureItem, that collect information for the underlying Win32 platform. The following code in Visual Basic® .NET shows what's needed to turn a menu item into an owner-drawn menu item.

Sub MakeItemOwnerDraw(ByVal item As MenuItem)
  item.OwnerDraw = True

  AddHandler item.DrawItem, AddressOf StdDrawItem
  AddHandler item.MeasureItem, AddressOf StdMeasureItem
End Sub

The MeasureItem event occurs when the menu needs to know the size of a menu item before drawing it. The DrawItem event occurs when the menu needs to paint a particular item. Knowing the size of each contained item, Windows can make the menu large enough to accommodate the largest menu item. Typically, the MeasureItem event handler calculates and returns the size of the menu item text given the size of the menu font. The DrawItem handler paints some text onto the provided Graphics object, plus a bitmap, a dithered background, or whatever else you might want to render on the menu.

Overriding the Menu of a Form

The steps to override the menu of a form to make it look more colorful and appealing are straightforward: just set the OwnerDraw attribute of each MenuItem to true and write proper handlers for key events. What's the best way to implement this? Should you inherit a new class from Menu or from MainMenu? Or is it preferable to create a brand new component class that works side by side with the default set of menu objects?

As I see things, the simplest thing to do is create a new external class—I'll call it GraphicMenu—that contains an initialization method. This method takes a reference to each menu you want to customize and turns on the owner-draw flag for each child item. The drawback of this approach is that an explicit call to the initializer is always needed. Perhaps more importantly, you won't get any support from the Visual Studio .NET designer. As a result, the menu only has an owner-drawn appearance at run time. In this column, I'll fully describe this approach.

Like Menu, the GraphicMenu class inherits Component:

Public Class GraphicMenu : Inherits Component
This means that it is docked in the designer's component tray at design time (see Figure 2). The GraphicMenu class acts as a transformer of existing menus—both the application's main menus and context menus. You add an instance of the GraphicMenu class to the form's set of controls and components and the form uses the class to transform regular menus into owner-drawn menus.

Figure 2 Visual Basic .NET Designer
Figure 2 Visual Basic .NET Designer

The code in Figure 3 shows the entry point method of GraphicMenu. The method Init transforms a gray, 3D, text-only standard menu into something more colorful. The method accepts a Menu object and configures each of its menu items to be owner-drawn. If the menu is a context menu—that is, if the type of the object is ContextMenu—then the owner-draw loop operates directly on it.

Figure 3 Graphic Menu Class (Close Figure 3)
Public Class GraphicMenu : Inherits Component

  ' Appearance Properties
  ' TODO:: List properties

  ' Behavior Properties
  ' TODO:: List properties

  Public Sub Init(ByVal menu As Menu)

     ' Return if the menu is null
     If menu Is Nothing Then
          Return
     End If

     ' Initialize the font object used to render the menu items
     ItemFont = New Font(FontName, FontSize)

     ' Initialize the hashtable used to hold bitmap/item bindings
     If menuItemIconCollection Is Nothing Then
         menuItemIconCollection = New Hashtable
     End If

     ' Context menu requires a different treatment
     If TypeOf menu Is ContextMenu Then
         HandleChildMenuItems(menu)
         Return
     End If

     ' Iterate on all top-level menus and handle their items
     For Each popup As MenuItem In menu.MenuItems
         HandleChildMenuItems(popup)
     Next

  End Sub

End Class

Why should you add more graphics to your menus? Well, other than improving the appearance of the application, one of the main reasons is to add explanatory icons alongside menu items. This means that some extra information must be added to each MenuItem in order to store the icon to be drawn next to the item. Ideally, you would replace the MenuItem class with a new derived class, say GraphicMenuItem, that looks like the parent but shows off a new Icon property. This approach works great if you always generate or manipulate the menu programmatically. Menus of many Visual Studio .NET-based applications are generated using the menu designer (see Figure 4), which automatically creates and registers MenuItem objects.

Figure 4 Menu Designer in Visual Studio .NET
Figure 4 Menu Designer in Visual Studio .NET

To avoid writing a brand new visual designer and tweaking the auto-generated Visual Studio .NET code, I decided to take another route. The association between icons and menu items is stored within the GraphicMenu class in an internal hash table. The hash table is populated with an AddIcon method:

extendedMenu.AddIcon(FilePrint, "..\images\print.bmp")
extendedMenu.AddIcon(FileNew, "..\images\new.bmp")
extendedMenu.AddIcon(FileOpen, "..\images\open.bmp")
extendedMenu.AddIcon(FileSave, "..\images\save.bmp")
AddIcon takes two parameters—the menu item object and the path to a small bitmap (or any other image file in a format supported by the System.Drawing.Image class). In the hash table, the instance of the menu item is the key and the icon file name is the value. While this might be less elegant from a theoretical point of view, this solution has the advantage of letting you work with your usual tools (Visual Studio .NET) in the traditional way. You just call an extra method to run some background code that reconfigures the menu for you.

The previous code assumes that the menu bitmaps are stored in external files—much like a Web app. If you plan to use images embedded in the app's assembly, you can overload the AddIcon method and make it accept an Image object instead of a file path:

Sub AddIcon(ByVal item As MenuItem, ByVal icon As Image)
The application would first extract the image from the assembly and then add it to the hash table.

The MeasureItem Event

Items marked as owner-draw fire a pair of events to allow for custom rendering. The first event is bound to the Win32 WM_MEASUREITEM message. When the form's window receives this message, it bubbles up a MeasureItem event to any owner-drawn MenuItem object. The event delegate is a class named MeasureItemEventHandler. This code demonstrates its prototype:

Sub StdMeasureItem(ByVal sender As Object, ByVal e As MeasureItemEventArgs)

Information about the event is stored in an instance of the MeasureItemEventArgs class passed to the event handler. Figure 5 lists and describes the properties of this class.

Figure 5 MeasureItemEventArgs (Close Figure 5)

Property | Description
Graphics | Gets the Graphics object to measure against
Index | Gets or sets the index of the item for which the height and width is needed
ItemHeight | Gets or sets the height of the item
ItemWidth | Gets or sets the width of the item

The purpose of the MeasureItem event is to determine how much space is required by a menu item. In order to compute this, you need to know the details of the surface on which the item will be painted. As such, the event data provides the GDI+ Graphics object representing the drawing surface on which the menu will be drawn. Typically, you compute the width and height based on the text that will be displayed and the desired font, which you get from the menu item. The MeasureItemEventArgs structure doesn't explicitly contain a reference to the MenuItem object, however you can easily grab it by casting the sender of the MeasureItem event to MenuItem. The code in Figure 6 shows how to do that. The text of the menu item plus any shortcut information (such as Ctrl+S) is measured against the current drawing surface and font using the MeasureString method. The resulting size is used to set the ItemWidth property. In the sample code in Figure 6, the height of the menu item is fixed for all items; this is the recommended approach. Note that the owner-draw mechanism is common to a few other controls like listbox and combobox, where items of different heights may sometimes be acceptable.

Figure 6 Define Size of Objects (Close Figure 6)
' ****************************************************************
' HELPER: StdMeasureItem
' INPUT : menu item, data used for measurement
' NOTES : Event handler for the MeasureItem event typical of
' owner-drawn objects
       Private Sub StdMeasureItem(ByVal sender As Object, _
           ByVal e As MeasureItemEventArgs)

    ' Grab a reference to the menu item being measured
    Dim item As MenuItem = CType(sender, MenuItem)

    ' If it is a separator, handle differently
    If (item.Text = "-") Then
        e.ItemHeight = SeparatorHeight
        Return
    End If

    ' Measure the item text with the current font. The text to
    ' measure includes keyboard shortcuts
    Dim stringSize As SizeF
    stringSize = e.Graphics.MeasureString(GetEffectiveText(item), ItemFont)

    ' Set the height and width of the item
    e.ItemHeight = MenuItemHeight
    e.ItemWidth = BitmapWidth + HorizontalTextOffset + _
        CInt(stringSize.Width) + RightOffset
End Sub

' ****************************************************************
' HELPER: StdDrawItem
' INPUT : menu item, ad hoc structure for custom drawing
' NOTES : Event handler for the DrawItem event typical of
' owner-drawn objects
Private Sub StdDrawItem(ByVal sender As Object, ByVal e As _
    DrawItemEventArgs)

    ' Grab a reference to the item being drawn
    Dim item As MenuItem = CType(sender, MenuItem)

    ' Saves helper objects for easier reference
    Dim g As Graphics = e.Graphics
    Dim bounds As RectangleF = MakeRectangleF(e.Bounds)
    Dim itemText As String = item.Text
    Dim itemState As DrawItemState = e.State

    ' Define bounding rectangles to use later
    CreateLayout(bounds)

    ' Draw the menu item background and text
    DrawBackground(g, itemState)

    ' Draw the bitmap leftmost area
    DrawBitmap(g, item, itemState)

    ' Draw the text
    DrawText(g, item)
End Sub

The largest menu item determines the size of the menu window. When the measurement phase is completed, the menu begins the actual rendering of the items in which each item is called to paint its own area. Each item can paint whatever it needs to, including text of various fonts, dithered backgrounds, and bitmaps. This step is handled by the DrawItem event, which I'll cover next.

The DrawItem Event

As mentioned, a menu item receives the DrawItem event when it is configured as owner-draw and a request is made for it to be drawn. The event is raised by the Win32 WM_DRAWITEM message and passes an instance of the DrawItemEventArgs class to all registered handlers. Figure 7 describes the members of this class and the information available to the menu item at rendering time.

Figure 7 DrawItemEventArgs (Close Figure 7)

Property | Description
BackColor | Gets the background color of the item being drawn
Bounds | Gets the rectangle that represents the bounds of the item being drawn
Font | Gets the font assigned to the item being drawn
ForeColor | Gets the foreground color of the item being drawn
Graphics | Gets the graphics surface to draw the item on
Index | Gets the index value of the item being drawn
State | Gets the state of the item being drawn

Three members of DrawItemEventArgs are particularly important: Graphics, Bounds, and State. Graphics represents the device context for any incoming GDI+ activity. All painting must take place on that Graphics object. The Bounds property is a Rectangle object that represents the drawing area whose width and height have been determined by the MeasureItem event. In addition, the Bounds rectangle provides the x,y coordinates of the item based on the height of previous items and the item's index. If you want to paint the background or to draw a bitmap or some text, this is the region in which it is safe to work.

Finally, the State property is a bit mask of DrawStateItem values and represents the state of the menu item. The DrawStateItem enumerated type contains binary flags denoting states like disabled, checked, hot-tracked, and selected. Each of these states influences the way a menu item is rendered. For example, a selected menu item would render with a different background color, and a disabled item would appear grayed out.

Things are a little more complex for items configured as check or radio items. A checked item is a menu item that represents a Boolean state rather than an action. When clicked, the application typically toggles an internal variable instead of or in addition to executing an action. Checked items are graphically rendered with a checkmark icon. Radio items render mutually exclusive options and are rendered with a bullet icon. Be aware that once you embark on the customization of a menu, you're totally responsible for consistently handling all of these situations.

Painting the Menu Item

The following code snippet shows a typical DrawItem event handler. The method receives the menu item and its related drawing parameters and goes through a pipeline of four steps.

Sub StdDrawItem(ByVal sender As Object, ByVal e As DrawItemEventArgs)
    Dim item As MenuItem = CType(sender, MenuItem)

    Dim g As Graphics = e.Graphics
    Dim itemState As DrawItemState = e.State

    CreateLayout(bounds)
    DrawBackground(g, itemState)
    DrawBitmap(g, item, itemState)
    DrawText(g, item, itemState)
End Sub

First, the code creates some helper rectangles used to draw the various parts of the graphical outline of each menu item. Next, it paints the background and then the bitmap. Finally, the text is rendered. Of course, the steps outlined here depend on the layout I've designed for the menu which, in this case, is similar to that of Visual Studio .NET, as illustrated in Figure 4.

My menu item consists of a bitmap area on the left and a text area on the right. The idea is to give each area different settings for foreground and background colors.

The DrawBackground method examines the state of the menu item and creates foreground and background brushes (optionally, gradient brushes). Next, it fills the bitmap area and the text area. The background colors of both areas are controlled through properties on GraphicMenu. In GDI+, you fill an area using a Brush object. Interestingly enough, the Brush object is the base class of the SolidBrush and LinearGradientBrush types so that having a solid or a dithered background for the menu requires no significant change to the code. The complete source used to paint a menu item is available in the code download (see the link at the top of this article), but I'll examine it in detail here.

The DrawBitmap method also looks at the state of the menu item. If the state requires the use of an embedded bitmap (such as a check or radio item), the necessary image is extracted from the control's assembly and rendered. Otherwise, the DrawBitmap method searches the internal hash table for an icon associated with the MenuItem, using the MenuItem as the key. If an image reference is found and the image exists, it is rendered.

You can embed images into a .NET assembly by adding the bitmap as an embedded resource. To do so, simply change the Build Action setting in the Properties box that appears when the bitmap is selected in the Solution box. (Note that Embedded Resource is not the default build action for a graphic, so this step is crucial to successfully using the image as a resource.) Likewise, it is important that you apply a particular naming convention to the image. The name of the image must be prefixed by the class namespace in Visual Basic .NET (if you're using C#, this works slightly differently). If the GraphicMenu class belongs to the MsdnMag namespace, any embedded image must be named MsdnMag.FileName.bmp. How can you retrieve these images at run time? Here's an example:

If item.RadioCheck Then
   bmp = ToolboxBitmapAttribute.GetImageFromResource( _
        Me.GetType(), "Bullet.bmp", False)
Else
   bmp = ToolboxBitmapAttribute.GetImageFromResource( _
        Me.GetType(), "Checkmark.bmp", False)
End If

You can use a shared member on the ToolboxBitmapAttribute class named GetImageFromResource. Don't be fooled by the name of the class—it is an attribute class defined in the System.Drawing assembly, but it can be used programmatically in a traditional way (as opposed to the attribute-based programming model). The GetImageFromResource method is just a helper method built around this code:

' t is the type whose namespace is used to scope
' the resource name (GraphicMenu in this context)
Dim img As Image
Dim str As Stream
str = t.Module.Assembly.GetManifestResourceStream(t, "Bullet.bmp");
If Not (str Is Nothing) Then
  img = new Bitmap(str)
End If

When applied to a control, the ToolboxBitmapAttribute attribute tells containers like the Visual Studio Form Designer to retrieve an icon that represents the control. The bitmap is normally embedded in the assembly that contains the control (but it doesn't have to be). The size of the menu bitmap is set by the programmer and should be the same for all bitmaps. In the sample code being discussed here, it is assumed to be 16 × 16.

The DrawBitmap method also manages to render any image transparently, irrespective of the native format, using the MakeTransparent method on the Bitmap class:

bmp.MakeTransparent()
The MakeTransparent method configures the Bitmap object to use the color of the first pixel in the image (0,0 coordinates) as the transparent color. By using a different overload of the method, you can define the transparency color. Finally, to render grayed bitmaps, you have two options. One is to use a totally different set of icons, much like toolbars which use separate image lists. I chose the second—render the image with a different gamma color so that the same image looks grayed out (see Figure 8).

Figure 8 Graying an Image (Close Figure 8)
If (disabled) Then
   Dim imageAttr As New ImageAttributes
   imageAttr.SetGamma(0.2F)
   Dim tmpRect = New Rectangle( BitmapBounds.X + 2, _
        BitmapBounds.Y + 2, BitmapBounds.Width - 2, _
        BitmapBounds.Right - 2)
   g.DrawImage(bmp, tmpRect, 0, 0, _
        bmp.Width, bmp.Height, GraphicsUnit.Pixel, imageAttr)
   ImageAttr.ClearGamma()
Else
   g.DrawImage(bmp, BitmapBounds.X + 2, BitmapBounds.Y + 2)
End If

When you render an image, you can control the color gamma correction for a few categories of graphic tools, including pens and brushes. You set and reset gamma correction using the ImageAttributes class, an instance of which can be passed to the DrawImage method of the Graphics class. By using a gamma value near zero, DrawImage pales colors out. As Figure 9 shows, this is a fairly good technique for automatically graying images (see the Print menu item). A better approach entails using a transformation matrix. (See the GDI+ SDK documentation for more information on this specific point.)

Figure 9 Greying Out the Print Menu Item
Figure 9 Greying Out the Print Menu Item

The final step consists of text rendering in the DrawText method. The text to paint can optionally include the textual representation of the shortcut—a string like Shift+Ctrl+F. If the Shortcut property on the menu item is set to true, the text to render is composed of two parts—the text and the shortcut. The code in the DrawText method uses an internal method—GetEffectiveText—to retrieve the item text and the shortcut. It is worth noting that the shortcut text is not immediately available in the typical display format—that is, Key+Key. Keyboard shortcuts are grouped in the Shortcut enum type, whose ToString method returns unseparated expressions such as "CtrlO." However, there's a common pattern in all of these expressions. Key names are camel-cased with the first letter uppercased and all the others lowercased. So if you insert a + separator whenever the occurrence of an uppercase character is found (except the first character), you can create the shortcut string you see in Figure 9.

Handling submenus is easy, regardless of the level of nesting. The trick is in the recursive algorithm I wrote to propagate the OwnerDraw setting in the GraphicMenu's Init method. No matter how many cascading popup menus you have, all of those items will correctly receive the handlers for the MeasureItem and DrawItem events. In addition, the MainMenu class supplies the glyph that indicates that an item has a child menu.

Context Menu and TextBoxes

Customizing a context menu involved the same code with one extension. Bear in mind that in the .NET Framework a context menu is just a main menu with exactly one child menu. The parent of a context menu, though, never appears. In light of this hierarchy, the code I built so far didn't require any changes to work.

A context menu can have a default item. When the user double-clicks a submenu that contains a default item, the default item is automatically selected. You can use the DefaultItem property on the MenuItem class to indicate the default action that is expected in a context menu. If not told otherwise, Windows renders default menu items in boldface. An owner-drawn menu must provide for this feature, too. The DrawItemState enumeration has a specific value—Default—that lets you know when you're rendering a default item. At that point, you just change the font in use and make it boldfaced. This code shows how:

Dim tmpFont As Font
Dim defaultItem As Boolean
defaultItem = (itemState And DrawItemState.Default)
If (defaultItem) Then
   tmpFont = New Font(ItemFont, FontStyle.Bold)
Else
   tmpFont = ItemFont
End If

The Font.Bold property is read-only, meaning that you can't add or remove the bold typeface from a font once it is created. For this reason, if you're going to render a default menu item, you must create a temporary font object (with the bold attribute on) and use that to render. Remember that it's good programming practice to dispose of font objects and other GDI+ objects when you're finished using them.

In a Windows Forms application, you create one or more context menus and bind them to individual controls using the ContextMenu property available on each control. The various context menu objects must be preprocessed using the GraphicMenu object and its Init method to ensure a customized look. Figure 10 shows an example. The button is bound to a context menu where Order is the default item.

Figure 10 Creating Context Menus
Figure 10 Creating Context Menus

The Textbox is the only control in Windows Forms that has a built-in context menu. The control exposes a ContextMenu property, but it doesn't return an instance of the context menu that appears when you right-click. Why? The code for the textbox's context menu is hardcoded in the Win32 API. In particular, the context menu is generated on the fly each time the user right-clicks. The menu is then released and destroyed once the selection is made. You can replace this menu with your own by setting the ContextMenu property, but there's no chance to manipulate the original menu. Since the menu has a very short (and unmanaged) life, how can you easily subclass it and mark it as owner-draw?

The simplest trick that I've found is the following: create a TextboxContextMenu class that represents a custom context menu with all of the items of a standard textbox context menu (Undo, Cut, Copy, and the like). Implementing these methods is relatively simple and samples can be found in the MSDN documentation. Once you hold an instance of a managed class that works like the typical context menu of a TextBox, you first make it owner-draw and then you bind it to any TextBox you want:

Dim txtMenu As New TextBoxContextMenu
gMenu.AddIcon(txtMenu.MenuItemCut, "..\images\cut.bmp")
gMenu.AddIcon(txtMenu.MenuItemCopy, "..\images\copy.bmp")
gMenu.AddIcon(txtMenu.MenuItemPaste, "..\images\paste.bmp")
gMenu.AddIcon(txtMenu.MenuItemDelete, "..\images\delete.bmp")
TextBox1.ContextMenu = txtMenu
gMenu.Init(TextBox1.ContextMenu)
The TextBoxContextMenu class lives in the same assembly that contains the GraphicMenu class. You should be aware that owner-drawn menus don't work correctly when attached to a NotifyIcon component. For more information on this problem see Knowledge Base article 827043.

Using Graphical Menus Seamlessly

The .NET Framework supports graphical menus using a programming interface that is the managed counterpart to the Win32 owner-draw mechanism. The managed API adds some abstraction, but doesn't redesign the underlying model. This means that a graphical menu can only be obtained with some custom code, as demonstrated in this column. Is there a way to make this process automatic so that no code must be written other than setting properties in the IDE? The code needed to set up GraphicMenu is minimal but necessary when the parent form loads. One way to hide it completely is to create a form class that contains an instance of the GraphicMenu class and in the Load event set up the main menu for owner-draw. If you add this class to the collection of inheritable forms, when you add a menu to the form it will automatically display with a custom look. So go ahead and give it a try.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Programming ASP.NET (Microsoft Press, 2003), he spends most of his time teaching classes on ADO.NET and ASP.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com.


The XML Files: XML Report from the Microsoft PDC 2003

The XML Files
XML Report from the Microsoft PDC 2003
Aaron Skonnard

Contents
Indigo
System.Xml in "Whidbey"
Whidbey XML Serialization
Visual Studio Whidbey
Whidbey Service-oriented App Designer
SQL Server Yukon
Longhorn's X Technology: XAML
XML in Action: PDC Blogging

This issue marks three years of The XML Files. As I reminisce over my personal XML journey and how simply it all began, I'm astonished at how firmly rooted XML is in today's software development landscape. I always said "XML will change the world," but only after reflection did I realize that it has actually happened.

XML was everywhere at the November 2003 PDC. There weren't many sessions that didn't at least mention XML or some other new "X" technology. Although Microsoft has always been a leader in XML, their innovative use of XML in future technologies highlights their commitment to interoperability. In this installment of The XML Files, I'll summarize the juicy XML activity at the PDC. You can find the PDC session slides at msdn.microsoft.com/events/pdc/agendaandsessions/sessions. Please note that much of the information in this month's column covers beta releases and, as such, is subject to change at any time.

Indigo

The next version of the Windows® operating system, code-named "Longhorn," comes with a new communications subsystem, code-named "Indigo." Indigo was the major XML-related topic at the show. Indigo is a set of Microsoft® .NET Framework technologies for building loosely coupled, connected systems. Although Indigo is a key ingredient of Longhorn, it will also run on Windows XP and Windows Server™ 2003.

The Indigo infrastructure is built around the advanced Web services architecture shown in Figure 1. The technologies that make up this architecture enable rich interoperability between systems, ranging from simple communications to more advanced features like security, reliability, and distributed transactions. With this foundation, Indigo is capable of providing solutions for many distributed application needs within a single unified programming model that's approachable by the masses.

Figure 1 Web Services Architecture
Figure 1 Web Services Architecture

Indigo represents the unification of ASP.NET Web Methods (.ASMX), .NET Remoting, and Enterprise Services. Indigo also brings along COM+ and Microsoft Message Queue Server (MSMQ), the most widely used .NET Framework distributed technologies, and understands their protocols natively. For more background and detailed information on how to use the Indigo components, check out Don Box's article "A Guide to Developing and Running Connected Systems with Indigo" in the January 2004 issue of MSDN® Magazine. The Web Services Developer Center can be found at http://msdn.microsoft.com/webservices.

To best position your applications for Indigo migration, Microsoft prescriptive guidance states that you should embrace ASMX, Enterprise Services, and a constrained use of .NET Remoting for cross-AppDomain or cross-process scenarios if you're looking for the smoothest transition possible. See The Road to Indigo Technology Roadmap for more information on the Indigo roadmap.

Overall, the Indigo unified programming model simplifies the developer experience and provides the flexibility required for a wide range of communications, from simple in-process scenarios, performance-sensitive intranet scenarios, to highly scalable Web services scenarios.

For more information, check out the Indigo section in the Longhorn SDK.

System.Xml in "Whidbey"

The new version of Visual Studio® and the .NET Framework, code-named "Whidbey," will include new System.Xml features. Most of the System.Xml improvements in Whidbey will be centered on performance or usability. In terms of performance, both XmlTextReader and XmlTextWriter are twice as fast as they were in version 1.1 implementations. XML Schema validation is a more modest 20 percent faster while the XSLT transformation engine blows away previous versions with a whopping 400 percent performance increase (Figure 2) comparing the time it takes to perform various tasks. These improvements are highly anticipated since performance speeds became one of the most common criticisms of the .NET Framework version 1.1.

Figure 2 Time to Complete Tasks
Figure 2 Time to Complete Tasks

In terms of usability improvements, common language runtime (CLR) type accessors are being added on XmlReader, XmlWriter, and XPathNavigator. These accessors will allow you to write the following convenient code, eliminating the need to coerce text to CLR types manually:

double weight = reader.ValueAsDouble;

Microsoft also introduced a new class called XmlFactory for instantiating XmlReader, XmlWriter, and XPathNavigator implementations based on user-defined settings:

XmlFactory factory = new XmlFactory();
factory.ReaderSettings.Schemas.Add    ("myschema.xsd");
factory.ReaderSettings.XsdValidate = true;
XmlReader r =
   factory.CreateReader("myinstance.xml");

Probably the biggest API change in version 2.0 is the shift away from XmlDocument, the .NET implementation of the W3C DOM Level 2 Recommendation. Although XmlDocument will still be present in the Whidbey release, the System.Xml team hopes that XPathDocument will supplant it as the primary in-memory XML store. Unlike the read-only version 1.1 implementation, XPathDocument will now provide editing capabilities through the XPathEditor class (see Figure 3).

Figure 3 XPathDocument
Figure 3 XPathDocument

XPathDocument provides many advantages over XmlDocument, including a 20 to 40 percent performance boost for XSLT operations, built-in XML Schema validation, support for strongly typed values (for example, xsd:integer stored as int), automatic change tracking at the node level (equivalent to what the ADO.NET DataSet provides), and data-binding support with Windows Forms and ASP.NET controls. You'll still use an XPathNavigator to read and query an XPathDocument in Whidbey, although a new class called XPathChangeNavigator has been introduced to access the changes made to an XPathDocument from its creation.

Another major addition to the library is XML Views, a mechanism for mapping between relational data and XML types. This makes it possible to view data stored in SQL Server™ in a specific XML format. This functionality is equivalent to what's provided by SQLXML 3.0 today, but with several enhancements. There's also a new class called XmlAdapter for connecting an XPathDocument to a database through an XML view (equivalent to the ADO.NET DataAdapter).

Whidbey also introduces support for the W3C XQuery language. Think of XQuery as SQL for XML. Its overall functionality is virtually equivalent to XSLT but most developers will probably have an easier time with its SQL-like syntax. As a result, Microsoft believes that XQuery will become the preferred query language when working with XML sources.

Since both XSLT and XQuery are built on XPath, they both require the same fundamental query engine. Hence, in Whidbey, System.Xml provides a common query engine for processing instructions from multiple query languages including XPath, XQuery, XSLT, or an XML view. Expressions from any of these languages compile down into an intermediate query language representation. The intermediate representation is then fed into the query engine, which generates Microsoft intermediate language (MSIL) code. Microsoft calls this new design the Common Query Architecture.

The Common Query Architecture provides a flexible layer of abstraction that allows the introduction of new query languages down the road. You would simply need to write a compiler to generate the intermediate language representation for the new language. The architecture also provides great performance improvements due to the fact that it generates MSIL code, which then compiles down to machine code before executing.

System.Xml.Query is a new namespace that encapsulates the classes making up this architecture. It includes a class named XQueryProcessor for compiling and processing XQuery expressions, and another class named XsltProcessor for compiling and processing XSLT transformations. Both of these classes share the same underlying query engine.

System.Xml has become one of the industry's favorite XML libraries since it first shipped with .NET version 1.0. The innovations introduced with System.Xml have improved the experience for XML developers (for example, pull-parsing is becoming commonplace, even in Java stacks). The improvements and additional functionality described here should raise the bar even higher.

Whidbey XML Serialization

Another way to program XML is to rely on serialization. Serialization is the process of transforming an object graph into a byte stream, while deserialization is the process of transforming the serialized byte stream back into the same object graph. Serialization also includes the process of transforming a type from one type system into a type in another type system (transforming a CLR type into an XSD type or the reverse, for example).

There are two different serialization engines available in .NET Framework 1.1. One provides XML-based serialization (System.Xml.Serialization), while the other provides CLR-based serialization (System.Runtime.Serialization). The former makes it possible to serialize objects or types in an interoperable manner, using XML and XSD, while the latter is useful for serializing CLR objects between .NET environments. This fundamental architecture remains the same in Whidbey, although they've made several improvements such as introducing support for generics and SqlTypes. The BinaryFormatter was also enhanced with version tolerance capabilities and performance improvements. The more significant changes, however, are slated for Indigo.

Moving towards Indigo has made two things clear when it comes to serialization: interoperability is a must, and there are different programming models where serialization comes into play. The interoperability requirement rules out the use of System.Runtime.Serialization as it exists today. There's an XML programming model where you start with XML Schema and use serialization to get back to the CLR. There's also a CLR programming model where you start with CLR types and use serialization to expose interoperable XSD contracts to the rest of the world.

Today System.Xml.Serialization supports both programming models. Supporting the XML programming model is challenging because there are so many aspects of XML Schema that don't map nicely to the CLR. Starting with CLR types is more straightforward because it's possible to define a standard XSD mapping for any CLR type. Catering to both programming models simultaneously, however, requires lowering the bar and restricting options. What's missing is a serialization layer that caters to a CLR programming model without sacrificing interoperability and loose coupling. Indigo introduces a new serialization engine accessible via XmlFormatter that fills this need. XmlFormatter can export all CLR types to XSD, providing support for a canonical mapping between the two. Once Indigo ships, XmlFormatter will likely become the serialization choice for most developers using .NET.

Visual Studio Whidbey

The XML editor that ships with Visual Studio .NET 2003 leaves most XML developers wanting more. This has been good news for ISVs because the void created a strong market for third-party tools. The Visual Studio .NET XML editor did a so-so job with IntelliSense® and offered little beyond that. The new, improved XML editor included in Whidbey represents a significant improvement.

First, and most importantly, the new XML editor does a much better job with IntelliSense and validation. I heard one happy developer proclaim that it's always right, even when namespaces come into play. IntelliSense can be driven by XML Schema definitions or Document Type Definitions (DTDs)—the original XML editor only provided support for XML Schema. The new editor also performs real-time validation against the target definition and provides traditional red squiggles along with helpful error messages to assist with detection and correction.

The other major new IDE feature is XSLT editing and debugging. The new editor provides complete XSLT IntelliSense as well as full-fledged interactive debugging support with language integration. Before, debugging support was only available through commercial third-party add-ons.

There are a few other usability improvements worth mentioning. One is XML outlining support, with expand and collapse functionality that makes it easier to navigate large XML files. Another is that you can select a piece of text and automatically wrap it in XML comments or CDATA sections. It also provides some nice reformatting features for automatically adjusting the style of your document. Overall, it appears that the new Whidbey XML editor will be much more enjoyable and productive to use.

Whidbey Service-oriented App Designer

At the 2003 PDC, Microsoft unveiled a new suite of Visual Studio tools for building service-oriented applications (SOA), code-named "Whitehorse." These tools have a simple objective: to make SOA development easier. The first version of Whitehorse is scheduled to ship with Whidbey and they'll continue to evolve into one of the major components in the subsequent version of Visual Studio, code-named "Orcas." The official name for Whitehorse is Visual Studio Whidbey Service-oriented Application Designer, but I'll simply refer to it as Whitehorse.

Whitehorse is intended to simplify the architecture, design, development, and deployment of applications that are composed from various distributed services. Whitehorse includes a distributed services designer for visualizing application architectures along with tools that cover data modeling, constraint validation, code generation, and reporting details. The tools provide friendly drag and drop-style environments that are intuitive for solution architects, abstracting away the details of lower-level code. These tools allow solution architects to direct the design without dealing with complex components like WSDL files directly. The generated code always stays in sync with the visual model and can be deployed to different environments.

Another major benefit is the support for matching application solutions with compatible deployment environments. The Whitehorse designers allow datacenter administrators and solution architects to work independently and simultaneously, each specifying their own constraints for their parts of the overall solution. Whitehorse can then ride in to help identify incompatibilities and assist in resolving conflicts early in the process.

The Whitehorse SOA designer was not included with the PDC Whidbey bits but Microsoft says it will be included in a future Whidbey beta. In the meantime, you can check out Roadmap to the Future for various Whitehorse articles including a roadmap to future Microsoft development tools (covers both Whidbey and Orcas, an overview to the Visual Studio Whidbey Service-oriented Application Designer, and a helpful FAQ). And for some Whitehorse screenshots, check out http://msdn.microsoft.com/vstudio/productinfo/roadmap.aspx.

SQL Server Yukon

Coverage of the next version of SQL Server, code-named "Yukon," was also one of the highlights of the show. One of the main Yukon features anticipated by developers is the expanded .NET language support. Yukon does this by hosting the CLR in the database engine, letting developers author stored procedures and user-defined types in any .NET-compatible language.

The other major new Yukon feature is richer XML support. In Yukon, XML is now a native datatype. This means that you can create a table with a column of type xml, as illustrated here:

CREATE TABLE BlogEntry(id int, entry xml)
In this case, entry is referred to as an untyped XML column. When you insert a value into an untyped XML column, the XML is checked for well-formedness before completing the operation. With untyped columns, the XML is stored in its native XML 1.0 format using the UCS-2 encoding. You can also create a typed XML column by associating it with an XML Schema definition. To do this, you must first register the schema through the CREATE XMLSCHEMA statement. Then you specify the schema mapping in the CREATE TABLE statement, as shown here:
CREATE XMLSCHEMA '<xsd:schema targetNamespace="http://example.org/blog"
...> ...'

CREATE TABLE BlogEntry(id int, entry xml ('http://example.org/blog'))

With the schema mapping in place, the database engine can perform schema validation on inserts or updates and it can optimize the way the XML is stored. Instead of storing it in XML 1.0 format, the database engine can create additional internal structures for storing the information in the XML document as native T-SQL types (such as varchar, int, and so on), which greatly improves performance and reduces storage overhead.

After you get the XML values into the database, there are various ways to get them back out. You can simply select the column if you want to retrieve the entire XML document. Yukon also provides native support for XQuery, which allows you to query XML columns for specific fragments or values:

SELECT entry::query('/item/author') FROM BlogEntry

In addition to the new XML datatype and XQuery support, Yukon also improves its support for Web services. Yukon can host Web services straight from the database using existing Web services standards including HTTP, XML Schema, SOAP, and WSDL.

Longhorn's X Technology: XAML

Longhorn also comes with a new presentation subsystem, code-named "Avalon," that relies heavily on XML. It provides a declarative programming model similar to what's used on the Web today. In fact, the easiest way to describe Avalon is to say that it brings together the best from the Web (for example, seamless deployment, flowable layout, and progressive download and rendering) and the best from Windows (unrestricted functionality, desktop integration, good offline support, performance, and so on).

The Avalon declarative programming model is based on a new XML vocabulary called Extensible Application Markup, code-named "XAML" (pronounced "zammel"). Writing an Avalon-based application starts with authoring a XAML file (typically through a designer) to define the layout of UI elements. Then you write the functionality of the application in procedural code, which you can embed in the XAML file itself or keep separate in a codebehind file (like in ASP.NET). When you build the application, the compiler reads the XAML markup and generates code using an equivalent object model (you can use the object model directly if you want). Avalon-based apps can run in a desktop window or be hosted in a browser. Figure 4 shows an example of a simple XAML document that contains a single button and a codebehind file. Figure 5 shows the codebehind file that simply changes the button's text to "Hello World" when it's pressed. For more information on Longhorn, Avalon, and XAML surf to http://longhorn.msdn.microsoft.com.

Figure 5 The CodeBehind (Close Figure 5)
namespace WCSample
{
    using System;
    using MSAvalon.Windows.Controls;

    public partial class MyPage
    {
        void HandleClick(object sender, ClickEventArgs e)
        {
            Button1.Content = "Hello World";
        }
    }
}
Figure 4 Simple XAML Doc (Close Figure 4)
<DockPanel xmlns="http://schemas.microsoft.com/2003/xaml"
            xmlns:def="Definition"
            def:CodeBehind="Pane1.xaml.cs"
            def:Class="WCSample.MyPage">

    <Button DockPanel.Dock="Top"
            Height="30px"
            Width="100px"
            Margin="10,10,10,10"
            Click="HandleClick"
            ID="Button1">Click Me</Button>
</DockPanel>

XML in Action: PDC Blogging

Over the past year, blogging has emerged as the latest XML trend as well as an effective form of communication among developers. There are several tools available today that make it easy to set up your own blog (.Text and dasBlog, for instance). Others can then subscribe to your blog through an RSS feed and read it with an RSS aggregator. An RSS aggregator makes reading blogs like reading e-mail (highlighted new entries, conversation threads, and so on). RSS is a simple XML vocabulary for structuring the content of a blog. Most developers find this new form of communication superior to mailing lists and newsgroups because you control your subscriptions, so you only have to see posts from those you want to listen to, thereby reducing noise and frustration.

Blogging was commonplace at this year's PDC. There were many attendees blogging in real-time during conference sessions, publishing major announcements, key points, and personal reactions. This made it possible for developers everywhere to vicariously experience the exciting moments without risking their lives in L.A. A few developers put together a Web site called Professional Developer Community Bloggers (http://pdcbloggers.net) to make it easier to find PDC-related posts. PDC Bloggers monitors numerous individual blogs and highlights pertinent posts for your convenience. If you didn't get a chance to go to the PDC, you should definitely spend some time on this site.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard teaches at Northface University in Salt Lake City. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000), and frequently speaks at conferences. Reach him at http://www.skonnard.com.


The ASP Column: Web Services: ATL Server Versus ASP.NET

The ASP Column
Web Services: ATL Server Versus ASP.NET
George Shepherd

Code download available at: ASPColumn0402.exe (630KB)
Contents
Web Services
ASP.NET Web Service Architecture
Web Methods
Managing State in ASP.NET
Consuming the Web Service
Managing Headers in ASP.NET
ATL Server Web Service Architecture
Managing State in ATL Server
Consuming the Web Services
Managing Headers in ATL Server
Conclusion

In the November 2003 issue of MSDN® Magazine I looked at the basic architectural differences between ATL Server and ASP.NET to explore two modern approaches to developing Web sites using Microsoft technologies. I looked primarily at developing a forms-based user interface with each framework. ASP.NET centers around the ASP.NET pipeline and some classes running in the common language runtime (CLR). ATL Server uses custom ISAPI extension DLLs and server response files (SRFs). This month I'm going to compare building Web services using ATL Server versus building them with ASP.NET.

Web Services

A Web service is basically a Web site that can interact with other computers and other remote applications. In a way, Web services represent a universal remote procedure call (RPC) mechanism.

The notion of calling methods between different computers is not new. The raw Win32®-based RPC mechanism in the early 1990s and DCOM throughout the mid-to-late 90s represented useful ways to invoke methods across hosts running Windows® operating systems. Unfortunately, these mechanisms did not work well for invoking methods between Windows-based machines and other vendors' systems (CORBA-based systems, for example). While several vendors offered bridge products to allow systems on different kinds of networks to interoperate, some were inconvenient and some simply didn't work. The fundamental problem of connecting multiple computers on heterogeneous networks basically boils down to incompatible wire formats and connection protocols.

Though a number of different protocols and wire formats have evolved over time, the industry is settling upon SOAP as a common wire format with HTTP as the connection protocol.

The beauty of Web services is that it doesn't matter how the endpoints implement the service because they agree on HTTP as the connection protocol and SOAP as the message format. Web services may be implemented any number of ways. For example, if you want to do all the work yourself, you could program a Web service using some socket code and an XML parser. However, that's a lot of work. Web service plumbing generally remains constant across implementations. As long as the plumbing intercepts a remote call, correctly unpacks the SOAP envelope, and turns the message into a real method call on the host machine, the client doesn't need to know nor does it need to care what goes on behind the scenes.

ASP.NET and ATL Server represent two separate ways of implementing Web services using Microsoft technologies. This month's sample code includes two similar applications, one written using ASP.NET and the other using ATL Server. Both apps demonstrate the canonical online Web service calculator. The calculator functions include the requisite Add and Subtract functions. They also include a method for calculating a specific prime number The applications illustrate the similarities and differences between the two frameworks. Using them, I'll illustrate the process of exposing methods, managing state in a Web service, and implementing SOAP headers. First I'll cover these techniques in ASP.NET.

ASP.NET Web Service Architecture

ASP.NET was designed with extensibility in mind. It is built around a well-defined pipeline that's capable of responding to virtually any kind of HTTP request. If you think about it, a SOAP request is just another HTTP request (albeit containing a SOAP envelope). Web services fit very cleanly into the ASP.NET pipeline. Figure 1 shows the ASP.NET architecture.

Figure 1 ASP.NET
Figure 1 ASP.NET

You have a couple of options when building Web services with ASP.NET. The first option is to use the [WebMethod] attribute and the optional System.Web.Services.WebService class. The second option is to write your own HTTP handler to manage the service calls. I'll focus on the first approach, which is more common. I'll cover writing custom SOAP handlers in depth in a future column.

The Visual Studio® .NET wizards include a template for building ASP.NET Web services. The wizard pumps out an ASP.NET project including an ASMX file and codebehind file. The ASMX file includes the Web service directive:

<%@ WebService Language="cs" class="TheWSClass" %>
This directive tells ASP.NET which class contains the Web methods to expose. The codebehind file includes the implementation of this class, optionally deriving from System.Web.Services.WebService. The wizards even stub out a method that's exposed by the [WebMethod] attribute to illustrate the syntax.

Web Methods

Probably the easiest way to get a Web service up and running is to use the [WebMethod] attribute in ASP.NET. WebMethods map SOAP messages to methods on a .NET class. To expose a method through SOAP, just add the WebMethod, as shown in the Calculator class in Figure 2. (To download all the code samples in this column, see the link at the top of this article.)

Figure 2 Using the WebMethod Attribute (Close Figure 2)
// Web service calculator
public class Calculator :
System.Web.Services.WebService
{
  [WebMethod]
  int Add(int x, int y)
  {
    return x + y;
  }

  [WebMethod]
  int Subtract(int x, int y)
  {
    return x - y;
  }

  [WebMethod]
  public long NthPrime(long nth)
  {
    long p = -1;
    // Calc the prime number natively
    // calculate natively...
    CNthPrime nthprime;
    nthprime = new CNthPrime();
    p = nthprime.GetNthPrime(nth);
    nthprime = null;
    return p;
  }
}

The Calculator class exposes several methods as SOAP methods. Notice that the Calculator class derives from System.Web.Services.WebService. This class exposes to the service common ASP.NET objects such as application and session state which are also accessible through Http Context Current. Deriving from WebService is a convenience, not a necessity. The Add, Subtract, and NthPrime methods are preceded by the WebMethod attribute. Add and Subtract are fairly straightforward. Notice that the NthPrime method uses a class named CNthPrime. Finding primes is a bit more complicated—I'll cover that in a minute.

When a client sends a SOAP message to the ASP.NET Web service, it includes the name of the method in two places: in the SOAPAction header and in the request element's name. By default, ASP.NET uses the SOAPAction header to determine which method in the class to call (of course, the method must be marked with the WebMethod attribute). The handler uses .NET reflection to find the method in the class being called by the client. After determining which WebMethod to call, the handler deserializes the SOAP message into .NET objects that can be supplied during method invocation.

For example, in the calculator example in Figure 2, the handler maps parameters for the Add method (both defined as int by the message schema) to .NET values which become the managed parameters used to call the Add method.

Managing State in ASP.NET

Using the WebMethod attribute to expose methods greatly simplifies the process of writing Web services. In addition, WebMethods have access to all other aspects of the .NET Framework, including session and application state. Let's see how that might be useful.

In the calculator example, the site exposes a method for finding a specified prime number. One of the most common algorithms for finding a prime number is the Sieve of Eratosthenes. To find prime numbers, the algorithm uses a couple of integer arrays, eventually storing the prime numbers in one of them. Because the algorithm for finding the nth prime number involves finding all the prime numbers up to that point, running the algorithm once for each separate request is inefficient. Figure 3 shows an ASP.NET class for finding prime numbers.

Figure 3 Find and Store Primes (Close Figure 3)
class CNthPrime
{
  long[] primes;
  long i, k;

  public long[] FindPrimes()
  {
    HttpContext ctx = HttpContext.Current;
    primes = (long[])ctx.Cache["primes"];

    if(primes == null)
    {
       primes = new long[1029];
       ctx.Cache["primes"] = primes;
       // Find the primes here...
       return primes;
    }

    return (long[])ctx.Cache["primes"];
  }

  public long GetNthPrime(long nth)
  {
    primes = FindPrimes();
    if(nth < 1028)
    {
       return(primes[nth]);
    } else
    {
       return -1;
    }
  }
}

The CNthPrime class calculates an array of long numbers representing prime numbers in ascending order. To avoid recalculating the prime numbers each time the method is called, the class stores the array in the ASP.NET cache. (Web methods have access to the global cache, application state, and session state.) Subsequent requests for a prime number are much faster because the numbers have already been calculated and all that's required is a cache hit. The Calculator class uses the CNthPrime class to find the nth prime number in a Web method.

Consuming the Web Service

Once a service is exposed over the Web, it can be consumed by any other application that can find it and that can understand the SOAP protocol. Because crafting SOAP consumers by hand is a tedious and error-prone process, both ASP.NET and ATL Server include client proxy generators. ASP.NET utilizes the System.Web.Services.Description.ServiceDescriptionImporter class. The proxy generator in ATL Server is named SPROXY.EXE.

The ASP.NET Web service example calculates the prime numbers in two ways: using the CNthPrime class and by delegating to the other Web service example from this month (the ATL Server example). It's very straightforward to build a proxy from the ATL Server project. Just add the Web Reference to the project using the Project Explorer. Visual Studio finds the application, retrieves the Web Services Description Language (WSDL), and generates a proxy class. (In addition, the ASP.NET Calculator will be used by the ATL Server Calculator in the same way. That is, the ATL Server Calculator will calculate prime numbers using both CNthPrime and the ASP.NET service.)

The ASP.NET Calculator Web service decides which method to use based on information that comes through some SOAP headers, as you'll see next.

Managing Headers in ASP.NET

Like most method invocation mechanisms, Web service methods are defined by a name and a set of parameters. For example, if you're writing a Web service to validate credit card numbers, you'd probably include the credit card number as a parameter and perhaps return a Boolean. What if you wanted to include some other information in the method call that wasn't directly related to the method call? An example might be that you'd like to make sure that only those clients authorized to look up credit information do so. You might restrict clients by having them pass a hash or some kind of token. On one hand, you could include the hash as a parameter. However, a better approach is to pass the information in SOAP headers. ASP.NET is very flexible when it comes to managing different kinds of SOAP headers.

At this point in time, the standard SOAP specification doesn't list what kinds of things should go in the header. SOAP headers exist to assist the server in handling the request. You'll often find authentication information, client identification information, transaction information, and so forth in the SOAP header.

ASP.NET automates handling SOAP headers with the SoapHeader class and the SoapHeaderAttribute attribute. For example, imagine you wanted to make sure that only certain clients have access to the NthPrime method. You could provide those clients with a special token and require that token be passed as part of a SOAP header. The code in Figure 4 shows how to define and use a SOAP header. It also shows that you can expose SOAP headers to other applications by defining a class representing the data bound for the header. In this case, the information going into the header is a key for unlocking the method and a Boolean flag indicating whether to calculate the number locally or by delegating to another Web service. Notice that the class declares a member variable of type CalcHeader and refers to the member variable in the [SoapHeader] attribute.

Figure 4 Using SOAP Headers in ASP.NET (Close Figure 4)
using localhost;

public class CalcHeader : SoapHeader
{
  public int nKey;
  public bool bDelegateToOtherWS;
}

public class Calculator : System.Web.Services.WebService
{
  public CalcHeader calcHeader;

  // Add and Subtract removed for clarity...
  [WebMethod]
  [SoapHeader("calcHeader", Required=true)]
  public long NthPrime(long nth)
  {
    long p = -1;
    // client must provide token
    //  12345 to get the nth prime
    if(calcHeader !=null && calcHeader.nKey == 12345)
    {
      if(calcHeader.bDelegateToOtherWS == true)
      {
        ATLServerOnlineCalcService
          atlscalc;
        try
        {
          atlscalc = new ATLServerOnlineCalcService();

          atlscalc.CalcHeaderValue = new CalcHeader();

          atlscalc.CalcHeaderValue.m_bDelegateToOtherWS = false;
          atlscalc.CalcHeaderValue.m_lKey =  12345;

          p = atlscalc.NthPrime((int)nth);
        }
        catch(Exception ex)
        {
          System.Console.WriteLine(ex);
        }
      } else
      {
        // Calc the prime number natively
        CNthPrime nthprime;
        nthprime = new CNthPrime();
        p = nthprime.GetNthPrime(nth);
        nthprime = null;
      }
    }
    return p;
  }
}

The second aspect is using SOAP headers from the client's point of view. In addition to exposing the SOAP headers to potential clients, the NthPrime method bases its own decisions on how to calculate the primes using the information provided in the headers. In this case, NthPrime is behaving as a client. The NthPrime method tests the key and examines bDelegateToOtherWS to figure out what to do. If the Web service client has asked to use the other Web service (the ATL Server Calculator), then the method uses the generated proxy class and generated SOAP header-management classes. NthPrime sets up the SOAP headers to send to the ATL Server Calculator.

ATL Server Web Service Architecture

Now let's take a look at the ATL Server architecture. As you saw earlier, ATL Server is another framework for handling HTTP requests. In the November 2003 installment of The ASP Column I looked at how ATL Server handles standard (UI-oriented) requests with standard HTML. ATL Server also handles method-oriented SOAP requests. ATL Server-based Web services use much the same architecture to handle SOAP requests. That is, the application consists of an ISAPI DLL and one or more application DLLs that run in an IIS virtual directory. However, ATL Server-based Web services don't use SRFs. Remember, SRFs incorporate UI tags and call into the application DLLs for processing the request. Figure 5 shows the ATL Server architecture.

Figure 5 ATL Server
Figure 5 ATL Server

Web service-based methods are called directly from the client computer and are generally not related to UI tags. Developing ATL Server-based Web services is in some ways similar to developing ASP.NET Web services.

In ATL Server, the programmatic interface to the Web service is represented as a COM interface. The methods are implemented in a class derived from the interface. The methods to be exposed are marked using the [soap_method] attribute, as shown in Figure 6.

Figure 6 An ATL Server-based Web Service Class (Close Figure 6)
class CATLServerOnlineCalcService :
    public IATLServerOnlineCalcService
{
public:
  [ soap_method ]
  HRESULT Add(LONG x, LONG y, LONG* pResult)
  {
    *pResult = x + y;
    return S_OK;
  }

  [ soap_method ]
  HRESULT Subtract(LONG x, LONG y, LONG* pResult)
  {
    *pResult = x - y;
    return S_OK;
  }

  [ soap_method ]
  HRESULT NthPrime(LONG nth, LONG* plPrime)
  {
    CNthPrime* pnthprime;
    pnthprime = new CNthPrime();
    *plPrime = pnthprime->GetNthPrime(nth, m_spBlobCache);
    delete pnthprime;
    return S_OK;
  }
}; // class CATLServerOnlineCalcService

The [soap_method] injects code into the class for parsing the SOAP method. The injected code unpacks the SOAP message, places parameters on the internal call stack, executes the method, and returns the result as a SOAP response. ATL Server uses reflection to accomplish this. That's why the SOAP methods need to appear in a COM interface—so they have type information generated for them.

Managing State in ATL Server

The ATL Server application uses the same strategy for optimizing performance of the prime number calculator as does the ASP.NET application. Rather than calculating the prime numbers each time the method is called, the ATL Server application calculates the prime numbers once and stores them for later. The code in Figure 7 shows how the ATL Server application stores the prime numbers in the ATL Server BLOB cache (which is very similar in function to the ASP.NET data cache).

Figure 7 Storing Primes in the ATL Server BLOB Cache (Close Figure 7)
class CNthPrime
{
  long rgprimes[8193];

public:
  void FindPrimes(CComPtr<IMemoryCache> spBlobCache)
  {
    HCACHEITEM hItem;
    if(FAILED(spBlobCache-> LookupEntry("rgprimes", &hItem))) {
      CalcPrimes();
      StorePrimes(spBlobCache);
      return;
    }

    void* pData = NULL;     DWORD dwSize;
    HRESULT hr = spBlobCache->GetData(hItem, &pData, &dwSize);
    if (FAILED(hr) || pData == NULL) {
      spBlobCache->ReleaseEntry(hItem);
      CalcPrimes();
      StorePrimes(spBlobCache);
      return;
    }

    memcpy(rgprimes, pData, sizeof(long) * 8193);
    spBlobCache->ReleaseEntry(hItem);
  }

  void CalcPrimes()
  {
    // CalcPrimes removed for clarity
  }

  void StorePrimes( CComPtr<IMemoryCache> spBlobCache)
  {
    DWORD dwSize = sizeof(long)*8193;
    long* pData =
    static_cast<long*>(::HeapAlloc(::GetProcessHeap(), 0, dwSize));
    memcpy(pData, rgprimes, dwSize);

    FILETIME ft = {0};
    HRESULT hr = spBlobCache->Add("rgprimes",
      pData, dwSize,
      &ft, 0, 0, NULL);
  }

  long GetNthPrime(long nth, CComPtr<IMemoryCache> spBlobCache)
  {
    FindPrimes(spBlobCache);
    return(rgprimes[nth]);
  }
};

Like the ASP.NET application, the ATL Server application uses a utility class for calculating the prime numbers. The code generated by Visual Studio stubs out smart pointers to ATL's caching, browser capabilities, and session state services. In this case, the utility class is using the ATL Server BLOB cache facilities represented by the m_spBlobCache member of the Web service. The ATL Server application first checks the BLOB cache for the array of primes. Only if the array is not found does the application take the time to generate the array.

Using the BLOB cache is more complex than using the ASP.NET data cache. Remember, ATL Server is a C++ framework, so all the rules for managing low-level pointers apply. The BLOB cache requires a HANDLE-based approach to storing data. However, the ATL Server BLOB cache is similar to the ASP.NET data cache in that they both are string key/value collections.

Consuming the Web Services

The ATL Server-based calculator also provides the same choice to clients as to how the actual prime number calculation takes place. It's calculated either within the ATL Server application itself or it's delegated to the ASP.NET-based Web service. The ATL Server application holds a proxy class generated from the ASP.NET application (just as the ASP.NET application holds a proxy class generated from the ATL Server application). The ATL Server application uses the proxy during the prime number calculation; this is shown in Figure 8.

Figure 8 Managing SOAP Headers in ATL Server (Close Figure 8)
class CATLServerOnlineCalcService :
    public IATLServerOnlineCalcService
{
public:
  class  CalcHeader
  {
  public:
    long m_lKey;
    bool m_bDelegateToOtherWS;
  };

  CalcHeader m_calcheader;

public:

  [ soap_method ]
  [soap_header(value="m_calcheader", required=true, in=true, out=false)]
  HRESULT NthPrime(LONG nth, LONG* plPrime)
  {
    *plPrime = -1;

    HRESULT hr = S_OK;
    int nUseWS = 0;

    if(m_calcheader.m_lKey == 12345) {
      // run prime number algorithm here...

      if(this->m_calcheader.m_bDelegateToOtherWS) {
        // Use the ASP.NET Web service here...
        Calculator::CCalculator calc;
        calc.CalcHeader0.nKey=12345;
        calc.CalcHeader0.bDelegateToOtherWS = false;

        __int64 l;
        hr = calc.NthPrime(nth, &l);

        *plPrime = l;
      } else
{
        // calculate natively...
        CNthPrime* pnthprime;
        pnthprime = new CNthPrime();
        *plPrime = pnthprime->GetNthPrime(nth);
        delete pnthprime;
      }
    }
    return hr;
  }
}; // class CATLServerOnlineCalcService

Managing Headers in ATL Server

The decision about how to calculate the prime numbers is made by examining information passed through the SOAP headers. ATL Server automates managing SOAP headers in much the same way that ASP.NET does—by defining a class to represent the header and by using the [soap_header] attribute, as shown in Figure 8.

The ATL Server approach to managing headers is almost identical to the ASP.NET approach. The CalcHeader class defines a key and a Boolean flag. The Web service class includes a member variable of type CalcHeader, which is mentioned in the [soap_header] attribute. ATL Server populates the m_calcheader class using information provided by clients through the header.

Conclusion

In the November 2003 column I compared the implementation of a UI-based site using ATL Server and ASP.NET. This time I looked at both frameworks from the Web services perspective. Both are fairly effective when it comes to building Web services and both are comparable in functionality. ATL Server and ASP.NET provide access to session state and support BLOB-style data caching and SOAP methods and headers.

I've found that it's generally much easier to get a Web service going using ASP.NET. Writing managed code tends to be simpler than dealing with pointers and the other minutiae involved with C++ code. On the other hand, if you find yourself firmly bound to C++ without the option of going to managed code, or if you need very fine control of the code in your application, ATL Server is an attractive option for building Web sites and Web services. It's important to note, however, that when "Indigo" (code name for the communications subsystem for the next release of Windows) was announced at this year's Professional Developer's conference, Microsoft articulated the plans for migrating code from a variety of technologies to Indigo. These include ASMX but not ATL Server.

Send your questions and comments for George to asp-net@microsoft.com.

George Shepherd specializes in software development for the .NET Framework. He is the author of Programming with Microsoft Visual C++ .NET (Microsoft Press, 2002) and coauthor of Applied .NET (Addison-Wesley, 2001). He teaches seminars with DevelopMentor and is a contributing architect for Syncfusion's .NET tools.


Wicked Code: Client-side Paging for DataGrids

Wicked Code
Client-side Paging for DataGrids
Jeff Prosise

Code download available at: WickedCode0402.exe (120KB)
Contents
Introducing the ClientPageDataGrid Control
Inside ClientPageDataGrid
Caveats
Wrapping Up

Developers enjoy a love-hate relationship with the ASP.NET DataGrid control. On one hand, DataGrids vastly simplify the task of rendering data sources into stylish HTML tables. On the other hand, DataGrids do the bulk of their work on the server side, causing pages to post back to the server more often than most developers would like. A pageable DataGrid, for example, posts back to the server every time the user navigates from one page to another. Postbacks reduce performance and inhibit responsiveness. No wonder, then, that high on many a developer's wish list is a smarter DataGrid that pages through records without repeatedly posting back to the server.

Conceptually, building a pageable HTML table that does its paging on the client side isn't difficult, thanks to the magic of client-side scripting. The HTML page in Figure 1 shows one technique for implementing client-side paging. When viewed in a browser (see Figure 2), the page displays what appears to be one table. But actually, there are three tables encapsulated in <div> elements. The style attribute attached to each <div> ensures that only one table is visible at a time, and a few lines of JavaScript wired to the Previous and Next links programmatically hide the <div> that's currently displayed, while making the one before or after it visible. The appearing and disappearing tables create the illusion that a single table is used to page through records.

Figure 1 PagingTable.html (Close Figure 1)
<html>
  <head>
    <script language="javascript">
    <!--
    function __onPrevPage ()
    {
        for (var i=2; i<4; i++) {
            if (document.getElementById ("Page" +i).
                style.display == "block") {
                document.getElementById ("Page" + i).
                    style.display = "none";
                document.getElementById ("Page" + (i - 1)).
                    style.display = "block";
                break;
            }
        }
    }

    function __onNextPage ()
    {
        for (var i=1; i<3; i++) {
            if (document.getElementById ("Page" + i).
                style.display == "block") {
                document.getElementById ("Page" + i).
                    style.display = "none";
                document.getElementById ("Page" + (i + 1)).
                    style.display = "block";
                break;
            }
        }
    }
    -->
    </script>
  </head>
  <body>
    <div id="Page1" style="display: block">
      <table border="1" width="40%">
        <tr><td>Page 1</td><td>Page 1</td><td>Page 1</td></tr>
        <tr><td>Page 1</td><td>Page 1</td><td>Page 1</td></tr>
        <tr><td>Page 1</td><td>Page 1</td><td>Page 1</td></tr>
      </table>
    </div>
    <div id="Page2" style="display: none">
      <table border="1" width="40%">
        <tr><td>Page 2</td><td>Page 2</td><td>Page 2</td></tr>
        <tr><td>Page 2</td><td>Page 2</td><td>Page 2</td></tr>
        <tr><td>Page 2</td><td>Page 2</td><td>Page 2</td></tr>
      </table>
    </div>
    <div id="Page3" style="display: none">
      <table border="1" width="40%">
        <tr><td>Page 3</td><td>Page 3</td><td>Page 3</td></tr>
        <tr><td>Page 3</td><td>Page 3</td><td>Page 3</td></tr>
        <tr><td>Page 3</td><td>Page 3</td><td>Page 3</td></tr>
      </table>
    </div>
    <a href="http://javascript:__onPrevPage ();">Prev</a>
    &nbsp;
    <a href="http://javascript:__onNextPage ();">Next</a>
  </body>
</html>
Figure 2 Pageable HTML Table
Figure 2 Pageable HTML Table

The same technique can be used to create DataGrids that support client-side paging. The challenge is to modify the DataGrid class to produce output similar to that in Figure 1. In case you haven't guessed, that's precisely what this installment of Wicked Code is about: improving the ASP.NET DataGrid control by building a plug-in replacement that pages on the client. In addition to adding a useful new tool to your toolbox, it's a wonderful example of how to modify ASP.NET controls by deriving from them and adding functionality of your own.

Introducing the ClientPageDataGrid Control

ClientPageDataGrid is a DataGrid-derived class that adds client-side paging support to ASP.NET. Because it's a functional replacement for the DataGrid, you can use it wherever you'd normally use a DataGrid. Client-side paging comes absolutely for free and can even be turned off.

The Web page in Figure 3 does paging the normal way: by repeatedly posting back to the server. It hosts a DataGrid that's configured to paginate its output (AllowPaging="true") and show 16 records per page (PageSize="16"). Clicking either of the arrows in the pager at the bottom of the grid posts back to the server and fires a PageIndexChanged event, activating the OnNewPage method. The OnNewPage method sets the DataGrid's CurrentPageIndex property to the index of the next or previous page, thus causing that page to render back to the client.

Figure 3 ServerPaging.aspx (Close Figure 3)
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>

<html>
  <body>
    <form runat="server">
      <asp:DataGrid RunAt="server" ID="MyDataGrid"
        AutoGenerateColumns="false" CellPadding="2" Width="100%"
        Font-Name="Verdana" Font-Size="8pt" AllowPaging="true"
        PageSize="16" OnPageIndexChanged="OnNewPage">
        <Columns>
          <asp:BoundColumn HeaderText="Product ID"
            DataField="productid" ItemStyle-HorizontalAlign="center" />
          <asp:BoundColumn HeaderText="Product Name"
            DataField="productname" />
          <asp:BoundColumn HeaderText="Units in Stock"
            DataField="unitsinstock" />
        </Columns>
        <HeaderStyle BackColor="teal" ForeColor="white"
          Font-Bold="true" HorizontalAlign="center" />
        <AlternatingItemStyle BackColor="beige" />
      </asp:DataGrid>
    </form>
  </body>
</html>

<script language="C#" runat="server">
void Page_Load (Object sender, EventArgs e)
{
    if (!IsPostBack)
        BindDataGrid ();
}

void OnNewPage (Object sender, DataGridPageChangedEventArgs e)
{
    MyDataGrid.CurrentPageIndex = e.NewPageIndex;
    BindDataGrid ();
}

void BindDataGrid ()
{
    SqlDataAdapter adapter = new SqlDataAdapter
        ("select productid, productname, unitsinstock from products",
        "server=localhost;database=northwind;Integrated Security=SSPI");
    DataSet ds = new DataSet ();
    adapter.Fill (ds);
    MyDataGrid.DataSource = ds;
    MyDataGrid.DataBind ();
}
</script>

Figure 4 shows the same page implemented with ClientPageDataGrid, with changes shown in red. The DataGrid control has been replaced with ClientPageDataGrid. AllowPaging="true" has been changed to AllowClientPaging="true" to enable client-side paging, and PageSize="16" now reads ClientPageSize="16". Both the OnPageIndexChanged attribute and the OnNewPage method have been deleted. Neither is needed now that paging will be handled entirely on the client.

Figure 4 ClientPaging.aspx (Close Figure 4)
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>
<%@ Register TagPrefix="msdn" Namespace="MSDN"
  Assembly="ClientPageDataGrid" %>

<html>
  <body>
    <form runat="server">
      <msdn:ClientPageDataGrid RunAt="server" ID="MyDataGrid"
        AutoGenerateColumns="false" CellPadding="2" Width="100%"
        Font-Name="Verdana" Font-Size="8pt"
        ClientPageSize="16" AllowClientPaging="true">
        <Columns>
          <asp:BoundColumn HeaderText="Product ID"
            DataField="productid" ItemStyle-HorizontalAlign="center" />
          <asp:BoundColumn HeaderText="Product Name"
            DataField="productname" />
          <asp:BoundColumn HeaderText="Units in Stock"
            DataField="unitsinstock" />
        </Columns>
        <HeaderStyle BackColor="teal" ForeColor="white"
          Font-Bold="true" HorizontalAlign="center" />
        <AlternatingItemStyle BackColor="beige" />
      </msdn:ClientPageDataGrid>
    </form>
  </body>
</html>

<script language="C#" runat="server">
void Page_Load (Object sender, EventArgs e)
{
    if (!IsPostBack)
        BindDataGrid ();
}

void BindDataGrid ()
{
    SqlDataAdapter adapter = new SqlDataAdapter
        ("select productid, productname, unitsinstock from products",
        "server=localhost;database=northwind;Integrated Security=SSPI");
    DataSet ds = new DataSet ();
    adapter.Fill (ds);
    MyDataGrid.DataSource = ds;
    MyDataGrid.DataBind ();
}
</script>

To take the ClientPageDataGrid control for a test drive, copy ClientPaging.aspx to a virtual directory (for example, wwwroot) on your ASP.NET Web server. Then copy the control assembly—ClientPageDataGrid.dll—to the virtual directory's bin subdirectory and open ClientPaging.aspx in your browser. You'll see a page similar to the one in Figure 5. Use the arrows to page backward and forward. ClientPaging.aspx looks and feels exactly like ServerPaging.aspx, but under the hood it is quite different. ServerPaging.aspx places more load on the Web server and can only page as quickly as the connection between client and server will allow. ClientPaging.aspx, by contrast, takes slightly longer to load due to the fact that it's pulling down all of the records rather than just one page worth, but once loaded it pages quite nimbly, regardless of the connection speed.

Figure 5 ClientPageDataGrid
Figure 5 ClientPageDataGrid

ClientPageDataGrid's programmatic interface consists of four public properties that it adds to those inherited from DataGrid (see Figure 6). AllowClientPaging turns paging on and off. The default is true, so you can omit AllowClientPaging="true" from ClientPageDataGrid tags if you want. ClientPageSize controls the number of records shown on each page, while ClientPageCount retrieves the page count. ClientCurrentPageIndex determines which page is currently displayed. The following Page_Load method configures the ClientPageDataGrid named "MyDataGrid" to initially show Page 2 (whose zero-based index is 1) rather than Page 1:

void Page_Load (Object sender, EventArgs e)
{
    if (!IsPostBack) {
        BindDataGrid ();
        MyDataGrid.ClientCurrentPageIndex = 1;
    }
}

Figure 6 ClientPageDataGrid Public Properties (Close Figure 6)

Property Name | Description | Type | Read | Write | Default
AllowClientPaging | Enables and disables client-side paging | bool |  |  | True
ClientPageSize | Gets or sets the number of records displayed per page | int |  |  | 10
ClientPageCount | Gets the current page count | int |  |  | N/A
ClientCurrentPageIndex | Gets or sets the index of the current page | int |  |  | 0

When a page that hosts a ClientPageDataGrid posts backs to the server, the ClientCurrentPageIndex property is updated to indicate which page was displayed on the client when the postback occurred. Between postbacks, the server does not know which page the user is currently viewing.

Inside ClientPageDataGrid

The ability of ClientPageDataGrid to output paginated HTML tables begins with the Render method, which is excerpted in Figure 7. (You can find the complete code download for this column at the link at the top of this article.) Render is a virtual method that all server controls inherit from System.Web.UI.Control. The Render method is called by the Microsoft® .NET Framework to render a control into HTML each time the page hosting the control is requested.

Figure 7 ClientPageDataGrid's Render Method (Close Figure 7)
protected override void Render (HtmlTextWriter writer)
{
    if (AllowClientPaging && Items.Count > 0) {
        for (int i=0; i<ClientPageCount; i++) {
            // Output a <div> tag
            writer.AddAttribute (HtmlTextWriterAttribute.Id,
                ClientID + "_page" + i.ToString ());
            writer.AddAttribute (HtmlTextWriterAttribute.Style,
                i == ClientCurrentPageIndex ?
                    "display: block" : "display: none");
            writer.RenderBeginTag (HtmlTextWriterTag.Div);

            // Render a page inside the <div>
            ShowItems (i * ClientPageSize,
                Math.Min ((i * ClientPageSize) +
                ClientPageSize - 1,
                Items.Count - 1));
            UpdatePager (i);
            base.Render (writer);

            // Output a </div> tag
            writer.RenderEndTag ();
        }

        // Restore all DataGrid items to the visible state
        ShowItems (0, Items.Count - 1);
    }
    else {
        base.Render (writer);
    }
}

ClientPageDataGrid overrides DataGrid's Render method and calls the base class implementation not once, but several times. Specifically, ClientPageDataGrid calls DataGrid.Render once per page of output, and it encapsulates the output in <div> elements by positioning calls to base.Render between calls to RenderBeginTag and RenderEndTag. Before calling the base class's Render method, ClientPageDataGrid.Render invokes a local method named ShowItems to hide rows that don't appear on the page currently being rendered by setting the Visible properties of those rows to false.

For a ClientPageDataGrid configured to display 16 rows per page, the result is a Page 1 consisting of a <div> containing the grid's first 16 items, a Page 2 consisting of a <div> containing the next 16 items, and so on. All <div> elements but the one representing the page that's visible include style="display: none" attributes that prevent them from being seen in a browser. Only the <div> representing the current page—the page whose index equals ClientCurrentPageIndex—is rendered with a style="display: block" attribute to make it visible to the user (see Figure 8).

Figure 8 HTML Generated by ClientPageDataGrid (Close Figure 8)
<div id="MyDataGrid_page0" style="display: block">
  <table ...>
    ...
  </table>
</div>
<div id="MyDataGrid_page1" style="display: none">
  <table ...>
    ...
  </table>
</div>
<div id="MyDataGrid_page2" style="display: none">
  <table ...>
    ...
  </table>
</div>
<div id="MyDataGrid_page3" style="display: none">
  <table ...>
    ...
  </table>
</div>
<div id="MyDataGrid_page4" style="display: none">
  <table ...>
    ...
  </table>
</div>

Each "page" rendered by ClientPageDataGrid includes a pager. The pager is present in the DataGrid because ClientPageDataGrid overrides DataGrid's DataBind method and calls it with AllowPaging set to true and PageSize set to the total number of items in the data source. Much of the code in ClientPageDataGrid.cs is devoted to making the pager work and ensuring that it supports both previous/next-style pagers (<PagerStyle Mode="NextPrev">) and numeric pagers (<PagerStyle Mode="NumericPages">).

A conventional DataGrid pager contains LinkButtons that post back to the server and fire PageIndexChanged events. ClientPageDataGrid replaces these LinkButtons with hyperlinks pointing to JavaScript functions that page on the client. The UpdatePager method (see Figure 9), which is called just before Render calls the base class's Render method, does the replacing. UpdatePager finds the pager by scanning the DataGrid for a row whose type is ListItemType.Pager. Then it deletes the controls inside the pager and adds controls of its own.

Figure 9 UpdatePager Method (Close Figure 9)
void UpdatePager (int page)
{
    // Get a reference to the pager
    TableCell pager = null;
    for (int i = Controls[0].Controls.Count - 1; i >=0; i—) {
        if (((DataGridItem) Controls[0].Controls[i]).ItemType ==
            ListItemType.Pager) {
            pager = (TableCell) Controls[0].Controls[i].Controls[0];
            break;
        }
    }

    if (PagerStyle.Mode == PagerMode.NextPrev) {
        // Implement a client-side Next-Prev pager
        pager.Controls.Clear ();
        pager.Controls.Add (CreatePrevPageControl (page));
        pager.Controls.Add (new LiteralControl ("&nbsp;"));
        pager.Controls.Add (CreateNextPageControl (page));
    }
    else if (PagerStyle.Mode == PagerMode.NumericPages) {
        pager.Controls.Clear ();

        if (ClientPageCount <= PagerStyle.PageButtonCount) {
            // Implement a "1 2 3"-style numeric pager
            for (int i=0; i<ClientPageCount - 1; i++) {
                pager.Controls.Add (CreatePageControl (page, i));
                pager.Controls.Add (new LiteralControl ("&nbsp;"));
            }
            pager.Controls.Add (CreatePageControl (page,
                ClientPageCount - 1));
        }
        else {
            // Implement a "... 1 2 3 ..."-style numeric pager
            int sections = (ClientPageCount +
                PagerStyle.PageButtonCount - 1) /
                PagerStyle.PageButtonCount;
            int section = page / PagerStyle.PageButtonCount;

            // Add leading "..." if not first section
            if (section > 0) {
                pager.Controls.Add (CreatePrevSectionControl (page,
                    section));
                pager.Controls.Add (new LiteralControl ("&nbsp;"));
            }

            // Add page numbers
            int start = section * PagerStyle.PageButtonCount;
            int end = Math.Min (start + PagerStyle.PageButtonCount -1,
                ClientPageCount - 1);

            for (int i=start; i<end; i++) {
                pager.Controls.Add (CreatePageControl (page, i));
                pager.Controls.Add (new LiteralControl ("&nbsp;"));
            }

            pager.Controls.Add (CreatePageControl (page, end));

            // Add trailing "..." if not final section
            if (section < sections - 1) {
                pager.Controls.Add (new LiteralControl ("&nbsp;"));
                pager.Controls.Add (CreateNextSectionControl (page,
                    section));
            }
        }
    }
}

Here's an example of the HTML that a conventional DataGrid control outputs when it renders a pager:

<tr>
  <td colspan="3">
    <a href= "javascript:__doPostBack('MyDataGrid$_ctl20$_ctl0','')">
      &lt;</a>&nbsp;
    <a href="http://javascript:__doPostBack('MyDataGrid$_ctl20$_ctl1','')">
      &gt;</a>
  </td>
</tr>
Here's the same pager rendered by a ClientPageDataGrid control:
<tr>
  <td colspan="3">
    <a href="http://javascript:__onPrevPage ('MyDataGrid');">&lt;</a>&nbsp;
    <a href="http://javascript:__onNextPage ('MyDataGrid');">&gt;</a>
  </td>
</tr>
The JavaScript functions referenced in the anchor tags are registered when ClientPageDataGrid.OnPreRender calls Page.RegisterClientScriptBlock. Calling RegisterClientScriptBlock ensures that the script block containing the functions is included in the output only once, even if a page hosts multiple instances of the ClientPageDataGrid control.

Even though a ClientPageDataGrid pager doesn't induce any postbacks of its own, other controls on the page might—and probably will—cause postbacks to occur. This presents two challenges. First, ClientCurrentPageIndex should be updated when a postback occurs so server-side code can determine which page was displayed when the postback occurred. Second, the ClientPageDataGrid rendered back to the client following a postback should preserve the current page. In other words, if Page 3 is displayed when the page posts back to the server, ClientPageDataGrid's rendering code should attach a style="display: block" attribute to the <div> element representing Page 3 rather than to the <div> element representing Page 1.

To meet these challenges, ClientPageDataGrid registers a hidden field named controlid__ PAGENUM:

Page.RegisterHiddenField (ClientID + "__PAGENUM",
    ClientCurrentPageIndex.ToString ());

The JavaScript functions that page the output on the client update the hidden field each time the current page changes. When the page posts back to the server, the value of the hidden field is included in the postback data. The LoadViewState method of ClientPageDataGrid reads the value from the postback data and assigns it to ClientCurrentPageIndex by calling a local method named RestoreCurrentPageIndex, which does the following:

string page = Page.Request[ClientID + "__PAGENUM"];
  •••
ClientCurrentPageIndex = Convert.ToInt32 (page);

A postback event handler can now determine which page the user was viewing by reading the ClientCurrentPageIndex property. And because ClientPageDataGrid.Render uses ClientCurrentPageIndex to determine which <div> to make visible, the page that was visible before the postback occurred will still be visible following the postback.

Restoring the current page index from postback data in LoadViewState has a couple of advantages. First, the current page index can't be set until after the control is populated with items, and it's in LoadViewState that the control gets populated following a postback. Additionally, because LoadViewState is called before the host page fires a Load event, a Page_Load method can find out what page the user was viewing when the postback occurred by reading ClientCurrentPageIndex. That might be important because the current page index could be used to affect other changes to the page.

RestoreCurrentPageIndex is also called by the DataBind method of the ClientPageDataGrid, but only if it wasn't called from LoadViewState. Why? Because if view state is disabled (that is, if the control's EnableViewState property is false), then LoadViewState isn't called by the ASP.NET runtime. If view state is disabled, the page repopulates the control by calling DataBind, so the DataBind method is the natural place for a second attempt at restoring the current page index from postback data.

You might have noticed that the DataBind method of ClientPageDataGrid binds to the data source not once, but twice (see Figure 10). The first call to the base class's DataBind method provides ClientPageDataGrid with a record count. The second call, which is made with AllowPaging set to true and PageSize set to the record count, produces a DataGrid containing all the records in the data source with a pager at the bottom. This DataGrid lies just beneath the surface of the ClientPageDataGrid and serves as the source of all rendering.

Figure 10 DataBind Method (Close Figure 10)
public override void DataBind ()
{
    if (AllowClientPaging && DataSource != null) {
        object OriginalDataSource = null;

        // If the data source is a DataReader, convert
        // it into a DataSet
        if (DataSource is IDataReader) {
            OriginalDataSource = DataSource;
            DataSource = DataSetFromDataReader
                ((IDataReader) DataSource);
        }

        // Bind once with paging disabled (and ItemCreated
        // and ItemDataBound events suppressed) to get an
        // item count
       _SuppressEvents = true;
        AllowPaging = false;
        base.DataBind ();
       _SuppressEvents = false;

        // Bind again with paging enabled to create a DataGrid
        // containing all items with a pager at the bottom
        PageSize = Items.Count;
        AllowPaging = true;
        base.DataBind ();

        // Restore the original data source
        if (OriginalDataSource != null)
            DataSource = OriginalDataSource;

        // Restore the current page index from postback data
        // if it hasn't been restored already
        if (Page.IsPostBack && AllowClientPaging &&
            !_RestoreCurrentPageIndexCalled)
            RestoreCurrentPageIndex ();
    }
    else {
        base.DataBind ();
    }
}

Another unusual aspect of ClientPageDataGrid's DataBind method is that it converts DataReaders into DataSets. The reason is twofold. First, a DataReader can only be bound to once because it's a forward-only data source. Second, ordinary DataGrids don't support DataReaders as data sources if AllowPaging is true unless AllowCustomPaging is also true. Internally converting DataReaders into DataSets neatly solves both problems and ensures that ClientPageDataGrid works as seamlessly with DataReaders as it does with DataSets.

Caveats

Are there drawbacks to paging on the client? You bet. The larger the recordset, the more slowly the page will load because all the records—not just the records comprising the current page—are returned in the HTTP response. For relatively small recordsets—say, 1,000 records or less—you probably don't care. For very large recordsets, load time could be unacceptably high.

An additional factor to consider when it comes to bandwidth utilization is the DataGrid's propensity to serialize its contents into view state. Large DataGrids increase download times in two ways. First, they generate large HTML tables. Second, they increase the view-state size. Because ASP.NET persists view state using a hidden field, a DataGrid's contents essentially appear once in every HTTP request and twice in every HTTP response. That's all the more reason to keep the record count small and resort to server-side paging for large recordsets. If you must use a ClientPageDataGrid to page through a large number of records, consider setting the control's EnableViewState property to false and binding to the data source in every page request (as opposed to only binding when Page.IsPostBack is false). This could decrease the size of the response by up to two-thirds.

A final thought to keep in mind regarding ClientPageDataGrid is that due to the way it renders individual pages, you'll probably want to make ClientPageSize an even number if you use the AlternatingItemStyle property to render odd-numbered items differently than even-numbered items. Otherwise, Page 1 will have a non-alternating item at the top, Page 2 will have an alternating item, and so on. The user may wonder why the format of the table changes each time he or she flips to the next or previous page.

Wrapping Up

The DataGrid class is a wonderful example of one of the most compelling features of ASP.NET: its ability to hide complex rendering and behavioral logic in reusable classes. ClientPageDataGrid takes the notion of server-side controls a step further and demonstrates not only how to modify built-in control types to work the way you want them to, but how to extend them to add client-side functionality as well. Server controls that rely on client-side script to work smarter and more efficiently are an idea whose time has come. Here's hoping that control authors agree.

Send your questions and comments to Jeff at wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect (http://www.wintellect.com), a software consulting and education firm that specializes in Microsoft .NET.


C++ Q&amp;A: Color Support, Console Apps, and Saving User Settings

C++ Q&A
Color Support, Console Apps, and Saving User Settings
Paul DiLascia

Code download available at: CQA0402.exe (619KB)
Contents
Update

Q I read your July 2001 column on how to create 256-color toolbars. However, I heard that Windows® 95 doesn't support 256-color toolbars unless you have Microsoft® Internet Explorer 3.0 or later. In these cases, the app should use a default 16-color toolbar. How can I determine if a 256-color or a high-color toolbar is supported.

Bayland Park

A I love it—this is 2004 and I'm still getting questions about Windows 95! There's an easy way to find out how many colors the screen or any window supports. Just get the handle to the device context (HDC) for the window or screen, call GetDeviceCaps (get device capabilities) to get the number of color planes (PLANES) and bits per pixel (BITSPIXEL), and multiply these to get the total number of bits in a pixel:

CWindowDC dc(NULL);
int nPlanes = dc.GetDeviceCaps(PLANES);
int nBitsPixel = dc.GetDeviceCaps(BITSPIXEL);
int totalBits = nPlanes * nBitsPixel;
Usually the number of color planes is 1 and the number of bits per pixel is 8 for a 256-color system or 16, 24, or 32 for high-color systems. GetDeviceCaps can get lots of useful information, such as the logical number of pixels per inch, the aspect ratio, raster capabilities, and other information. I wrote a little program, ColorBits (see Figure 1), to show this. You can download it from the link at the top of this article.

Figure 1 ColorBits
Figure 1 ColorBits

Q I've seen programs that can run as either a console app or as a Windows-based app. That is, if you type the name of the program from the command prompt, it runs as a normal Windows-based app, but if you type a command-line option like "-batch", it runs in batch mode as a console app, with all output directed to the console. How does this work?

Joe Tadmann

A I can't tell you how other apps do it because in Windows there's always more than one way to skin the operating system. But I can show you one way that works—the way Visual Studio® .NET does it. You may not have noticed, but you can run Visual Studio .NET by typing devenv at a command prompt, in which case Windows launches the GUI app. But if you type devenv with a command-line switch such as -? for help or -build to build your project, it runs in the console without a user interface. For example

   devenv -build MyProject.sln
builds the solution file MyProject.sln.

It's unfortunate that ever since the advent of GUI, too many programmers have forgotten the power of the command line, which lets users run your app from scripts in batch mode. You can be sure the folks in Redmond don't build their products by opening a project file in Visual Studio! If you've written a program that does something really useful such as converting .wav files to mp3 or computing stock projections, it behooves you to implement a batch interface. What good is your program if someone can't compress their entire music collection or perform stock analysis overnight, without babysitting the computer?

OK, now that I'm off my soapbox, how can you implement a combined GUI/console app? As nearly every programmer targeting Windows knows by now, Windows divides the EXE universe into two camps: console apps and GUI apps. This architecture goes back to the earliest days of Windows, when it first evolved from MS-DOS®. Nowadays, you tell the linker which kind of app you want to generate by using a switch: either /subsystem:Windows or /subsystem:console.

So if you want to build a combined app, the first question you should ask is: will it be a console app or a GUI app? At first, you might think the thing to do is build a console app since it can always run a GUI. There's nothing that says a console app can't create windows or process messages. What makes a console app a console app is that Windows will create a console for it if none exists. But therein lies the problem: if you run a console app from Windows—by double-clicking from Windows Explorer or a shortcut—then Windows will create a console. You can destroy the console by calling FreeConsole, but the console window flashes briefly, announcing to the whole world that you don't really know what you're doing.

OK then, make it a GUI app. But then how do you write to the console? There are many articles explaining how to reroute printf or cout to the console, but they all involve creating a new console window, not using the one that already exists if the program is invoked from the command line. Even if there were a way to use the existing console, how do you know whether your app was invoked from Windows Explorer or a command prompt?

There's a new (for Windows XP) function that would seem to be just the ticket: AttachConsole. This function lets you "attach" your program to a console window of another process. If you use the special process ID ATTACH_PARENT_CONSOLE, AttachConsole attaches to the console from which your program was invoked. Great! But there're two problems. First, AttachConsole only exists in Windows XP, so if you want your program to run in other versions of Windows, you're out of luck; and second, AttachConsole doesn't work exactly as hoped. You can write stuff to the console, but the command prompt is all messed up after your program exits—oops!

In short, a Windows-based app must be either a console app or a GUI app and there's no way to have your cake and eat it too. (Unless you want to write your own startup code—which is waaaay more than I'm up for tonight!) But you know it can be done because I already told you Visual Studio does it—but how?

If you look in your Visual Studio hierarchy, you'll discover that there are actually two—count 'em—programs: devenv.exe and devenv.com. Does anyone remember what a .com is? I don't mean the Web kind, I mean the executable kind. Way, way back when you were just a tot, Windows-based programs had large, small, and huge memory models. There was another model called the compact or tiny model, which produced a different kind of executable with a .com extension. (.com files were a sort of direct memory image that required no fixups on loading, which made them extremely fast—but they had to be small.) Well, memory models are no more, and now all executables use the PE format. But the command interpreter still recognizes .com files as executables, and you can rename any program from foo.exe to foo.com and it'll still execute by typing the name. So the trick is to create two programs: foo.com and foo.exe. One a console app; the other, a Windows-based app.

Figure 2 Process List in a Dialog
Figure 2 Process List in a Dialog

To show how this works, I modified the lp (list processes) program from my July 2002 column to give it the hybrid GUI/console treatment. If you type ListProc with no arguments, it lists the processes in a dialog like the one in Figure 2. If you type ListProc -c, it lists them on the console as in Figure 3. ListProc has two main program files: ListProc.cpp, which is a normal MFC app; and ListProc-cons.cpp, which is a console app. Both programs call the same module, EnumProc, to actually generate the process list. ListProc-cons processes its command line and either displays information on the console or—if there are no command-line args—invokes the GUI version by calling ShellExecute:

// change myself.com to myself.exe and run
TCHAR lpExeName[_MAX_FNAME];
GetModuleFileName(NULL, lpExeName, _MAX_FNAME);
LPTSTR ext = lpExeName + _tcslen(lpExeName) - 3;
_tcscpy(ext,_T("exe"));
ShellExecute(NULL, _T("open"),
  lpExeName, NULL, NULL, SW_SHOWNORMAL);

Figure 3 Process List in a Console
Figure 3 Process List in a Console

Figure 4 shows the complete code for ListProc-cons. The Visual Studio solution file has two projects: ListProc and ListProc-cons. The latter has a custom build step to rename the output file ListProc-cons.exe to ListProc.com (see Figure 5). When you install your program, make sure you put both the .com and the .exe in the same directory, and make sure any shortcuts you create point to the .exe. That way, running directly from Windows invokes the EXE, whereas running from a console invokes the COM (Windows prefers .com to .exe if both exist in the user's PATH). Got it?

Figure 4 ListProc-cons (Close Figure 4)
////////////////////////////////////////////////////////////////
// ListProc.cpp shows how to write an app that can run from a single EXE
// as either a console or GUI app. To run in GUI mode, just type the name
// of the program with command-line args. To run in console/batch mode,
// type the name followed by any option, such as -help.
//
#include "stdafx.h"
#include "EnumProc.h"

static void help();
static BOOL bBare=FALSE;
static BOOL bClassName=FALSE;
static BOOL bTitle=FALSE;

// check for switch: / or -
inline BOOL isswitch(TCHAR c) { return c==L'/' || c==L'-'; }

/////////////////
// Main entry point for console app.
//
int _tmain(int argc, _TCHAR* argv[])
{
   HANDLE hcon = GetStdHandle(STD_OUTPUT_HANDLE);
   TRACE("hcon=%p\n",hcon);

   if (argc <= 1) {
      // Invoked with no command-line args: run in GUI mode.
      TCHAR lpExeName[_MAX_FNAME];
      GetModuleFileName(NULL, lpExeName, _MAX_FNAME);
      LPTSTR ext = lpExeName + _tcslen(lpExeName) - 3;
      _tcscpy(ext,_T("exe"));
      ShellExecute(NULL, _T("open"), lpExeName, NULL, NULL, SW_SHOWNORMAL);
      return 0;
   }

   // Parse command line. Switches can come in any order.
   for (int i=1; i<argc; i++) {
      if (isswitch(argv[i][0])) {
         for (UINT j=1; j<_tcslen(argv[i]); j++) {
            switch(tolower(argv[i][j])) {
            case 'b':
               bBare=TRUE;
               break;
            case 'c': // console : do nothing
               break;
            case 'n':
               bClassName=TRUE;
               break;
            case 't':
               bTitle=TRUE;
               break;
            case '?':
            default:
               help();
               return 0;
            }
         }
      } else {
         help();
         return 0;
      }
   }

   // command-line mode: printf process list
   _tprintf(FormatProcessList(bClassName, bTitle, bBare));
   return 0;
}

void help()
{
   _tprintf(_T("lp:        List top-level proceses.\n"));
   _tprintf(_T("           Copyright 2002 Paul DiLascia.\n\n"));
   _tprintf(_T("Format:    lp [-bcnt]\n\n"));
   _tprintf(_T("Options:\n"));
   _tprintf(_T(" None:          run GUI version\n"));
   _tprintf(_T(" -b(are)        no header\n"));
   _tprintf(_T(" -c(onsole)     display on console (no GUI)\n"));
   _tprintf(_T(" -n(className)  show window class names\n"));
   _tprintf(_T(" -t(itle)       show window titles\n"));
   _tprintf(_T("\n"));
}
Figure 5 Renaming an EXE to a .com File
Figure 5 Renaming an EXE to a .com File

Q I'm building a Windows-based app using the Microsoft .NET Framework and Windows Forms with C#. I'm trying to save the window position so my app remembers the previous position each time it starts up. Is there some special way to do this using the .NET Framework? Should I use the configuration file?

Frank Jacobs

A The .NET Framework supports the notion of configuration files, which are XML files that store application configuration information, but this isn't really what you want here. Configuration files are intended to be used by administrators to configure your application, not by users to save settings. For that, you should use either the registry, INI files, or your own custom data file. The registry is a bad choice because it's difficult to edit and not easily copied. One of the explicit goals of .NET programming is XCOPY deployment, which means you should be able to move your app from point A to B just by copying files. So I would recommend using either an INI file or some other kind of data file.

If you want to store complex data structures like linked lists, hash tables, or your own private data structures, you're better off using serialization (in either binary or XML) to read and write your settings to a special file such as MyApp.dat. Many common classes in the .NET Framework are already serializable, and you can easily make your own classes serializable just by declaring them in the following manner:

[Serializable]
public class MyClass {
  public int myInt;
  public String myStr;
}

For saving simple things like the window position, INI files are a better choice. They're straightforward, simple, human-readable, easily copied and easily edited with any ASCII text editor. Alas, there are no functions in the .NET Framework to access INI files directly, so I wrote a little class called IniFile to encapsulate them for you, and a test app called SavePos that uses IniFile to remember the main window position. SavePos is a typical Windows Forms app. The main form class creates an instance of IniFile like so:

using Ini;
public class Form1 : System.Windows.Forms.Form
{
  private IniFile iniFile = new IniFile("SavePos.ini",true);
•••
}
SavePos.ini is the name of the file; I'll describe the second constructor argument in a moment. The Form constructor loads the window position when the form is first created, before it's displayed, as shown here:
// in Form1 class
public Form1()
{
  •••
  iniFile.RestoreWinPos(this,"MainWindow");
}
And finally, Form1 saves the window position when the form is destroyed, as shown in this code:
// in Form1 class
protected override void Dispose( bool disposing )
{
  if( disposing ) {
    // save window pos
    iniFile.SaveWinPos(this,"MainWindow");
    •••
}

The following code shows the resulting INI file:

[MainWindow]
X=282
Y=442
Width=417
Height=234

As you can see, IniFile provides two functions, SaveWinPos and RestoreWinPos, to save/restore the window position. Figure 6 shows the code; it's mostly straightforward. The save/restore functions call more primitive functions like GetIntVal and SetVal to read/write the key/value pairs. These functions in turn use the interop services to call the Win32® API profile functions GetPrivateProfileInt and WritePrivateProfileString to actually read/write the INI file. The only trick is that when you restore the window position, you have to remember to set the following line of code

form.StartPosition = FormStartPosition.Manual;
if you want your Form to pay attention to Form.Location. (For more on this, see my April 2003 column.)

Figure 6 IniFile (Close Figure 6)
using System;
using System.IO;
using System.Text;
using System.Collections;
using System.Drawing;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;

////////////////////
// .NET wrapper for an initialization file.
//
namespace IniProfile {

   public class IniFile
   {
      [DllImport("Kernel32.dll")]
      private static extern bool WritePrivateProfileString(
         String section, String key, String val, String filename);

      [DllImport("Kernel32.dll")]
      private static extern int GetPrivateProfileInt(
         String section, String key, int dflt, String filename);

      [DllImport("Kernel32.dll")]
      private static extern int GetPrivateProfileString(
         String section, String key, String dflt,
         StringBuilder buf, int nBufLen, String filename);

      private String iniFileName; // .INI file name
      private Hashtable settings = new Hashtable();
      private String currentSection = "Settings";

      public String Section
      {
         get { return currentSection; }
         set { currentSection = value; }
      }

      // ctor: initialize file name
      public IniFile(String fn, bool useAppDataPath) {
         if (useAppDataPath) {
            iniFileName = Application.UserAppDataPath + "\\" + fn;
         } else {
            iniFileName = fn;
         }
      }

      public int GetIntVal(String key, int dflt)
      {
         return GetPrivateProfileInt(currentSection, key, dflt,
         iniFileName);
      }

      public String GetStrVal(String key, String dflt)
      {
         StringBuilder sb = new StringBuilder(256);
         GetPrivateProfileString(currentSection, key, dflt, sb,
            sb.Capacity, iniFileName);
         return sb.ToString();
      }

      public void SetVal(String key, String val)
      {
         WritePrivateProfileString(currentSection, key, val, iniFileName);
      }

      public void SetVal(String key, int val)
      {
         WritePrivateProfileString(currentSection, key, val.ToString(),
         iniFileName);
      }

      public void RestoreWinPos(Form form, String section)
      {
         this.Section = section;
         form.Location = new Point(
            GetIntVal("X",form.Location.X),
            GetIntVal("Y",form.Location.Y));
         form.Size  = new Size(
            GetIntVal("Width", form.Size.Width),
            GetIntVal("Height",form.Size.Height));
         form.StartPosition = FormStartPosition.Manual; // important!
      }

      public void SaveWinPos(Form form, String section)
      {
         this.Section = section;
         SetVal("X",form.Location.X);
         SetVal("Y",form.Location.Y);
         SetVal("Width", form.Size.Width);
         SetVal("Height",form.Size.Height);
      }
   }
}

Astute readers will recall that I owe you an explanation about the second argument to the INI file constructor. The Win32 Get/WriteProfileXxx functions all take an INI file name as an argument. If you pass a relative name like "foo.ini", Windows looks for the file in the WINDOWS directory. If you give an absolute path name, it uses that instead. Since you probably want to save the window position on a per-user basis (Jane should get the last window position she used, not Fred's), you probably want to put the INI file in the user's application data folder for your application. That's what the second argument, useAppDataPath, is for. If you pass true for this value, IniFile looks for the INI file in the user's application data folder.

How does IniFile do this? The Application class makes it easy: Application.UserAppDataPath holds the path name of the user's data folder. This is some humongous path name like \Base\[CompanyName]\[ProductName]\[ProductVersion] where Base is something like C:\Documents and Settings\[username]\Application Data. Ever compulsive in its version management, the common language runtime (CLR) adds the product version number to the path, which is either a feature or a disaster depending on your perspective: if you use an asterisk in your version number (for example, 1.0.*), as many programmers do, to make the framework create a new version for each compile, then you'll end up with hundreds of folders—a new one every time you recompile and run. Perhaps the Redmondtonians are planning to go into the disk business. If you don't want hundreds or thousands of directories clogging up your disk, you can either remove the asterisk from the version number, remove the version part of the path name, or put the file somewhere else. As always you can grab the full source from the link at the top of this article). Happy programming!

Update

In my November 2003 column, I described how to get the HWND of a popup menu by enumerating windows and looking for one with the special class name "#32768". Reader Jim White pointed out the following trick: if you call GetMenuItemRect with NULL as the window handle, Windows will return the HWND of the popup window—amazing! It just goes to show that even supposed experts like me can learn something new every day. Jim reports that the GetMenuItemRect trick works "brillliantly on Windows 2000 and Windows XP, but not on Windows 98, despite MSDN's explicit insistence that it should." Thanks, Jim!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


Resource File: Mobile and Embedded Application Development

Resource File
Mobile and Embedded Application Development

On this page we provide information on the most current Microsoft developer resources. This month we're focusing on mobile and embedded application development.

When embarking on a mobile or embedded project, there are many options from which to choose. From a developer's perspective, there are two core operating systems for mobile development: Windows CE .NET and Windows XP. Windows CE .NET includes Windows Mobile software for Pocket PC and Windows Mobile software for Smartphone as well as custom hardware running Windows CE .NET. There are also a vast array of vendors who provide custom devices running Windows CE .NET. The Windows XP platform includes the Windows XP Tablet PC Edition. Visual Studio .NET 2003 provides you with a single development tool that can target both of these platforms. The latest version of Visual Studio .NET includes the Microsoft .NET Compact Framework which is a subset of the full Microsoft .NET Framework. The .NET Compact Framework is designed specifically for resource-constrained devices such as PDAs and smartphones. Developers familiar with creating apps using Visual Studio .NET will have an easy transition when they begin programming for devices.

The .NET Compact Framework features support for Windows Forms classes, including a complete set of controls, enabling rapid development of rich user interfaces with a structure consistent with the underlying platform. The ADO.NET and XML data classes provide complete support for data handling. The ability to access Web services is integral to the .NET Compact Framework. As with the full .NET Framework, invoking a Web service is as simple as making a function call. The compact version of the full common language runtime (CLR) is optimized for devices and provides many of the same services such as just-in-time (JIT) compilation, memory management, and error handling. The CLR supports the same Microsoft intermediate language (MSIL) and multilanguage capability.

Microsoft SQL Server 2000 Windows CE Edition (also known as SQL Server CE) is designed to integrate with the .NET Compact Framework by means of Visual Studio .NET, simplifying database application development for smart devices. Using the new SQL Server CE data provider to managed code via the CLR, you can build highly extensible mobile applications with offline data management capability for disconnected scenarios.

SQL Server CE enables straightforward, efficient access to enterprise data whether a device is always connected or intermittently connected to the computer running SQL Server. When used in conjunction with SQL Server, SQL Server CE provides extended capabilities for synchronization through merge replication.

Visual Studio .NET 2003 ships with a Windows CE .NET emulator, so that even if you do not own a device you can use the emulation technology to create and test your smart device projects. Visual Studio .NET provides powerful debugging capabilities for smart device projects, consistent with the desktop-based Visual Studio .NET debugger. The debugger works with both physical devices and the emulator.

The Windows Mobile (for Pocket PC and Smartphone) and Windows XP Tablet PC Edition SDKs integrate with Visual Studio .NET 2003 and provide access to platform-specific features. The Tablet PC SDK enables you to access the handwriting and speech extensions to the Windows XP platform. The Windows Mobile SDKs include emulator images for the latest platform releases, allowing you to test their solutions on these new platforms, even before having access to physical devices.

If you want to create mobile Web applications, you can take advantage of ASP.NET Mobile controls. This server-side technology enables ASP.NET to render markup to a wide variety of mobile devices from a single code base. These devices include Wireless Markup Language (WML) and cHTML cell phones, HTML pagers, and PDAs.

To help demystify the solutions available, MSDN Online has created the MSDN Mobile and Embedded Developer Center. The site covers application development for platforms ranging from Windows Mobile software for Smartphone to Windows XP Tablet PC Edition. Mobile-related server technologies and services are also offered on this site.

Developers wishing to learn more about mobile application development should check out Microsoft Mobile DevCon 2004, taking place in San Francisco in late March. See http://www.microsoftmdc.com for the latest details on the conference.